Text image translation method and related device and system

Abstract

The embodiment of the invention provides a text image translation method and a related device and system.The method comprises the steps that after a first picture containing a first text is determined as a first processing object, first image processing is conducted on the first processing object, and a first processing result is obtained; when a translation cache result corresponding to the first processing result exists in the target hierarchy cache, determining a translation object corresponding to the first picture according to the translation cache result, and when the translation object does not exist, if the first image processing is the last hierarchy image processing in the multi-hierarchy image processing, inputting the first processing result into a text translation model to obtain a text translation result, and determining a translation image corresponding to the first picture according to a text translation result, if the image processing is not the last level image processing, determining a first processing object corresponding to the first processing result, and continuing to execute the step of performing the first image processing on the first processing object. Translation of text images is achieved, the image translation processing efficiency is improved, and the image translation overhead is reduced.

Description

TECHNICAL FIELD

[0001] The application relates to the field of computers, and in particular to a text image translation method and related devices and systems. BACKGROUND

[0002] With the continuous development of Internet technology and the acceleration of globalization, cross-language interaction scenarios are becoming more common, and interface translation has become a core requirement for product adaptation to multi-language markets and improvement of user experience. Whether it is a mobile application, a cloud service terminal, a web platform or an interactive interface of intelligent hardware, it contains a large amount of text elements, including not only native vector text such as buttons and navigation bars, but also image text such as picture buttons, activity posters and screenshot embedded text. The accurate conversion of these interface texts is directly related to the smoothness of user operation and the efficiency of information understanding, and is a key prerequisite for product cross-regional promotion.

[0003] In many fields such as cross-border e-commerce, game entertainment and intelligent office, the demand for cross-language conversion of interface text by users is increasingly urgent, and the translation demand for various texts in the interface, especially image embedded text, continues to grow. Under this background, there is an urgent need for a translation method specifically for interface image text to meet the actual application requirements of interface translation in multiple scenarios. SUMMARY

[0004] The embodiments of the present application provide a text image translation method and related devices and systems, which can perform hierarchical processing on the picture to be translated, set different levels of translation cache for different levels of image processing, and only when the translation cache of each level is not hit, perform translation through a language translation model, realize translation for text images, and improve the processing efficiency of picture translation and reduce the overhead of picture translation.

[0005] The first aspect of the embodiments of the present application provides a text image translation method, in which, for a first picture containing a first text, the first picture is determined as a first processing object, the first processing object is subjected to first image processing to obtain a first processing result, when a translation cache result corresponding to the first processing result exists in a target level cache, a translation object corresponding to the first picture is determined according to the translation cache result, when the translation cache result corresponding to the first processing result does not exist in the target level cache, if the first image processing is the last level image processing in multi-level image processing, the first processing result is input into a text translation model to obtain a text translation result, and a translation image corresponding to the first picture is determined according to the text translation result, if the first image processing is not the last level image processing in multi-level image processing, a first processing object corresponding to the first processing result is determined, and the step of performing first image processing on the first processing object is continued.

[0006] The first image processing is a layer in the multi-level image processing. The multi-level image processing includes two or more layers of image processing performed in a specified order. It includes image processing for processing the first image layer by layer to obtain the first text. The first image processing is the image processing that has not been executed in the multi-level image processing and is the first image processing in the specified processing order. The multi-level cache contains multiple levels of cache. The target level cache is one of the levels of cache. Each level of image processing in the multi-level image processing has a unique corresponding level of cache in the multi-level cache. The target level cache and the first image processing are uniquely corresponding to each other.

[0007] In this method, the image to be translated is processed in a hierarchical manner, and different levels of translation cache are set for different levels of image processing. Translation is performed only when the translation caches at all levels are not hit, thereby realizing the translation of text images, improving the processing efficiency of image translation, and reducing the overhead of image translation.

[0008] The even distribution of storage tasks improves the overall performance of the storage system.

[0009] In one alternative implementation, multi-level image processing includes sequentially performed image-aware hash value processing, text region segmentation processing, and region text recognition processing.

[0010] In another alternative implementation, the multi-level cache includes three levels of cache: image translation cache, region translation cache, and text translation cache. The image translation cache is uniquely associated with image-aware hash value processing, the region translation cache is uniquely associated with text region segmentation processing, and the text translation cache is uniquely associated with region text recognition processing.

[0011] In another optional implementation, after determining the translated image corresponding to the first frame based on the text translation result, the method further includes: updating the caches at each level of the multi-level cache based on the translated image corresponding to the first frame and the text translation result. Updating the multi-level cache improves the cache hit rate in subsequent text-image translation processes, thereby improving translation efficiency.

[0012] In another optional implementation, updating the cache at each level of the multi-level cache based on the translated image or text translation result corresponding to the first frame can specifically be as follows: The perceptual hash value of the first frame and the corresponding translated image of the first frame are stored in the image translation cache. The perceptual hash value of the first frame is used as an index in the image translation cache to retrieve the corresponding translated image of the first frame.

[0013] The multi-dimensional feature vector hash value of the text region is determined, and the multi-dimensional feature vector hash value is stored in the region translation cache along with the text translation result. The text region is the first processing result obtained when the first image processing is performed on the text region segmentation process. The multi-dimensional feature vector hash value is used as an index in the region translation cache to retrieve the text translation result.

[0014] The text hash value of the region recognition text is determined, and the text hash value is stored in the text translation cache along with the text translation result. The region recognition text is the first processing result obtained when the first image processing is region text recognition processing. The text hash value is used as an index in the text translation cache to retrieve the text translation result.

[0015] In another alternative implementation, the first frame is a display frame from the first application. This method can also: Acquire the user's historical operation data for the first application. Based on this data, determine the target for prediction translation, which may include one or more of the following: predicted image, predicted text region, or predicted text. Translate the target to obtain the predicted translation result. Store the result in a multi-level cache. Flexibly update the multi-level cache according to user operation habits to improve cache hit rate and translation efficiency in subsequent text and image translation processes.

[0016] In another alternative implementation, when the first image processing is image perceptual hash value processing, the first processing result includes the perceptual hash value of the first image. Specifically, the first image processing on the first processing object can be performed as follows: The first image is reduced to a first size, the reduced first image is converted into a grayscale image, a discrete cosine transform is performed on the grayscale image to obtain the frequency domain feature information of the grayscale image, the low frequency information component in the frequency domain feature information is extracted, and the perceptual hash value of the first image is determined based on the low frequency information component.

[0017] In another alternative implementation, when the first image processing is text region segmentation processing, the first processing result contains at least one text region, and the first image processing of the first processing object may specifically include: Edge detection is performed on the first processing object to obtain the corresponding binary edge image of the first processing object.

[0018] Morphological processing is performed on the binary image of the edge.

[0019] Connectivity analysis was performed on the morphologically processed binary edge image to obtain multiple image regions.

[0020] Extract the regional feature information of each region in multiple image regions.

[0021] The regional feature information of each image region is input into the text region recognition model to obtain at least one text region from multiple image regions.

[0022] In another optional implementation, when the first image processing is region text recognition processing, the first processing result contains at least one region-recognized text, and the first image processing of the first processing object may specifically include: The first processing object is preprocessed with an image. The preprocessed first processing object is then subjected to text line recognition. The recognized text lines are segmented to obtain at least one text line image. The at least one text line image is then input into a text recognition model to obtain at least one region-recognized text.

[0023] In another optional implementation, when the first image processing is text region segmentation, the first processing result contains multiple text regions; when the first image processing is region text recognition, the first processing result contains region-recognized text corresponding to each text region. Inputting the first processing result into the text translation model can specifically include: Based on the location information of the identified text in each region in the first processing result, the identified text in each region is grouped.

[0024] Supplement the context information for each region-identified text. The context information includes one or more of the following: the location information of the region-identified text in the first frame, the attribute information of the first frame, and other region-identified texts belonging to the same group.

[0025] Batch input of region-identified text belonging to the same group and supplemented with contextual information into the text translation model.

[0026] In another optional implementation, where the first image processing is image-aware hash value processing, and the image translation cache contains a translation cache result corresponding to the first processing result, determining the translated image corresponding to the first frame based on the translation cache result may specifically include: The translation cache result corresponding to the first processing result is determined as the translation image corresponding to the first screen.

[0027] In another optional implementation, when the first image processing is text region segmentation or region text recognition, and the target-level cache contains a translation cache result corresponding to the first processing result, determining the translated image corresponding to the first frame based on the translation cache result specifically includes: Obtain the translation cache result corresponding to the first processing result, generate a background mask in the area where the first text is located in the first screen, render and display the translated text corresponding to the translation cache result within the background mask, and perform layer compositing on the translated text corresponding to the translation cache result, the background mask, and the layers where the first screen is located to obtain the translation image corresponding to the first screen.

[0028] A second aspect of this application provides a text-image translation apparatus, including an acquisition unit and a processing unit. Wherein: The acquisition unit is used to acquire the first screen, which contains the first text.

[0029] The processing unit is used to determine the first screen as the first processing object, perform first image processing on the first processing object, and obtain a first processing result. The first image processing is one layer in a multi-level image processing. The multi-level image processing includes two or more layers of image processing performed in a specified processing order, including image processing for processing the first screen layer by layer to obtain the first text. The first image processing is the image processing that has not been performed in the multi-level image processing and is the first image processing in the specified processing order.

[0030] The processing unit is also used for: Determine whether the target-level cache contains the translation cache result corresponding to the first processing result. The target-level cache is a cache at one level of the multi-level cache. The multi-level cache contains multiple levels of cache. Each level of image processing in the multi-level image processing has a unique corresponding level of cache in the multi-level cache. The target-level cache and the first image processing are uniquely corresponding to each other.

[0031] If the judgment result is yes, the translated image corresponding to the first screen is determined based on the translation cache result.

[0032] If the judgment result is negative, and if the first image processing is the last level of multi-level image processing, the first processing result is input into the text translation model to obtain the text translation result, and the translation image corresponding to the first screen is determined based on the text translation result.

[0033] If the first image processing is not the last level of image processing in the multi-level image processing, determine the first processing object corresponding to the first processing result, and perform the first image processing action on the first processing object.

[0034] In another alternative implementation, multi-level image processing includes sequentially performing image-aware hash value processing, text region segmentation processing, and region text recognition processing.

[0035] In another alternative implementation, multi-level caching includes three levels of caching: image translation cache, region translation cache, and text translation cache.

[0036] Among them, the image translation cache and the image perception hash value processing are uniquely related, the region translation cache and the text region segmentation processing are uniquely related, and the text translation cache and the region text recognition processing are uniquely related.

[0037] In another alternative implementation, the processing unit is also used for: After determining the translated image corresponding to the first frame based on the text translation results, the cache of each level in the multi-level cache is updated based on the translated image corresponding to the first frame and the text translation results.

[0038] In another alternative implementation, the processing unit is specifically used for: The perceptual hash value of the first frame and the corresponding translated image of the first frame are stored in the image translation cache. The perceptual hash value of the first frame is used as an index in the image translation cache to retrieve the corresponding translated image of the first frame.

[0039] Determine the multi-dimensional feature vector hash value of the text region, and store the multi-dimensional feature vector hash value and the corresponding text translation result in the region translation cache. The text region is the first processing result obtained when the first image processing is text region segmentation processing. The multi-dimensional feature vector hash value is used as an index to query and retrieve the text translation result in the region translation cache.

[0040] The text hash value of the region recognition text is determined, and the text hash value is stored in the text translation cache along with the text translation result. The region recognition text is the first processing result obtained when the first image processing is region text recognition processing. The text hash value is used as an index in the text translation cache to retrieve the text translation result.

[0041] In another alternative implementation, the processing unit is also used for: Obtain the user's historical operation data for the first application.

[0042] The target for prediction translation is determined based on historical operational data. The target for prediction translation includes one or more of the following: the predicted image, the predicted text region, or the predicted text.

[0043] The predicted translation object is translated to obtain the predicted translation result.

[0044] The predicted translation results are stored in a multi-level cache.

[0045] In another optional implementation, when the first image processing is image perceptual hash value processing, the first processing result includes the perceptual hash value of the first image, and the processing unit is specifically used for: The first image is reduced to a first size, the reduced first image is converted into a grayscale image, a discrete cosine transform is performed on the grayscale image to obtain the frequency domain feature information of the grayscale image, the low frequency information component in the frequency domain feature information is extracted, and the perceptual hash value of the first image is determined based on the low frequency information component.

[0046] In another alternative implementation, when the first image processing is text region segmentation processing, the first processing result contains at least one text region, and the processing unit is specifically used for: Edge detection is performed on the first processing object to obtain the corresponding binary edge image. Morphological processing is performed on the binary edge image. Connectivity analysis is performed on the morphologically processed binary edge image to obtain multiple image regions. Regional feature information of each image region is extracted from the multiple image regions. The regional feature information of each image region is input into the text region recognition model to obtain at least one text region from the multiple image regions.

[0047] In another alternative implementation, when the first image processing is region text recognition processing, the first processing result includes at least one region-recognized text, and the processing unit is specifically used for: The first processing object is preprocessed with an image. The preprocessed first processing object is then subjected to text line recognition. The recognized text lines are segmented to obtain at least one text line image. The at least one text line image is then input into a text recognition model to obtain at least one region-recognized text.

[0048] In another optional implementation, when the first image processing is text region segmentation processing, the first processing result includes multiple text regions; when the first image processing is region text recognition processing, the first processing result includes region-recognized text corresponding to each text region in the multiple text regions. The processing unit is specifically used for: Based on the location information of each region-identified text in the first processing result, the region-identified text is grouped, and context information is added to each region-identified text. The context information includes one or more of the following: the location information of the region-identified text in the first screen, the attribute information of the first screen, and other region-identified texts belonging to the same group. The region-identified texts belonging to the same group and supplemented with context information are then input into the text translation model in batches.

[0049] In another optional implementation, if the first image processing is image-aware hash value processing and the image translation cache contains a translation cache result corresponding to the first processing result, the processing unit is specifically used to determine the translation cache result corresponding to the first processing result as the translated image corresponding to the first frame.

[0050] In another optional implementation, when the first image processing is text region segmentation or region text recognition, and the target-level cache contains a translation cache result corresponding to the first processing result, the processing unit is specifically used for: Obtain the translation cache result corresponding to the first processing result, generate a background mask in the area where the first text is located in the first screen, render and display the translated text corresponding to the translation cache result within the background mask, and perform layer compositing on the translated text corresponding to the translation cache result, the background mask, and the layers where the first screen is located to obtain the translation image corresponding to the first screen.

[0051] This application provides a cloud terminal service system as a third aspect of the embodiments. The system includes a cloud server subsystem, a translation processing subsystem, a caching subsystem, and an intelligent service subsystem, wherein: The cloud server subsystem is used to run the cloud terminal and to capture the first screen of the cloud terminal in real time, which includes the first text.

[0052] The translation processing subsystem is used to identify the first screen as the first processing object, perform first image processing on the first processing object, and obtain the first processing result. The first image processing is a layer in the multi-level image processing. The multi-level image processing includes two or more layers of image processing performed in a specified processing order, including image processing for processing the first screen layer by layer to obtain the first text. The first image processing is the image processing that has not been executed in the multi-level image processing and is the first image processing in the specified processing order.

[0053] The caching subsystem contains multi-level caches. Each level of image processing in the multi-level image processing has a unique corresponding level of cache in the multi-level cache.

[0054] The intelligent service subsystem includes a text translation model.

[0055] The translation processing subsystem is also used for: Determine whether the translation cache result corresponding to the first processing result exists in the target level cache. The target level cache is a cache at one level of a multi-level cache, and the target level cache and the first image processing result are uniquely associated with each other.

[0056] If the judgment result is yes, the translation image corresponding to the first screen is determined based on the translation cache result.

[0057] If the result is negative, and the first image processing is the last level of image processing in a multi-level image processing system, then the first processing result is input into the text translation model to obtain the text translation result, and the translated image corresponding to the first frame is determined based on the text translation result. If the first image processing is not the last level of image processing in a multi-level image processing system, the first processing object corresponding to the first processing result is determined, and the action of performing the first image processing on the first processing object is executed.

[0058] A fourth aspect of this application provides a computer-readable storage medium storing program code that, when executed on a computer, causes the computer to perform the methods described in the first aspect and any possible implementation thereof.

[0059] The fifth aspect of this application provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the methods described in the first aspect and any possible implementation thereof. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating a text-image translation method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating another cloud terminal display screen translation method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a multi-level caching architecture provided in an embodiment of this application; Figure 4 This is a schematic diagram of a text region segmentation processing flow provided in an embodiment of this application; Figure 5 This is a schematic diagram of a regional text recognition processing method provided in an embodiment of this application; Figure 6 This is a schematic diagram illustrating a process for translating text using a text translation model, provided in an embodiment of this application. Figure 7 This is a schematic diagram of a process for synthesizing a translated image corresponding to a first frame, provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a text-image translation device provided in this application example; Figure 9 This is a system architecture diagram of a cloud terminal service system provided in an embodiment of this application; Figure 10 This is a schematic diagram of another text-image translation device provided in an embodiment of this application. Detailed Implementation

[0061] The present application / disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application / disclosure and are not intended to limit the scope of the present application / disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present application / disclosure are shown in the accompanying drawings, not the entire structure.

[0062] This application provides a text-image translation method and related apparatus and system. This method can be applied to various screen translation scenarios, including but not limited to screen translation scenarios for cloud terminals, such as cloud phones, cloud computers, and cloud tablets; screen translation scenarios for physical terminal devices, such as physical phones, physical computers, and physical tablets; and screen translation scenarios for interactions between devices such as storage devices and servers. The screen to be translated can be a real-time screen from a terminal application, webpage display, or device interaction, or a preset non-real-time screen.

[0063] The method provided in the embodiments of this application is described below. The method provided in this application can be implemented by a text-image translation device, which will be described in detail below. Figure 1 , Figure 1 This is a flowchart illustrating a text-to-image translation method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method may include steps S101-S106.

[0064] S101, capture the first screen.

[0065] The first image can be captured by the text-image translation device, such as the real-time image of the cloud terminal. The text-image translation device includes a server responsible for running the cloud terminal. The server can capture the display screen of the cloud terminal in real time. Alternatively, the first image can be received by the text-image translation device, such as the real-time image of the physical terminal. After the physical terminal captures the first image, it transmits it to the text-image translation device for translation.

[0066] The first screen contains the first text. For example, the first screen is the homepage of a cross-border shopping application in a cloud terminal. The homepage may include a navigation bar at the top of the screen, with the application's brand name on the left, a search bar in the middle, a search trigger button on the right with the word "Search" on it, and a login / registration button to the right with the word "Login / Register" on it. The homepage may also include a functional area at the bottom of the screen, containing multiple function buttons, each with its own function label text. These function label texts can be business label texts such as "Home," "Categories," and "My," or service label texts related to platform services such as "Customer Service Center," "After-Sales Guarantee," and "Privacy Policy." The homepage may also include a promotional scrolling bar for sales activities in the middle of the screen, as well as images of recommended products. The promotional scrolling bar may include the brand text, product name text, and discount text of the promoted products, and a brief description of the product may be included below the recommended product images, and so on.

[0067] S102, the first screen is determined as the first processing object.

[0068] The first processing object is the object to be processed in the first image processing steps. In some application scenarios with relatively uniform image format, size, and resolution, the first image can be directly used as the first processing object. In some application scenarios with more diverse image formats, sizes, and resolutions, the first image can be preprocessed in a standardized manner, such as compression, format conversion, and color space conversion, and the preprocessed first image can be determined as the first processing object.

[0069] S103, perform first image processing on the first processing object to obtain a first processing result.

[0070] The first image processing is one layer in a multi-level image processing. The multi-level image processing includes two or more layers of image processing performed in a specified processing order. It includes image processing for processing the first image layer by layer to obtain the first text. The first image processing is the image processing that has not been performed in the multi-level image processing and is the first image processing in the specified processing order.

[0071] The first image can be processed layer by layer to obtain the first text. For example, the first image can be processed sequentially through two layers of image processing: text region segmentation and region text recognition. Multi-level image processing, involving two or more layers of image processing, must be performed according to a pre-set processing order. These two or more layers of image processing can be image processing that processes the first image layer by layer to obtain the first text. For example, the two or more layers of image processing in multi-level image processing may only include the first step of text region segmentation and the second step of text recognition. These two or more layers of image processing can also include other image processing performed on the first image for cache hit retrieval in the multi-level cache. For example, multi-level image processing may include three layers of image processing: image-aware hash value processing (first step), text region segmentation processing (second step), and text region segmentation processing (third step). The first step is used to check for cache hits in the target level cache, while the second and third steps are used to process the first image into the first text. It is understood that each layer of image processing in the above multi-level image processing can also contain multiple processing sub-steps.

[0072] Step S103 can be executed after step S102 or after step S106. It can be understood that if step S103 is executed immediately after step S102, that is, the first execution, the image processing that is ranked first in the multi-level image processing is executed. In the subsequent execution after step S106, the second image processing in the multi-level image processing is executed, that is, the image processing that has not been executed in the multi-level image processing but is ranked first in the specified processing order. And so on. In the subsequent loop, step S103 always executes the image processing that has not been executed in the multi-level image processing but is ranked first in the specified processing order.

[0073] In one optional implementation, if the first image processing is image perceptual hash value processing, the first processing result includes the perceptual hash value of the first image. Specifically, the first image is reduced to a first size, the reduced first image is converted into a grayscale image, a discrete cosine transform is performed on the grayscale image to obtain the frequency domain feature information of the grayscale image, the low-frequency information component in the frequency domain feature information is extracted, and the perceptual hash value of the first image is determined based on the low-frequency information component.

[0074] In another alternative approach, if the first image processing is text region segmentation processing and the first processing result contains at least one text region, in a specific implementation, edge detection can be performed on the first processing object to obtain the edge binary image corresponding to the first processing object, morphological processing can be performed on the edge binary image, and then connected component analysis can be performed on the morphologically processed edge binary image to obtain multiple image regions. The regional feature information of each image region in the multiple image regions can be extracted, and the regional feature information of each image region can be input into the text region recognition model to obtain at least one text region from the multiple image regions.

[0075] In another alternative approach, if the first image processing is region text recognition processing, the first processing result contains at least one region-recognized text. Specifically, the first object can be preprocessed, the preprocessed first object can be used for text line recognition, the recognized text lines can be segmented to obtain at least one text line image, and finally the at least one text line image can be input into the text recognition model to obtain at least one region-recognized text.

[0076] S104, determine whether the translation cache result corresponding to the first processing result exists in the target level cache.

[0077] The target-level cache is a cache level within a multi-level cache. The multi-level cache contains multiple pre-set cache levels. Each level of image processing in the multi-level image processing has a unique corresponding cache level within the multi-level cache. The target-level cache uniquely corresponds to the first image processing. The aforementioned target-level cache is set up specifically for the first image processing. If, before step S104, a translation cache result corresponding to the first processing result appears in the text-image translation device, it can be stored in the target-level cache. This translation cache result can be reused during the current execution of the text-image translation method, improving efficiency. It is understood that the cache level division in the aforementioned multi-level cache is a logical division of the various levels of image processing within the multi-level image processing. In actual deployment, the management relationships and master-slave relationships between various cache media do not conflict with the cache level relationship, and a certain level of cache can also have more subdirectory caches.

[0078] The unique correspondence between a certain level of image processing and a certain level of cache means that in multi-level image processing, there is exactly one level of image processing that corresponds to the cache at that level, and in multi-level caching, there is exactly one level of cache that corresponds to the image processing at that level. These unique correspondences are pre-defined. For example, a multi-level image processing layer might contain two levels of image processing: text region segmentation and text recognition, which need to be executed sequentially. A multi-level cache might contain two levels of cache: a region translation cache and a text translation cache. The text region segmentation cache and the region translation cache are uniquely correlated, as are the text recognition cache and the text translation cache. Similarly, a multi-level image processing layer might contain three levels of image processing: image-aware hashing, text region segmentation, and text region recognition, which need to be executed sequentially. A multi-level cache might contain three levels of cache: an image translation cache, a region translation cache, and a text translation cache. The image translation cache and the image-aware hashing process are uniquely correlated, as are the region translation cache and the text region segmentation process, and the text translation cache and the region text recognition process.

[0079] Specifically, if the first image processing is image-aware hash value processing, the translation cache result corresponding to the first processing result includes the translated image; if the first image processing is text region segmentation processing, the translation cache result corresponding to the first processing result may include the translated text, and optionally may also include the original text; if the first image processing is region text recognition processing, the translation cache result corresponding to the first processing result may include the translated text.

[0080] S105, determine the translation image corresponding to the first screen based on the translation cache result.

[0081] If the translation cache result is the translated image corresponding to the first screen, the translation cache result is obtained as the translated image corresponding to the first screen. If the translation cache result is the result of image processing at a certain level in the process of processing the first screen into the first text layer by layer, or is the first text, then the translation cache result needs to be processed again until the translated image corresponding to the first screen is obtained.

[0082] For example, in the example of multi-level image processing including three-level image processing and multi-level caching including three-level caching in step S104 above, if the first image processing is image perceptual hash value processing and the image translation cache contains the translation cache result corresponding to the first processing result, the translation cache result corresponding to the first processing result can be determined as the translated image corresponding to the first screen; if the first image processing is text region segmentation processing or region text recognition processing and the target level cache contains the translation cache result corresponding to the first processing result, the translation cache result corresponding to the first processing result can be obtained, a background mask can be generated in the region where the first text is located in the first screen, and then the translated text corresponding to the translation cache result can be rendered and displayed in the background mask. Then, the translated text corresponding to the translation cache result, the background mask, and the layers where the first screen is located are layer composited to obtain the translated image corresponding to the first screen.

[0083] S106, if the first image processing is the last level of image processing in a multi-level image processing, then the first processing result is input into the text translation model to obtain the text translation result, and the translated image corresponding to the first frame is determined based on the text translation result. If the first image processing is not the last level of image processing in a multi-level image processing, the first processing object corresponding to the first processing result is determined.

[0084] If the first image processing is the last level of image processing in a multi-level image processing, that is, the first processing result contains the text finally identified in the first frame (ideally, the text is the first text), inputting the text into the text translation model will yield the text translation result (that is, the translated text). Then, the translation image corresponding to the first frame is determined based on the text translation result. The specific implementation method is similar to the implementation method of generating the translation image corresponding to the first frame based on the translation cache result in step S105, and will not be described in detail here.

[0085] Furthermore, the text translation model can be a model trained based on contextual information. Before inputting the first processing result into the text translation model, the contextual information can be fused with the first processing result before being input into the model. Specifically, based on the positional information of the region-identified text in each region of the first processing result, the region-identified text can be grouped, and then contextual information can be added to each region-identified text. The contextual information can include the positional information of the region-identified text in the first frame, the attribute information of the first frame, and other region-identified text belonging to the same group. Finally, the region-identified text belonging to the same group and supplemented with contextual information is input into the text translation model in batches. It can be understood that the supplemented contextual information here has the same format as the contextual information supplemented for samples during model training.

[0086] If the first image processing is not the last level of image processing in the multi-level image processing, it means that the text contained in the first image has not yet been identified. It is necessary to determine the first processing object corresponding to the first processing result and continue to execute step S103 to continue processing the first image into the first text until the text contained in the first image is identified.

[0087] If the first image processing is not the last level of image processing in multi-level image processing, the determination of the first processing object corresponding to the first processing result is related to the design of the first image processing in step S103. For example, in the above example where multi-level image processing includes three levels of image processing and multi-level caching includes three levels of caching, if the first image processing in step S103 is image perception hash value processing, then the first processing object corresponding to the first processing result is the first screen. If the first image processing in step S103 is text region segmentation processing or region text recognition processing, then the first processing object is the first processing result obtained in step S103.

[0088] Optionally, after step S106, the text-image translation device may also update the cache of each level in the multi-level cache according to the translation image and text translation result corresponding to the first screen.

[0089] Specifically, the text-image translation device can store the perceptual hash value of the first frame and the corresponding translated image in an image translation cache, determine the multi-dimensional feature vector hash value of the text region, and store the multi-dimensional feature vector hash value and the text translation result in a region translation cache. It can also determine the text hash value of the region-identified text and store the text hash value and the text translation result in a text translation cache. The hash value of the first frame is used as an index in the image translation cache to retrieve the corresponding translated image. The aforementioned text region refers to the first processing result obtained when the first image processing is text region segmentation processing; the multi-dimensional feature vector hash value is used as an index in the region translation cache to retrieve the aforementioned text translation result. The region-identified text refers to the first processing result obtained when the first image processing is region text recognition processing; the text hash value is used as an index in the text translation cache to retrieve the aforementioned text translation result. In a multi-level cache, each level of cache can be stored in the form of key-value pairs. In image cache translation, the perceptual hash value of the first image is used as the cache key, and the corresponding translated image of the first image is stored as the value. In region translation cache, the hash value of the multi-dimensional feature vector is used as the cache key, and the text translation result is stored as the value. In text translation cache, the text hash value is used as the cache key, and the text translation result is stored as the value.

[0090] Alternatively, the hash values ​​of the multi-dimensional feature vectors can be stored in the region translation cache along with the text translation results and the region recognition text. In other words, the hash values ​​of the multi-dimensional feature vectors can be used as cache keys, and the text translation results and region recognition texts can be stored as values. This allows caching of both the region recognition text contained in the identified text regions and the corresponding text translation results. For example, when changing the text translation model, the region recognition text in the region translation cache can still be retrieved and input into the new text translation model for translation, thus improving flexibility.

[0091] By employing the above process, in the event of a cache miss, the results obtained through model translation are stored in more detail in a multi-level cache. In subsequent image translation processes, this can improve the cache hit rate and thus enhance image translation efficiency.

[0092] Furthermore, the first screen can be a frame displayed in the first application. The text-image translation device can acquire the user's historical operation data for the first application, determine the predicted translation object based on the historical operation data, translate the predicted translation object, obtain the predicted translation result, and store the predicted translation result in a multi-level cache. The predicted translation object can include one or more of the following: the predicted screen, the predicted text region, or the predicted text.

[0093] The aforementioned predicted translation objects can be determined based on the model. Based on the user's historical operation data, it is possible to analyze and identify some screens, text areas, or related texts that are frequently displayed or difficult to translate in the first application. These predicted translation objects can then be stored in a multi-level cache in advance to improve the cache hit rate during the text and screen translation process, thereby improving translation efficiency.

[0094] In this method, after identifying the first image containing the first text as the first processing object, the first image processing is performed on the first processing object to obtain the first processing result. If a translation cache result corresponding to the first processing result exists in the target-level cache, the translation object corresponding to the first image is determined based on the translation cache result. If a translation cache result corresponding to the first processing result does not exist in the target-level cache, if the first image processing is the last level of image processing in a multi-level image processing, the first processing result is input into the text translation model to obtain the text translation result, and the translation image corresponding to the first image is determined based on the text translation result. If the first image processing is not the last level of image processing in a multi-level image processing, the first processing object corresponding to the first processing result is determined, and the step of performing the first image processing on the first processing object continues.

[0095] The first image processing is a layer in the multi-level image processing. The multi-level image processing includes two or more layers of image processing performed in a specified order. It includes image processing for processing the first image layer by layer to obtain the first text. The first image processing is the image processing that has not been executed in the multi-level image processing and is the first image processing in the specified processing order. The multi-level cache contains multiple levels of cache. The target level cache is one of the levels of cache. Each level of image processing in the multi-level image processing has a unique corresponding level of cache in the multi-level cache. The target level cache and the first image processing are uniquely corresponding to each other.

[0096] In this method, the image to be translated is processed in a hierarchical manner, and different levels of translation cache are set for different levels of image processing. Translation is performed only when the translation caches at all levels are not hit, thereby realizing the translation of text images, improving the processing efficiency of image translation, and reducing the overhead of image translation.

[0097] This method can be applied to the translation of real-time displays on cloud terminals, combined with... Figure 2 For introduction and reference Figure 2 , Figure 2 This is a flowchart illustrating another cloud terminal display screen translation method provided in this application embodiment, as shown below. Figure 2 As shown, the method includes steps S201-S211.

[0098] S201 captures the first view from the cloud terminal.

[0099] For example, the first screen could be the real-time display of the first application on the cloud terminal. The first screen contains the first text.

[0100] S202, perform image perception hash value processing on the first image to obtain the perception hash value of the first image.

[0101] Image perceptual hash value processing includes calculating the perceptual hash value of the first image using a preset hash algorithm. The preset hash algorithm can be average hash (aHash), differential hash (dHash), or perceptual hash (pHash) based on DCT (Discrete Cosine Transform). For example, if the preset hash algorithm is pHash, the specific steps are: 1. Reduce the first image to 32×32 pixels; 2. Convert the reduced first image to a grayscale image; 3. Perform a discrete cosine transform on the grayscale image; 4. Extract the low-frequency components to generate a 64-bit hash value (i.e., the perceptual hash value of the first image).

[0102] Optionally, if the first image is a real-time display in the cloud terminal, the images in the cloud terminal can be captured periodically for translation. However, the images captured in two adjacent acquisition cycles may change or not. Translation can be performed only when the images change. Therefore, the Hamming distance between the perceptual hash value of the first image and the perceptual hash value of the image in the previous acquisition cycle can be calculated and compared. When the Hamming distance is less than a preset threshold, it is determined that the image has not changed and no further steps are required. When the Hamming distance is greater than or equal to the preset threshold, it is determined that the image has changed, and the subsequent step S203 is executed. The perceptual hash value of the first image is recorded for use in determining whether the image has changed after the image is captured in the next acquisition cycle.

[0103] S203, determine whether there is a translated image corresponding to the perceptual hash value of the first frame in the image translation cache.

[0104] If not, execute S204; if yes, execute S210.

[0105] The multi-level cache in this embodiment includes three levels of cache, which can be found in the following document. Figure 3 For example, Figure 3 This is a schematic diagram of a multi-level caching architecture provided in an embodiment of this application, such as... Figure 3 As shown, this multi-level cache includes a first level: image translation cache, a second level: region translation cache, and a third level: text translation cache. Data can be stored as key-value pairs in each level of the cache (see details). Figure 1 The corresponding embodiment describes the cache levels in the multi-level cache, which have capacity and TTL (Time To Live) limits. Each level of cache can be managed based on the LRU (Least Recently Used) eviction policy, and can be preprocessed based on a certain cache warm-up mechanism to improve the hit rate. Statistical analysis can also be performed on each level of cache. In step S203, a cache query can be started. First, by comparing the perceptual hash value of the first image, the query is performed to see if there is a matched translated image in the first-level cache. If no match is found, the first image is processed and the query is performed on the next level of cache (i.e., the second-level cache) (corresponding to the action in step S205). If no match is found, the first image is processed again and the query is performed on the third-level cache (corresponding to the action in step S207). If no match is found again, the query is stopped and S204 is executed.

[0106] S204, perform text region segmentation processing on the first screen to obtain at least one text region.

[0107] Can be combined Figure 4 For specific implementation examples, please refer to [link / reference].Figure 4 , Figure 4 This is a schematic diagram of a text region segmentation processing flow provided in an embodiment of this application, such as... Figure 4 As shown, the text region segmentation process can be specifically as follows: The first step is to perform edge detection on the first image. Specifically, noise reduction can be achieved by using a Gaussian filter, for example, the Gaussian kernel size can be 5×5 with a standard deviation of 1.0. Then, the Canny Edge Detection Algorithm (Canny algorithm for short) is used for edge detection, for example, the low threshold is set to 50 and the high threshold is set to 150. Finally, the binary image of the edges of the first image is obtained.

[0108] The second step is to perform morphological processing on the edge binary image. Specifically, morphological closing and opening operations can be performed on the edge binary image using a 3×3 structuring element to connect broken edges and remove small noise points.

[0109] The third step is to perform connected component analysis. Specifically, an 8-connected component editing algorithm can be used to determine the connected components of the morphologically processed image. Then, the bounding box of each connected component is calculated, and regions with an area smaller than a certain area threshold are filtered out.

[0110] The fourth step is feature extraction. Feature extraction is performed on each connected component obtained from the connected component analysis, including aspect ratio features, fill rate features, color distribution features, and texture features (such as LBP (Local Binary Pattern) features).

[0111] The fifth step is region classification. Specifically, regions can be classified based on the extracted features. The various feature vectors obtained from feature extraction are input into a pre-trained SVM classification model. Regions with a confidence level higher than a certain threshold and whose classification result is a text region are identified as at least one text region obtained in step S204.

[0112] In addition, the coordinates of at least one of the above-mentioned text regions can be recorded, and these coordinates can be used as location information in subsequent steps.

[0113] S205, determine whether there is at least one translated text corresponding to a text region in the region translation cache.

[0114] If not, proceed to S206; if yes, proceed to S209.

[0115] Optionally, if the result of step S205 is negative, it can be further determined whether there is at least one original text corresponding to a text region in the region translation cache. If yes, then step S208 is executed; if no, then step S206 is executed.

[0116] S206, Perform region text recognition processing on at least one text region to obtain at least one region-recognized text.

[0117] Combination Figure 5 For specific implementation examples, please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of a region text recognition processing method provided in an embodiment of this application, such as... Figure 5 As shown, the specific processing of regional text recognition can be as follows: The first step is image preprocessing. Specifically, the text region can be converted to grayscale using a weighted average method, then the grayscale image can be adaptively binarized (the threshold can be automatically determined using the Otsu algorithm), and the text line tilt angle can be detected and corrected using Hough transform. Finally, a 3×3 window median filter is used to remove salt-and-pepper noise.

[0118] The second step is to perform text line detection. Specifically, a projection method can be used to detect text lines in the preprocessed text region of the image, then the horizontal projection histogram is calculated, and the text lines are segmented by troughs.

[0119] The third step is to perform CRNN (Convolutional Recurrent Neural Network) model recognition. Specifically, image features of the text region can be extracted first using a CNN network (such as a VGG-like network). Then, the extracted image features are input into an RNN model (such as a bidirectional LSTM model (Long Short-Term Memory network)) to capture the contextual dependencies between sequences. Finally, the output sequence is input into the CTC (Connectionist Temporal Classification) decoding module to decode the feature sequence into a text sequence.

[0120] The fourth step is to perform a confidence assessment. Specifically, first, the confidence of the text sequence obtained by CTC decoding is calculated, and then it is determined whether the confidence is higher than a threshold (e.g., the preset value is 0.8). If it is higher than the threshold, the text sequence is output as the recognized text; otherwise, secondary recognition is initiated and the recognized text is output. During secondary recognition, different recognition model parameters can be used, or the image can be enhanced before recognition.

[0121] S207, determine whether there is at least one region in the text translation cache that corresponds to the translated text.

[0122] If not, proceed to S208; if yes, proceed to S209.

[0123] S208: Input the region-identified text into the text translation model to obtain the translated text.

[0124] Combination Figure 6 For specific implementation examples, please refer to [link / reference]. Figure 6 , Figure 6 This is a schematic diagram illustrating a process for translating text using a text translation model, as provided in an embodiment of this application. Figure 6 As shown, after inputting the text to be translated for region recognition, the specific steps are as follows: The first step is to construct the context. Specifically, the region-identified text can be grouped first, for example, based on the position information of each region-identified text in the first screen (such as the coordinates of the text region). Then, the Euclidean distance between each region-identified text is calculated, and region-identified texts with an Euclidean distance less than a preset distance threshold are grouped together. Then, context analysis is performed on each region-identified text. This can include semantic relevance analysis (which can more accurately filter out more relevant context texts within the same group of region-identified texts by combining semantics), UI (User Interface) hierarchy analysis (which can determine the interface type of the current first screen, such as a login page, settings page, etc.), spatial relationship analysis (which can determine the application type of the first application, etc.), etc. Based on the analysis results, the context of each region-identified text is constructed.

[0125] The second step is to generate contextual prompts. Based on the contextual prompts generated in the first step and the preset system prompt template, contextual prompts are generated for each text region. This converts the identified text and its contextual information into a format that the translation model can recognize. For example, the preset system prompt template might look like this: Please translate the following mobile application UI text, maintaining consistency in terminology.

[0126] Application type: [Automatically identified application type] Current interface: [Interface type] Relevant text context: [other semantically related text in the same group]".

[0127] The third step is to translate the text based on the translation model. Specifically, a translation request can be constructed first, and then the large language model API can be called to translate the text and obtain the translated text. When constructing the translation request, batch input can be performed, and a maximum number of texts in the batch can be set. For example, contextual hints of region recognition texts that are less than the maximum number of texts and belong to the same group can be packaged into the same translation request and input into the translation model through the large language model API. The model can better understand the context and improve processing efficiency.

[0128] The fourth step is post-processing. Specifically, this includes length adaptation, terminology consistency checks, and formatting preservation. Specifically, the length of the translated text output in step three can be adjusted based on the size of the original text area in the first frame. If it is too long, the model can be requested to generate a shorter version of the translated text without losing key information. The style of terminology needs to be consistent with the translation style of other terms in the first frame. Furthermore, the capitalization, punctuation, and other formatting of the translated text must be consistent with the original text in the first frame.

[0129] Step 5, quality assessment. Specifically, the quality assessment may include checking whether all the content of the region-identified text has been translated, whether there are obvious grammatical errors, whether the length is appropriate, etc. If the assessment is satisfactory, the translated text is output; if the assessment is unsatisfactory, step 3 will be repeated.

[0130] S209, synthesize the translated image corresponding to the first screen based on the translated text.

[0131] Combination Figure 7 For specific implementation examples, please refer to [link / reference]. Figure 7 , Figure 7 This is a schematic diagram of a process for synthesizing a translated image corresponding to a first frame, as provided in an embodiment of this application. Figure 7 As shown, after entering the first screen and the translated text, you can: The first step is background mask generation. Specifically, the background color is extracted first, then the edges are feathered, and the transparency is adjusted. Within a certain range (5×5 pixels) around the text area, the background color is extracted using the K-means clustering algorithm. Then, a gradient background mask with a slightly larger area than the text area is generated based on this background color. For example, the mask extends outwards by 2 pixels on each of the four sides of the text area, and the edge transparency of the background mask gradually changes from 100% to 0%.

[0132] The second step is to determine the rendering parameters. This includes font selection, font size calculation, color matching, and alignment. Specifically, a default font can be used, or a custom font can be selected. The font size is determined based on the original height of the text area in the first frame, such as setting "font size = text area height × 0.7". Furthermore, the font size can be dynamically adjusted according to the length of the translated text to fit the text area.

[0133] The third step is translation rendering. Specifically, based on the rendering parameters determined in the second step, the FreeType library (an open-source font rendering library) is used to render the translated text, and subpixel rendering is used to improve the display clarity of the translated text. In addition, the alignment of the translated text can be set according to the alignment of the text in the text area of ​​the first screen, and automatic line wrapping can be enabled in the text area to avoid exceeding the text area and obscuring other content in the first screen.

[0134] The fourth step is layer compositing. Specifically, layer compositing can be performed using an Alpha blending algorithm (Alpha Blending Algorithm, a color blending algorithm based on pixel transparency weights). For example, the specific values ​​of each pixel in the RGB three color channels can be calculated using the following three formulas: R out =R src ×alpha+R dst ×(1-alpha) G out =G src ×alpha+G dst ×(1-alpha) B out =B src ×alpha+B dst ×(1-alpha) The value of alpha can be set to 0.5, and R... src G src and B src It is the RGB value of the foreground layer pixels, R out G out and B out These are the RGB values ​​of the background layer pixels. You can first treat the layer containing the translated text as the foreground layer, the layer containing the background mask as the background layer, and composite them into an intermediate layer. Then, treat the intermediate layer as the foreground layer and the first image as the background layer to complete the final layer compositing. For the edges of the text area, you can use feathered alpha values ​​to achieve a smooth transition.

[0135] Step 5: Quality Optimization. After layer compositing, a slight sharpening filter can be used to enhance text clarity, and the text comparison can be checked to ensure the readability of the translated text. If necessary, the text color can be automatically adjusted to improve contrast and ensure readability.

[0136] Finally, the translated image is output.

[0137] S210, output the translated image corresponding to the first screen.

[0138] For example, the translated image corresponding to the first screen can be transmitted to the cloud terminal for display.

[0139] S211, Update the cache at each level in the multi-level cache.

[0140] Specifically, this may include updating and storing the translated image corresponding to the first frame in the image translation cache; updating and storing the translated text of the text region in the region translation cache, optionally also updating and storing the region-identified text corresponding to the text region in the region translation cache; and storing the translated text corresponding to the original text in the text translation cache.

[0141] In practical applications, various other performance optimization strategies can be employed. For example, parallel processing optimization can utilize thread pools to concurrently process text recognition across multiple text regions, and GPUs (Graphics Processing Units) can be used for layer compositing. Memory optimization can also be achieved by using object pools to reduce frequent memory allocation, promptly releasing large image objects to prevent memory leaks, periodically compressing and storing cached data, and employing other resource pooling methods such as thread pool reuse, connection pool management, and memory pool allocation to reuse resources, reduce overhead, and improve stability. Network optimization can address the transmission of translation requests, images, and text by using HTTP / 2 (Hypertext Transfer Protocol Version 2) multiplexing to reduce connection overhead, enabling gzip compression to reduce data transmission volume, merging model translation requests, and batch processing. Furthermore, progressive processing can be employed, prioritizing the translation of text in user-focused regions and using predictive caching. For example, incremental updates reduce the amount of data transmitted or stored. This can be achieved through differential frame transmission, which only transmits the differences between the current frame and the reference frame (usually the previous frame); local area updates, which only update the data in the changed text areas; and incremental cache synchronization, which records only the changed parts of cached data. Furthermore, lazy loading techniques such as on-demand loading models and lazy initialization caches enable on-demand creation and cache reuse.

[0142] In this embodiment, for the first screen displayed in real time on the cloud terminal, the first screen is processed in a hierarchical manner, and the cache at different levels is retrieved level by level. Only when all levels of screen are hit, the translation model is used to translate only, thereby improving the processing efficiency of screen translation, reducing the overhead of screen translation, and ensuring the smoothness of screen translation and display in multilingual scenarios on the cloud terminal.

[0143] The apparatus and system provided in the embodiments of this application are described below, see below. Figure 8 ,Figure 8 This is a schematic diagram of the structure of a text-image translation device provided in this application example, such as... Figure 8 As shown, the text-image translation device 80 includes an acquisition unit 801 and a processing unit 802. Wherein: Acquisition unit 801 is used to acquire the first screen, which contains the first text.

[0144] The processing unit 802 is used to determine the first screen as the first processing object, perform first image processing on the first processing object, and obtain a first processing result. The first image processing is one layer in a multi-level image processing. The multi-level image processing includes two or more layers of image processing performed in a specified processing order, including image processing for processing the first screen layer by layer to obtain the first text. The first image processing is the image processing that has not been executed in the multi-level image processing and is the first image processing in the specified processing order.

[0145] Processing unit 802 is also used for: Determine whether the target-level cache contains the translation cache result corresponding to the first processing result. The target-level cache is a cache at one level of the multi-level cache. The multi-level cache contains multiple levels of cache. Each level of image processing in the multi-level image processing has a unique corresponding level of cache in the multi-level cache. The target-level cache and the first image processing are uniquely corresponding to each other.

[0146] If the judgment result is yes, the translated image corresponding to the first screen is determined based on the translation cache result.

[0147] If the judgment result is negative, and if the first image processing is the last level of multi-level image processing, the first processing result is input into the text translation model to obtain the text translation result, and the translation image corresponding to the first screen is determined based on the text translation result.

[0148] If the first image processing is not the last level of image processing in the multi-level image processing, determine the first processing object corresponding to the first processing result, and perform the first image processing action on the first processing object.

[0149] Optionally, multi-level image processing includes image-aware hash value processing, text region segmentation processing, and region text recognition processing performed sequentially.

[0150] Optionally, multi-level caching includes three levels: image translation cache, region translation cache, and text translation cache.

[0151] Among them, the image translation cache and the image perception hash value processing are uniquely related, the region translation cache and the text region segmentation processing are uniquely related, and the text translation cache and the region text recognition processing are uniquely related.

[0152] Optionally, the processing unit 802 is also used for: After determining the translated image corresponding to the first frame based on the text translation results, the cache of each level in the multi-level cache is updated based on the translated image corresponding to the first frame and the text translation results.

[0153] Optionally, the processing unit 802 is specifically used for: The perceptual hash value of the first frame and the corresponding translated image of the first frame are stored in the image translation cache. The perceptual hash value of the first frame is used as an index in the image translation cache to retrieve the corresponding translated image of the first frame.

[0154] Determine the multi-dimensional feature vector hash value of the text region, and store the multi-dimensional feature vector hash value and the corresponding text translation result in the region translation cache. The text region is the first processing result obtained when the first image processing is text region segmentation processing. The multi-dimensional feature vector hash value is used as an index to query and retrieve the text translation result in the region translation cache.

[0155] The text hash value of the region recognition text is determined, and the text hash value is stored in the text translation cache along with the text translation result. The region recognition text is the first processing result obtained when the first image processing is region text recognition processing. The text hash value is used as an index in the text translation cache to retrieve the text translation result.

[0156] Optionally, the processing unit 802 is also used for: Obtain the user's historical operation data for the first application.

[0157] The target for prediction translation is determined based on historical operational data. The target for prediction translation includes one or more of the following: the predicted image, the predicted text region, or the predicted text.

[0158] The predicted translation object is translated to obtain the predicted translation result.

[0159] The predicted translation results are stored in a multi-level cache.

[0160] Optionally, when the first image processing is image perceptual hash value processing, the first processing result includes the perceptual hash value of the first image. The processing unit 802 is specifically used for: The first image is reduced to a first size, the reduced first image is converted into a grayscale image, a discrete cosine transform is performed on the grayscale image to obtain the frequency domain feature information of the grayscale image, the low frequency information component in the frequency domain feature information is extracted, and the perceptual hash value of the first image is determined based on the low frequency information component.

[0161] Optionally, when the first image processing is text region segmentation processing, the first processing result contains at least one text region, and the processing unit 802 is specifically used for: Edge detection is performed on the first processing object to obtain the corresponding binary edge image. Morphological processing is performed on the binary edge image. Connectivity analysis is performed on the morphologically processed binary edge image to obtain multiple image regions. Regional feature information of each image region is extracted from the multiple image regions. The regional feature information of each image region is input into the text region recognition model to obtain at least one text region from the multiple image regions.

[0162] Optionally, when the first image processing is region text recognition processing, the first processing result includes at least one region-recognized text, and the processing unit 802 is specifically used for: The first processing object is preprocessed with an image. The preprocessed first processing object is then subjected to text line recognition. The recognized text lines are segmented to obtain at least one text line image. The at least one text line image is then input into a text recognition model to obtain at least one region-recognized text.

[0163] Optionally, when the first image processing is text region segmentation processing, the first processing result includes multiple text regions; when the first image processing is region text recognition processing, the first processing result includes region-recognized text corresponding to each text region in the multiple text regions. Processing unit 802 is specifically used for: Based on the location information of each region-identified text in the first processing result, the region-identified text is grouped, and context information is added to each region-identified text. The context information includes one or more of the following: the location information of the region-identified text in the first screen, the attribute information of the first screen, and other region-identified texts belonging to the same group. The region-identified texts belonging to the same group and supplemented with context information are then input into the text translation model in batches.

[0164] Optionally, if the first image processing is image-aware hash value processing and the image translation cache contains a translation cache result corresponding to the first processing result, the processing unit 802 is specifically used to determine the translation cache result corresponding to the first processing result as the translated image corresponding to the first frame.

[0165] Optionally, when the first image processing is text region segmentation processing or region text recognition processing, and the target-level cache contains a translation cache result corresponding to the first processing result, the processing unit 802 is specifically used for: Obtain the translation cache result corresponding to the first processing result, generate a background mask in the area where the first text is located in the first screen, render and display the translated text corresponding to the translation cache result within the background mask, and perform layer compositing on the translated text corresponding to the translation cache result, the background mask, and the layers where the first screen is located to obtain the translation image corresponding to the first screen.

[0166] It is understood that the text-image translation device 80 in the embodiments of this application can achieve... Figure 1 or Figure 2 The method in the corresponding embodiment. Regarding... Figure 8 For details on the specific implementation methods and corresponding beneficial effects of the functional components included in the text-image translation device 80, please refer to the foregoing. Figure 1 or Figure 2 A detailed description of the embodiments.

[0167] This application also provides a cloud terminal service system, which includes a cloud server subsystem, a translation processing subsystem, a caching subsystem, and an intelligent service subsystem, wherein: The cloud server subsystem is used to run the cloud terminal and to capture the first screen of the cloud terminal in real time, which includes the first text.

[0168] The translation processing subsystem is used to identify the first screen as the first processing object, perform first image processing on the first processing object, and obtain the first processing result. The first image processing is a layer in the multi-level image processing. The multi-level image processing includes two or more layers of image processing performed in a specified processing order, including image processing for processing the first screen layer by layer to obtain the first text. The first image processing is the image processing that has not been executed in the multi-level image processing and is the first image processing in the specified processing order.

[0169] The caching subsystem contains multi-level caches. Each level of image processing in the multi-level image processing has a unique corresponding level of cache in the multi-level cache.

[0170] The intelligent service subsystem includes a text translation model.

[0171] The translation processing subsystem is also used for: Determine whether the translation cache result corresponding to the first processing result exists in the target level cache. The target level cache is a cache at one level of a multi-level cache, and the target level cache and the first image processing result are uniquely associated with each other.

[0172] If the judgment result is yes, the translation image corresponding to the first screen is determined based on the translation cache result.

[0173] If the result is negative, and the first image processing is the last level of image processing in a multi-level image processing system, then the first processing result is input into the text translation model to obtain the text translation result, and the translated image corresponding to the first frame is determined based on the text translation result. If the first image processing is not the last level of image processing in a multi-level image processing system, the first processing object corresponding to the first processing result is determined, and the action of performing the first image processing on the first processing object is executed.

[0174] In the process of performing the first image processing on the first processing object, the translation processing subsystem can be executed by its own processor or by calling other modules. For example, the intelligent service subsystem may also include a text recognition model. When the first image processing is regional text recognition processing, the text recognition model in the intelligent service subsystem can be called for processing.

[0175] Combination Figure 9 The system provided in the embodiments of this application is illustrated by example. See also: Figure 9 , Figure 9 This is a system architecture diagram of a cloud terminal service system provided in an embodiment of this application, such as... Figure 9 The cloud terminal service system shown includes a cloud server subsystem, a translation processing subsystem, a caching subsystem, and an intelligent service subsystem, among which: The cloud server subsystem includes a cloud-based Android virtual machine, a screen capture module, and a screen stream encoder. The cloud-based Android virtual machine can run a primary application, which can then display the screen stream on the client side. The screen capture module can periodically capture the first frame of the screen stream. The screen stream encoder can encode the first frame captured by the screen capture module and transmit it to the translation processing subsystem.

[0176] The caching subsystem includes a cache management module, an image translation cache, a region translation cache, and a text translation cache. The cache management module manages the image translation cache, region translation cache, and text translation cache. The image translation cache caches the translated images corresponding to the screen. The region translation cache caches the translated text and the original text corresponding to the text region. The text translation cache caches the translated text of the original text identified in the text region.

[0177] The intelligent service subsystem includes a text recognition service module and a large language model translation service module. The text recognition service module may include an OCR (Optical Character Recognition) model, which is used to recognize the text contained in a text region. The large language model translation service module includes a language translation model, which is used to translate the text.

[0178] The translation processing subsystem includes a screen decoder, a screen change detection module, a text region recognition module, a text recognition engine, a translation processing engine, an image synthesis module, and an image encoder. The screen decoder decodes the first screen transmitted by the screen stream encoder in the source service subsystem. The screen detection module detects whether the current first screen has changed compared to the first screen acquired in the previous period. If there is no change, no processing is performed. If there is a change, the screen detection module checks the screen translation cache for a corresponding translated image of the first screen. If a translation image is found, it is retrieved; otherwise, it triggers the text region recognition module to perform an action. The text region recognition module, triggered by the image change detection module, performs text region segmentation on the first image to obtain text regions. It then checks the region translation cache for corresponding translated text. If translated text exists, it triggers the image synthesis module to generate a translated image of the first image based on the translated text. If no translated text exists but the source text exists, it retrieves the source text and triggers the translation engine to translate it. If neither translated nor source text exists, it triggers the text recognition engine to perform text recognition processing on the text regions. The text recognition engine checks the text translation cache for corresponding translated text. If found, it retrieves the translated text and transmits it to the image synthesis module. If not found, it calls the text recognition service module to perform text recognition on the text regions based on an OCR model. The translation processing engine calls the large language model translation service module to translate the region-recognized text based on a language translation model to obtain translated text. The image synthesis module generates a translated image of the first image based on the translated text. The image encoder transmits the translated image corresponding to the first image to the encoder and then to the client, so that the client can decode the translated image corresponding to the first image through the image decoder and display it through the image display module.

[0179] The cloud terminal service system provided in this application embodiment can implement each step of the text-image translation method provided in this application embodiment based on its various subsystems. For specific implementation and beneficial effects, please refer to [link / reference]. Figure 1 or Figure 2 The corresponding descriptions in the embodiments will not be repeated here.

[0180] SeeFigure 10 , Figure 10 This is a schematic diagram of another text-image translation device provided in an embodiment of this application, such as... Figure 10 As shown, the data processing device 100 includes a processor 1001, a memory 1002, and a communication interface 1003. The processor 1001 is connected to the memory 1002 and the communication interface 1003, for example, the processor 1001 can be connected to the memory 1002 and the communication interface 1003 via a bus.

[0181] Processor 1001 is configured to support data processing device execution Figure 1 or Figure 2 The corresponding function in the corresponding method. The processor 1001 can be a Central Processing Unit (CPU), a Network Processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip can be an Application-Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The aforementioned PLD can be a Complex Programmable Logic Device (CPLD), a Field-Programmable Gate Array (FPGA), a Generic Array Logic (GAL), or any combination thereof.

[0182] Memory 1002 is used to store program code, etc. Memory 1002 includes internal memory, which may include at least one of the following: volatile memory (e.g., dynamic random access memory (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.) and non-volatile memory (e.g., one-time programmable read-only memory (OTPROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM). Memory 1202 may also include external memory, which may include at least one of the following: hard disk drive (HDD) or solid-state drive (SSD), flash drive, such as high-density flash (CF), secure digital (SD), micro SD, mini SD, extreme digital (xD), memory stick, etc.

[0183] Communication interface 1003 is used to receive or send data.

[0184] Processor 1001 can call the program code to perform the following operations: Get the first data; Based on the first data identifier, the first group identifier is determined using a consistent hashing algorithm, where the first data identifier is the identifier of the first data. Based on a random selection algorithm, the first storage node corresponding to the first group identifier is determined from at least two storage nodes; Send the first data identifier and the first data to the first storage node.

[0185] It should be noted that the implementation of each operation can also be referenced accordingly. Figure 1 or Figure 2 The corresponding description of the method embodiments shown; the processor 1001 can also be used to perform other operations in the above method embodiments.

[0186] This application also provides a computer program product that, when run by a computer, can perform the above-described... Figure 1 or Figure 2 The method in the illustrated embodiment.

[0187] This invention also provides a computer storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in the foregoing embodiments. The computer may be part of the aforementioned text-image translation device.

[0188] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0189] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A text-image translation method, characterized in that, The method includes: Get the first screen, which contains the first text; The first image is identified as the first processing object; the first image is processed to obtain a first processing result; the first image processing is a layer in a multi-level image processing, which includes two or more layers of image processing performed in a specified processing order, including image processing for processing the first image layer by layer to obtain the first text; the first image processing is the image processing that has not been performed in the multi-level image processing and is the first image processing in the specified processing order. Determine whether the target-level cache contains a translation cache result corresponding to the first processing result; the target-level cache is a cache at one level of a multi-level cache; the multi-level cache contains multiple levels of cache, and each level of image processing in the multi-level image processing has a unique corresponding level of cache in the multi-level cache; the target-level cache and the first image processing are uniquely corresponding to each other; If the determination result is yes, the translated image corresponding to the first screen is determined based on the translation cache result; If the determination result is negative, and if the first image processing is the last level of image processing in the multi-level image processing, then the first processing result is input into the text translation model to obtain the text translation result, and the translated image corresponding to the first screen is determined based on the text translation result; if the first image processing is not the last level of image processing in the multi-level image processing, the first processing object corresponding to the first processing result is determined, and the action of performing the first image processing on the first processing object is executed.

2. The method according to claim 1, characterized in that, The multi-level image processing includes image-aware hash value processing, text region segmentation processing, and region text recognition processing, which are performed sequentially.

3. The method according to claim 2, characterized in that, The multi-level cache includes three levels: image translation cache, region translation cache, and text translation cache. Specifically, the image translation cache and the image perception hash value processing are uniquely correlated; the region translation cache and the text region segmentation processing are uniquely correlated; and the text translation cache and the region text recognition processing are uniquely correlated.

4. The method according to claim 3, characterized in that, After determining the translated image corresponding to the first frame based on the text translation result, the method further includes: Based on the translated image corresponding to the first screen and the text translation result, update the cache of each level in the multi-level cache.

5. The method according to claim 4, characterized in that, The step of updating the cache at each level of the multi-level cache according to the translated image corresponding to the first screen or the text translation result includes: The perceptual hash value of the first image and the corresponding translated image of the first image are stored in the image translation cache; the perceptual hash value of the first image is used as an index in the image translation cache to retrieve the corresponding translated image of the first image. A multi-dimensional feature vector hash value is determined for a text region, and the multi-dimensional feature vector hash value is stored in the region translation cache in correspondence with the text translation result; the text region is the first processing result obtained when the first image processing is text region segmentation processing; the multi-dimensional feature vector hash value is used as an index in the region translation cache to retrieve the text translation result; The text hash value of the region-identified text is determined, and the text hash value is stored in the text translation cache in correspondence with the text translation result; the region-identified text is the first processing result obtained when the first image processing is region text recognition processing; the text hash value is used as an index to query and retrieve the text translation result in the text translation cache.

6. The method according to any one of claims 1-5, characterized in that, The first frame is a display frame in the first application; the method further includes: Obtain the user's historical operation data for the first application; The predicted translation object is determined based on the historical operation data, and the predicted translation object includes one or more of the following: predicted image, predicted text region, or predicted text. The predicted translation object is translated to obtain the predicted translation result; The predicted translation results are stored in the multi-level cache.

7. The method according to any one of claims 2-5, characterized in that, When the first image processing is the image perception hash value processing, the first processing result includes the perception hash value of the first image; The first image processing of the first processing object includes: Reduce the first image to the first size; Convert the scaled-down first image into a grayscale image; Perform a discrete cosine transform on the grayscale image to obtain the frequency domain feature information of the grayscale image; Extract the low-frequency information component from the frequency domain feature information; The perceptual hash value of the first image is determined based on the low-frequency information components.

8. The method according to any one of claims 2-5, characterized in that, When the first image processing is the text region segmentation processing, the first processing result contains at least one text region, and the first image processing on the first processing object includes: Edge detection is performed on the first processing object to obtain the edge binary image corresponding to the first processing object; Morphological processing is performed on the binary image of the edge; Connectivity analysis is performed on the morphologically processed binary edge image to obtain multiple image regions; Extract the regional feature information of each image region from the multiple image regions; The regional feature information of each image region is input into the text region recognition model to obtain at least one text region from the multiple image regions.

9. The method according to any one of claims 2-5, characterized in that, When the first image processing is the region text recognition processing, the first processing result contains at least one region-recognized text, and the first image processing of the first processing object includes: Perform image preprocessing on the first processing object; The first processed object after image preprocessing is subjected to text line recognition; The identified text lines are segmented to obtain at least one text line image; Input the at least one text line image into the text recognition model to obtain the at least one region-recognized text.

10. The method according to any one of claims 2-5, characterized in that, When the first image processing is text region segmentation processing, the first processing result includes multiple text regions; when the first image processing is region text recognition processing, the first processing result includes region recognition text corresponding to each text region in the multiple text regions. The step of inputting the first processing result into the text translation model includes: Based on the location information of the identified text in each region in the first processing result, the identified text in each region is grouped. Supplement the context information for each of the region-identified texts; the context information includes one or more of the following: the location information of the region-identified text in the first screen, the attribute information of the first screen, and other region-identified texts belonging to the same group; The region-identified texts belonging to the same group and supplemented with contextual information are batch-input into the text translation model.

11. The method according to any one of claims 3-5, characterized in that, When the first image processing is image-aware hash value processing, and the image translation cache contains a translation cache result corresponding to the first processing result, determining the translated image corresponding to the first frame based on the translation cache result includes: The translation cache result corresponding to the first processing result is determined as the translated image corresponding to the first screen.

12. The method according to any one of claims 3-5, characterized in that, When the first image processing is text region segmentation processing or region text recognition processing, and the target-level cache contains a translation cache result corresponding to the first processing result, determining the translated image corresponding to the first frame based on the translation cache result includes: Obtain the translation cache result corresponding to the first processing result; A background mask is generated in the area where the first text is located in the first image; The translated text corresponding to the translation cache result is rendered and displayed within the background mask; The translated text corresponding to the translation cache result, the background mask, and the layer where the first image is located are layer composited to obtain the translated image corresponding to the first image.

13. A text-image translation device, characterized in that, The device includes: The acquisition unit is used to acquire the first screen, which contains the first text. A processing unit is configured to determine the first image as a first processing object; perform first image processing on the first processing object to obtain a first processing result; the first image processing is a layer in a multi-level image processing, the multi-level image processing includes two or more layers of image processing performed in a specified processing order, including image processing for processing the first image layer by layer to obtain the first text; the first image processing is the image processing that has not been performed in the multi-level image processing and is the first one in the specified processing order. The processing unit is also used for: Determine whether the target-level cache contains a translation cache result corresponding to the first processing result; the target-level cache is a cache at one level of a multi-level cache; the multi-level cache contains multiple levels of cache, and each level of image processing in the multi-level image processing has a unique corresponding level of cache in the multi-level cache; the target-level cache and the first image processing are uniquely corresponding to each other; If the determination result is yes, the translated image corresponding to the first screen is determined based on the translation cache result; If the result is negative, and if the first image processing is the last level of the multi-level image processing, then the first processing result is input into the text translation model to obtain the text translation result, and the translated image corresponding to the first screen is determined based on the text translation result. If the first image processing is not the last level of image processing in the multi-level image processing, determine the first processing object corresponding to the first processing result, and perform the first image processing action on the first processing object.

14. The apparatus according to claim 13, characterized in that, The multi-level image processing includes image-aware hash value processing, text region segmentation processing, and region text recognition processing, which are performed sequentially.

15. The apparatus according to claim 14, characterized in that, The multi-level cache includes three levels: image translation cache, region translation cache, and text translation cache. Specifically, the image translation cache and the image perception hash value processing are uniquely correlated; the region translation cache and the text region segmentation processing are uniquely correlated; and the text translation cache and the region text recognition processing are uniquely correlated.

16. The apparatus according to claim 15, characterized in that, The processing unit is further configured to: After determining the translated image corresponding to the first screen based on the text translation result, the cache of each level in the multi-level cache is updated based on the translated image corresponding to the first screen and the text translation result.

17. The apparatus according to claim 16, characterized in that, The processing unit is specifically used for: The perceptual hash value of the first image and the corresponding translated image of the first image are stored in the image translation cache; the perceptual hash value of the first image is used as an index in the image translation cache to retrieve the corresponding translated image of the first image. Determine the multi-dimensional feature vector hash value of the text region, and store the multi-dimensional feature vector hash value and the text translation result in the region translation cache; The text region is the first processing result obtained when the first image processing is text region segmentation processing; the multi-dimensional feature vector hash value is used as an index to query and obtain the text translation result in the region translation cache; Determine the text hash value of the region-identified text, and store the text hash value and the corresponding text translation result in the text translation cache; The region-identified text is the first processing result obtained when the first image processing is region text recognition processing; the text hash value is used as an index to retrieve the text translation result in the text translation cache.

18. The apparatus according to any one of claims 13-17, characterized in that, The first screen is a frame displayed in the first application; The processing unit is further configured to: Obtain the user's historical operation data for the first application; The predicted translation object is determined based on the historical operation data, and the predicted translation object includes one or more of the following: predicted image, predicted text region, or predicted text. The predicted translation object is translated to obtain the predicted translation result; The predicted translation results are stored in the multi-level cache.

19. The apparatus according to any one of claims 14-17, characterized in that, When the first image processing is the image perception hash value processing, the first processing result includes the perception hash value of the first image; The processing unit is specifically used for: Reduce the first image to the first size; Convert the scaled-down first image into a grayscale image; Perform a discrete cosine transform on the grayscale image to obtain the frequency domain feature information of the grayscale image; Extract the low-frequency information component from the frequency domain feature information; The perceptual hash value of the first image is determined based on the low-frequency information components.

20. The apparatus according to any one of claims 14-17, characterized in that, When the first image processing is the text region segmentation processing, the first processing result contains at least one text region; The processing unit is specifically used for: Edge detection is performed on the first processing object to obtain the edge binary image corresponding to the first processing object; Morphological processing is performed on the binary image of the edge; Connectivity analysis is performed on the morphologically processed binary edge image to obtain multiple image regions; Extract the regional feature information of each image region from the multiple image regions; The regional feature information of each image region is input into the text region recognition model to obtain at least one text region from the multiple image regions.

21. The apparatus according to any one of claims 14-17, characterized in that, When the first image processing is the region text recognition processing, the first processing result contains at least one region-recognized text. The processing unit is specifically used for: Perform image preprocessing on the first processing object; The first processed object after image preprocessing is subjected to text line recognition; The identified text lines are segmented to obtain at least one text line image; Input the at least one text line image into the text recognition model to obtain the at least one region-recognized text.

22. The apparatus according to any one of claims 14-17, characterized in that, When the first image processing is text region segmentation processing, the first processing result includes multiple text regions; when the first image processing is region text recognition processing, the first processing result includes region recognition text corresponding to each text region in the multiple text regions. The processing unit is specifically used for: Based on the location information of the identified text in each region in the first processing result, the identified text in each region is grouped. Supplement the context information for each of the region-identified texts; the context information includes one or more of the following: the location information of the region-identified text in the first screen, the attribute information of the first screen, and other region-identified texts belonging to the same group; The region-identified texts belonging to the same group and supplemented with contextual information are batch-input into the text translation model.

23. The apparatus according to any one of claims 15-17, characterized in that, When the first image processing is image-aware hash value processing, and the image translation cache contains a translation cache result corresponding to the first processing result, the processing unit is specifically used to determine the translation cache result corresponding to the first processing result as the translated image corresponding to the first screen.

24. The apparatus according to any one of claims 14-16, characterized in that, When the first image processing is text region segmentation processing or region text recognition processing, and the target-level cache contains a translation cache result corresponding to the first processing result, the processing unit is specifically used for: Obtain the translation cache result corresponding to the first processing result; A background mask is generated in the area where the first text is located in the first image; The translated text corresponding to the translation cache result is rendered and displayed within the background mask; The translated text corresponding to the translation cache result, the background mask, and the layer where the first image is located are layer composited to obtain the translated image corresponding to the first image.

25. A cloud terminal service system, characterized in that, The system includes a cloud server subsystem, a translation processing subsystem, a caching subsystem, and an intelligent service subsystem; The cloud server subsystem is used to run the cloud terminal and to capture the first screen of the cloud terminal in real time, the first screen containing first text; The translation processing subsystem is used to determine the first image as the first processing object; to perform first image processing on the first processing object to obtain a first processing result; the first image processing is a layer in a multi-level image processing, the multi-level image processing includes two or more layers of image processing performed in a specified processing order, including image processing for processing the first image layer by layer to obtain the first text; the first image processing is the image processing that has not been performed in the multi-level image processing and is the first one in the specified processing order. The caching subsystem includes multi-level caching, and each level of image processing in the multi-level image processing has a unique corresponding level of caching in the multi-level caching. The intelligent service subsystem includes a text translation model; The translation processing subsystem is also used for: Determine whether the translation cache result corresponding to the first processing result exists in the target-level cache; the target-level cache is a cache of one level in the multi-level cache. The target-level cache and the first image processing are uniquely corresponding to each other; If the determination result is yes, the translated image corresponding to the first screen is determined based on the translation cache result; If the determination result is negative, and if the first image processing is the last level of image processing in the multi-level image processing, then the first processing result is input into the text translation model to obtain the text translation result, and the translated image corresponding to the first screen is determined based on the text translation result. If the first image processing is not the last level of image processing in the multi-level image processing, determine the first processing object corresponding to the first processing result, and perform the first image processing action on the first processing object.

Images