Translation method of short text and related device

Abstract

The application discloses a translation method of a short text and a related device. In the translation method of the short text, an electronic device can generate a context of the short text, and input the generated context and the short text into an NMT model to translate the short text from one language to another language. The translation method of the short text provided by the embodiments of the application can improve the accuracy of translating the short text by introducing the context, thereby improving the user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence (AI) and the field of machine translation (MT), and in particular to a translation method for short text and a related device. BACKGROUND

[0002] Machine translation, also known as automatic translation, is a process of converting one natural language (source language) into another natural language (target language) by using a computer. With the improvement of computer computing power and the explosive growth of data, machine translation based on deep neural networks (DNN), i.e., neural machine translation (NMT), is increasingly widely used.

[0003] With the rapid development of communication technology, short text is widely used in users' life, and users' demand for translation of short text is also increasing. How to use NMT to accurately translate short text and meet users' actual needs is a research hotspot in the field of machine translation. SUMMARY

[0004] The present application provides a translation method for short text and a related device, which can improve the accuracy of translating short text by introducing context, thereby improving user experience.

[0005] In a first aspect, a translation method for short text is provided, which includes: an electronic device displays a first image; after the electronic device receives a first operation, the electronic device identifies a short text in the first image; the electronic device generates a context of the short text; and the electronic device translates the short text by using the context to obtain a translation of the short text.

[0006] Implementing the method provided in the first aspect, the electronic device can generate the context of the short text and input the generated context and the short text into an NMT model to translate the short text from one language to another language. The accuracy of translating short text can be improved by introducing context, thereby improving user experience.

[0007] In combination with the first aspect, in an implementation, the short text refers to a text with a relatively short length, sparse content, and concise expression, which is usually no more than a certain number of characters, such as a text composed of 2-5 words or a text with no more than 160 characters. The short text usually forms a paragraph by itself, and thus has no context or limited context. The short text may, for example, include but is not limited to: dish names, road signs, tables, micro blogs, chat information, news topics, opinion reviews, mobile phone messages, or literature abstracts, etc.

[0008] In an implementation of the first aspect, the electronic device can extract text information in the first image using optical character recognition (OCR) or other text recognition technology, and identify the short text from the extracted text information according to the characteristics of the short text.

[0009] In an implementation of the first aspect, the context of the short text generated by the electronic device has the same effect as the actual context, can reflect the context of the short text, and is some factors that have an impact on the short text, such as time, place, scene, degree of formality of speech, topic, or register, etc.

[0010] In an implementation of the first aspect, the electronic device can generate the context of the short text in the following two ways:

[0011] (1) generating the context of the short text using supplementary information.

[0012] Specifically, the electronic device can obtain the first text, and determine the text in the first text that has a relevance to the short text exceeding a first value as the context of the short text.

[0013] The first text includes text in the first image that has a distance to the short text within a second value, or first explanation information of a first term corresponding to a first entity obtained by the electronic device in a knowledge base, the knowledge base contains term explanation information corresponding to a plurality of entities, the first entity is an entity in the short text, and the plurality of entities include the first entity.

[0014] In some implementations of the (1)th way, the electronic device can determine, as the context of the short text, a word in the first text that has a relevance to any one of the words in the short text exceeding the first value. In other implementations, the electronic device can determine, as the context of the short text, a word in the first text that has a relevance to each of the words in the short text exceeding the first value.

[0015] In the (1)th way, before the electronic device determines, as the context of the short text, the text in the first text that has a relevance to the short text exceeding the first value, the electronic device can also determine the relevance of a first word and a second word. The first word includes any one of the words in the first text, and the second word includes any one of the words in the short text.

[0016] Specifically, the electronic device can perform semantic encoding on the first word in the first text to obtain a first feature vector corresponding to the first word, obtain a request vector according to the first feature vector, perform semantic encoding on the second word in the short text to obtain a second feature vector corresponding to the second word, obtain a key vector according to the second feature vector, and determine the relevance of the first word and the second word according to the request vector and the key vector.

[0017] (2) Generating the context of the short text by using a generation network.

[0018] Specifically, the electronic device can train the generation network, take the short text as the input of the generation network, and determine the output of the generation network as the context of the short text.

[0019] In the second mode, the manner in which the electronic device trains the generation network can include the following two modes:

[0020] 1. The electronic device trains a sequence-to-sequence (Seq2Seq) network by taking an entity in a knowledge base as the input and taking the explanation information corresponding to the entity as the output, to obtain the generation network; the knowledge base contains explanation information corresponding to a plurality of entities.

[0021] 2. The electronic device pre-trains a language model by using unlabeled corpus, and fine-tunes the language model by using collected corpus pairs to obtain the generation network, wherein the collected corpus pairs contain entities and explanation information corresponding to the entities. The pre-training and fine-tuning of the generation network can improve the efficiency of training the generation network.

[0022] Through the above-mentioned second mode, even if some short texts in the first image have no supplementary information or lack external knowledge bases, the electronic device can still generate the context of the short text.

[0023] In combination with the first aspect, in some embodiments, the electronic device can translate the short text according to the generated context by using a neural network model, which can include, for example, a transformer model.

[0024] In some embodiments, the electronic device can first perform semantic encoding on the context to obtain a first feature vector matrix corresponding to the context, obtain a request vector matrix according to the first feature vector matrix, perform semantic encoding on the short text to obtain a second feature vector matrix corresponding to the short text, obtain a key vector matrix according to the second feature vector matrix, determine the relevance of the context and the short text according to the request vector matrix and the key vector matrix, perform encoding processing on the short text according to the relevance of the context and the short text to obtain an encoding result of the short text, and perform decoding processing on the encoding result of the short text to obtain a translation corresponding to the short text.

[0025] In some embodiments of the first aspect, the method of the first aspect can be applied to a scenario of translating a pre-stored image. Specifically, after the electronic device displays the first image, before the electronic device receives the first operation, the electronic device can further receive a second operation acting on the first image; display the first option; and wherein the first operation comprises a user operation acting on the first option.

[0026] Here, the first image can be an image stored in the electronic device and obtained by photographing. For example, before the electronic device displays the first image, the electronic device can further display a first photograph preview interface, the first photograph preview interface comprising a first preview frame, the first preview frame displaying a first image obtained by the camera of the electronic device in real time; and the electronic device saves the first image in the first preview frame.

[0027] Here, the first image can also be an image downloaded from a network or sent by another electronic device.

[0028] In some embodiments of the first aspect, the method of the first aspect can be applied to a scenario of augmented reality (AR) translation. Specifically, before the electronic device displays the first image, the method further comprises: the electronic device displaying a first photograph preview interface, the first photograph preview interface comprising a first preview frame and a first control, the first preview frame displaying an image obtained by the camera of the electronic device in real time; the electronic device receiving a user operation acting on the first control; the electronic device displaying a second photograph preview interface, the second photograph preview interface comprising a second preview frame and the first option, the second preview frame displaying the first image obtained by the camera in real time; and the first operation comprises a user operation acting on the first option.

[0029] In some embodiments of the first aspect, after the electronic device translates the short text using the context to obtain the translation of the short text, the electronic device can further display the translation of the short text on the first image. Here, the translation can be displayed above the short text, covering the short text; or the translation can be displayed below or around the short text without affecting the display of the short text. In this way, the translation result can be intuitively presented to the user, providing the user with a good translation experience.

[0030] In a second aspect, a method for generating a context of a short text is provided, the method comprising: an electronic device displaying a first image; after the electronic device receives a first operation, the electronic device identifying a short text in the first image; the electronic device training a generation network; and the electronic device inputting the short text into the generation network and determining an output of the generation network as the context of the short text.

[0031] The method provided by the second aspect can be directly used to generate the context of the short text, and can be applied to various scenarios lacking context, such as natural language processing scenarios such as intent recognition of the short text, machine reading comprehension, speech recognition, automatic summarization, automatic question answering, and the like.

[0032] In combination with the second aspect, in some embodiments, the manner in which the electronic device trains the generation network can include the following two manners:

[0033] 1. The electronic device trains a sequence-to-sequence (Seq2Seq) network by taking an entity in a knowledge base as input and taking the article explanation information corresponding to the entity as output, to obtain the generation network; the knowledge base includes article explanation information corresponding to a plurality of entities.

[0034] 2. The electronic device pre-trains a language model by using unlabeled corpus, and fine-tunes the language model by using collected corpus pairs to obtain the generation network, where the collected corpus pairs include entities and article explanation information corresponding to the entities. The pre-training and fine-tuning of the generation network can improve the efficiency of training the generation network.

[0035] In the third aspect, an electronic device is provided, which includes a display screen, a memory, and one or more processors; the display screen, the memory, and the one or more processors are coupled together; the memory is configured to store computer program code including computer instructions; and the one or more processors are configured to invoke the computer instructions to cause the electronic device to perform the following operations: displaying a first image; after receiving a first operation, identifying a short text in the first image; generating a context of the short text; and using the context to translate the short text to obtain a translation of the short text.

[0036] Here, the short text can refer to the related description of the first aspect.

[0037] In combination with the third aspect, in an embodiment, the electronic device can extract text information in the first image by using optical character recognition (OCR) or other text recognition technologies, and identify the short text from the extracted text information according to the characteristics of the short text.

[0038] In combination with the third aspect, in an embodiment, the context of the short text generated by the electronic device has the same effect as the actual context, and can reflect the context of the short text, which is some factors that have an impact on the short text, such as time, place, scene, degree of formality of speech, topic, or register, and the like.

[0039] In combination with the third aspect, in an embodiment, the electronic device can generate the context of the short text in the following two manners:

[0040] (1) using supplementary information to generate the context of the short text.

[0041] Specifically, the electronic device can obtain the first text, and determine the text in the first text that has a relevance to the short text exceeding a first value as the context of the short text.

[0042] The first text includes: text in the first image that has a distance to the short text within a second value, or first article explanation information corresponding to a first entity obtained by the electronic device in a knowledge base; the knowledge base contains article explanation information corresponding to a plurality of entities respectively, the first entity is an entity in the short text, and the plurality of entities include the first entity.

[0043] In some embodiments of the first mode, the electronic device can determine a word in the first text that has a relevance to any word in the short text exceeding the first value as the context of the short text. In other embodiments, the electronic device can determine a word in the first text that has a relevance to each word in the short text exceeding the first value as the context of the short text.

[0044] In the first mode, before the electronic device determines the text in the first text that has a relevance to the short text exceeding the first value as the context of the short text, the electronic device can also determine the relevance of the first word and the second word. The first word includes any word in the first text, and the second word includes any word in the short text.

[0045] Specifically, the electronic device can perform semantic coding on the first word in the first text to obtain a first feature vector corresponding to the first word; obtain a request vector according to the first feature vector; perform semantic coding on the second word in the short text to obtain a second feature vector corresponding to the second word; obtain a key vector according to the second feature vector; and determine the relevance of the first word and the second word according to the request vector and the key vector.

[0046] (2) using a generation network to generate the context of the short text.

[0047] Specifically, the electronic device can train the generation network, and determine the output of the generation network as the context of the short text by taking the short text as the input of the generation network.

[0048] In the second mode, the electronic device can train the generation network in the following two ways:

[0049] 1. The electronic device trains a sequence-to-sequence (Seq2Seq) network by taking an entity in a knowledge base as input and taking article explanation information corresponding to the entity as output to obtain the generation network; the knowledge base contains article explanation information corresponding to a plurality of entities respectively.

[0050] 2. The electronic device pre-trains a language model using unlabeled corpus. The language model is then fine-tuned using collected corpus pairs, which contain entities and their corresponding term explanations, to produce a generative network. Pre-training and fine-tuning the generative network improves training efficiency.

[0051] Through the above-mentioned method (2), even if there are some short texts in the first image without supplementary information, or lacking an external knowledge base, the electronic device can generate the context of the short texts.

[0052] In conjunction with the third aspect, in some embodiments, the electronic device may utilize a neural network model to translate a short text based on the generated context. The neural network model may include, for example, a transformer model.

[0053] In some embodiments, the electronic device may first semantically encode the context to obtain a first eigenvector matrix corresponding to the context; obtain a request vector matrix based on the first eigenvector matrix; semantically encode the short text to obtain a second eigenvector matrix corresponding to the short text; obtain a key vector matrix based on the second eigenvector matrix; determine the correlation between the context and the short text based on the request vector matrix and the key vector matrix; encode the short text based on the correlation between the context and the short text to obtain an encoding result of the short text; decode the encoding result of the short text to obtain a translation corresponding to the short text.

[0054] In combination with the third aspect, in some embodiments, after the electronic device displays the first image and before receiving the first operation, the electronic device may also receive a second operation acting on the first image; display the first option; wherein the first operation includes: a user operation acting on the first option.

[0055] Here, the first image may be an image captured and stored in the electronic device. For example, before displaying the first image, the electronic device may also display a first shooting preview interface, the first shooting preview interface including a first preview frame, which displays the first image captured in real time by the electronic device's camera; the electronic device stores the first image in the first preview frame.

[0056] Here, the first image may also be an image downloaded from a network or sent from other electronic devices.

[0057] In some embodiments, before the electronic device displays the first image, the electronic device can display a first photograph preview interface, the first photograph preview interface including a first preview frame and a first control, the first preview frame displaying an image captured by the camera of the electronic device in real time. The electronic device receives a user operation on the first control; and the electronic device displays a second photograph preview interface, the second photograph preview interface including a second preview frame and a first option, the second preview frame displaying the first image captured by the camera in real time. The first operation includes a user operation on the first option.

[0058] In some embodiments, after the electronic device translates the short text using the context to obtain a translation of the short text, the electronic device can further display the translation of the short text on the first image. Here, the translation can be displayed above the short text, covering the short text; or the translation can be displayed below or around the short text, without affecting the display of the short text. In this way, the translation result can be intuitively presented to the user, providing the user with a good translation experience.

[0059] In a fourth aspect, an electronic device is provided, including a memory, one or more processors, a display screen, the memory and the one or more processors being coupled, the memory being configured to store computer program code including computer instructions, and the one or more processors being configured to invoke the computer instructions to cause the electronic device to perform the method described in the second aspect or any one of the embodiments of the second aspect.

[0060] In a fifth aspect, a computer-readable storage medium is provided, including instructions, when the instructions are executed on an electronic device, causing the electronic device to perform the method described in the first aspect or any one of the embodiments of the first aspect.

[0061] In a sixth aspect, a computer program product is provided, when the computer program product is executed on a computer, causing the computer to perform the method described in the first aspect or any one of the embodiments of the first aspect.

[0062] In a seventh aspect, a computer-readable storage medium is provided, including instructions, when the instructions are executed on an electronic device, causing the electronic device to perform the method described in the second aspect or any one of the embodiments of the second aspect.

[0063] In an eighth aspect, a computer program product is provided, when the computer program product is executed on a computer, causing the computer to perform the method described in the second aspect or any one of the embodiments of the second aspect.

[0064] By implementing the short text translation method provided in this application, an electronic device can generate a context for the short text and input the generated context and the short text into an NMT model to translate the short text from one language to another. Introducing context can improve the accuracy of short text translation, thereby enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 An example of short text translation provided in an embodiment of the present application;

[0066] Figure 2A A hardware structure diagram of an electronic device provided in an embodiment of the present application;

[0067] Figure 2B The software architecture of the electronic device provided in the embodiments of the present application;

[0068] Figures 3A-3E A set of user interfaces implemented on an electronic device in the photo translation scenario exemplified in this application;

[0069] Figures 4A-4D A set of user interfaces implemented on an electronic device in the AR translation scenario exemplified in this application;

[0070] Figure 5 A flowchart of a short text translation method provided in an embodiment of the present application;

[0071] Figure 6 An example of attention between supplementary information and short text provided in an embodiment of the present application;

[0072] Figure 7 A schematic diagram of the structure of the Seq2Seq network provided in an embodiment of the present application;

[0073] Figure 8A and Figure 8B A schematic diagram of the structure of the transformer model provided in the embodiment of the present application. DETAILED DESCRIPTION

[0074] The following is a clear and detailed description of the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0075] Hereinafter, the terms "first", "second", "third", "fourth", "fifth", "sixth", "seventh" and "eighth" are used only for descriptive purposes and should not be construed as implying or suggesting relative importance or an implied direction about the number of the technical features indicated. Thus, the features defined with "first", "second", "third", "fourth", "fifth", "sixth", "seventh" and "eighth" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0076] The term "user interface (UI)" in the embodiments of the present application is a medium interface for interaction and information exchange between an application program or an operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is source code written in a specific computer language such as Java, extensible markup language (XML), etc., and the interface source code is parsed, rendered and finally presented as content recognizable by the user on the electronic device. The commonly used form of user interface is graphic user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be visual interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, etc. displayed in the display screen of the electronic device.

[0077] NMT is widely used in long text translation at the paragraph level. By inputting the context of the text to be translated into an NMT model (also referred to as an NMT engine) together, the accuracy of translation can be improved. The context of the text to be translated refers to the text before or after the text to be translated in the same paragraph. Other texts that do not belong to the same paragraph as the text to be translated, even if they are very close, do not belong to the context of the text to be translated.

[0078] However, for short texts without context, the accuracy of the translation result obtained by using the NMT model is low. For example, as shown in Figure 1 , when using an NMT model (such as a transformer) to translate the dish name "battered whiting", "battered" means to suffer heavy injuries, and "battered whiting" will be translated into a battered cod. However, in a dining scenario, "battered" should mean "fried", and the dish name "battered whiting" should be translated as "fried cod".

[0079] The accuracy of NMT and the context of the text to be translated are closely related. Since short text contains less content and lacks context, the current NMT can only extract a small amount of effective content (for example Figure 1 the dashed portion in FIG. 1) under such a small amount of information, resulting in low translation accuracy.

[0080] To solve the problem of inaccurate translation of short text, the embodiments of the present application provide a translation method for short text and related devices. In the translation method for short text, the electronic device can generate the context of the short text and input the generated context and the short text into the NMT model to translate the short text from one language to another language. By introducing the context, the translation method for short text provided by the embodiments of the present application can improve the accuracy of translating short text and improve the user experience.

[0081] In the embodiments of the present application, short text refers to text that is short in length, sparse in content, and concise in expression, usually not more than a certain number of characters, for example, text composed of 2-5 words or text within 160 characters. Short text usually forms a paragraph by itself, so there is no context or limited context. Short text exists in large quantities in users' lives, such as dish names, road signs, tables, microblogs, chat information, news topics, opinion reviews, mobile phone messages, literature abstracts, etc. Through the translation method for short text provided by the present application, users can obtain accurate translation results for short text in various translation scenarios, such as photo translation, augmented reality (AR) translation, screenshot translation, etc.

[0082] The context of the short text generated by the electronic device in the embodiments of the present application does not refer to the text before or after the short text in the same paragraph, but refers to the text that has the same effect as the actual context. The context generated by the electronic device can reflect the context of the short text, which is some factors that affect the short text, such as time, place, scene, degree of formality of speech, topic or register, etc. Here, the specific implementation of the electronic device generating the context of the short text can refer to the related description in the subsequent method embodiments, which will not be described here.

[0083] The NMT model mentioned in the embodiments of the present application is based on an encoder-decoder structure, and the specific type of the NMT model is not limited in the present application. The NMT model can be, for example, a transformer model and other encoder-decoder structure models in future development, etc. The specific implementation of the NMT model in translating short text will be described in detail in the subsequent method embodiments, which will not be described here.

[0084] To more clearly describe the short text translation method provided by the embodiments of the present application, first, an electronic device for performing the short text translation method is introduced.

[0085] The electronic device of the embodiments of the present application includes, but is not limited to, a smart phone, a tablet computer, a personal digital assistant (PDA), a wearable electronic device with wireless communication function (such as a smart watch, smart glasses), an augmented reality (AR) device, a virtual reality (VR) device, etc. The exemplary embodiments of the electronic device include, but are not limited to, a portable electronic device with Linux or other operating system. The electronic device can also be other portable electronic devices, such as a laptop computer, etc. It should also be understood that in some other embodiments, the electronic device can also not be a portable electronic device, but a car computer, a desktop computer, etc.

[0086] Figure 2A The structure of the electronic device 100 provided by the embodiments of the present application is shown.

[0087] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0088] ​It can be understood that the structural schematic of the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than those shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0089] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0090] The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0091] The memory in the processor 110 can also be configured to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instructions or data again, it can be directly called from the memory. Avoiding repeated access, reducing the waiting time of the processor 110, thus improving the efficiency of the system.

[0092] The wireless communication function of the electronic device 100 can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0093] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antenna. For example: the antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0094] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transfer the same to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor, and radiate the same as electromagnetic waves through the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the same device as at least part of the modules of the processor 110.

[0095] The modem processor can include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the microphone 170B, etc.), or displays an image or a video through the display screen 194. In some embodiments, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 110, and disposed in the same device as the mobile communication module 150 or other functional modules.

[0096] The wireless communication module 160 can provide a solution for wireless communication including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives an electromagnetic wave via the antenna 2, demodulates and filters the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 can also receive a signal to be transmitted from the processor 110, frequency-modulate it, amplify it, and radiate it as an electromagnetic wave via the antenna 2.

[0097] In some embodiments, the antenna 1 and the mobile communication module 150 of the electronic device 100 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, so that the electronic device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include a global positioning system (GPS), a global navigation satellite system (GLONASS), a beidu navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite based augmentation systems (SBAS).

[0098] The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0099] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), or the like. In some embodiments, the electronic device 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0100] In the embodiments of the present application, the user interface displayed by the display screen 194 can refer to the related description in the subsequent method embodiments, which will not be described here in detail.

[0101] The electronic device 100 can implement the photographing function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor.

[0102] The ISP is configured to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize the algorithm for the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be arranged in the camera 193.

[0103] The camera 193 is configured to capture still images or videos. Objects generate optical images through lenses and project them onto photosensitive elements. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard RGB, YUV, or the like format image signal. In some embodiments, the electronic device 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0104] After the camera 193 captures a still image or a video, the image or the video can be stored in a read-only memory (ROM) or a random access memory (RAM) of the memory 121. When the image or the video is stored in the ROM, the electronic device 100 performs a photograph preview operation; when the image or the video is stored in the RAM, the electronic device 100 performs a photograph and save operation.

[0105] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, and the like.

[0106] The NPU is a neural-network (NN) computing processor, which can process input information quickly by referring to a biological neural network structure, for example, by referring to a transmission mode between human brain neurons, and can also constantly self-learn. Through the NPU, intelligent cognition and other applications of the electronic device 100 can be realized, for example: text translation, image recognition, face recognition, speech recognition, text understanding, and the like.

[0107] In the embodiment of the present application, the NPU can be used to generate a context of a short text to be translated.

[0108] In a possible manner, the NPU can extract, from supplementary information of the short text to be translated, part of information that has a relevance to the short text exceeding a certain value, and use the part of information as context information of the short text. The supplementary information is information in an image stored by the electronic device 100, and a distance between the information and the short text is within a threshold. The image can be an image stored in the RAM, for example, an image obtained by the camera 193 when performing a photograph preview, a screenshot preview, and the like; the image can also be an image stored in the ROM, for example, a photo obtained by the camera 193 after performing a photograph operation, an image obtained by the electronic device 100 through screenshot, an image downloaded from a network by the mobile communication module 150 or the wireless communication module 160, and the like. Here, the specific implementation of the NPU generating the context of the short text according to the supplementary information of the short text can refer to the detailed description of subsequent method embodiments.

[0109] In another possible manner, the NPU can perform feature expansion on the short text to be translated and an external knowledge base to obtain supplementary information of the short text, and then extract part of the information having a relevance to the short text exceeding a certain value from the supplementary information as the context of the short text. The specific implementation of the NPU performing feature expansion on the short text can refer to the detailed description in subsequent method embodiments. The external knowledge base can be downloaded from the network by the mobile communication module 150 or the wireless communication module 160.

[0110] In yet another possible manner, the NPU can use a trained neural network, such as a sequence to sequence (Seq2Seq) model, a generative pre-training model, etc., to generate the context of the short text to be translated. Here, the training method of the neural network, and the specific implementation manner of generating the context of the short text using the neural network can refer to the related description in subsequent method embodiments. The trained neural network can be downloaded from the network by the mobile communication module 150 or the wireless communication module 160, or can be obtained by self-training of the electronic device 100, which is not limited here.

[0111] After the electronic device 100 generates the context of the short text to be translated, the short text and the context can be input into an NMT model, such as a transformer model / other encoder-decoder structure model in future development, etc., to translate the short text into another language.

[0112] The internal memory 121 can include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs).

[0113] The random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, such as the fifth generation of DDR SDRAM commonly referred to as DDR5 SDRAM), and the like; the non-volatile memory can include a magnetic disk storage device, a flash memory.

[0114] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of an operating system or other programs running, and can also be used to store data of users and application programs, and the like.

[0115] The non-volatile memory can also store executable programs and store data of users and application programs, and the like, and can be loaded in advance into the random access memory for direct reading and writing by the processor 110.

[0116] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. Embodiments of the present application take the Android system with a layered architecture as an example to exemplarily illustrate the software structure of the electronic device 100.

[0117] Figure 2B is a software structure block diagram of the electronic device 100 of embodiments of the present application.

[0118] The layered architecture divides the software into several layers, each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and the system library, and the kernel layer.

[0119] The application layer can include a series of application packages.

[0120] As shown in Figure 2B , the application package can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, and the like.

[0121] The application framework layer provides an application programming interface (API) and programming framework for applications of the application layer. The application framework layer includes some pre-defined functions.

[0122] As shown in Figure 2B the application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.

[0123] The window manager is used to manage window programs. The window manager can acquire the size of the display screen, determine whether there is a status bar, lock the screen, and intercept the screen, etc.

[0124] The content provider is used to store and acquire data, and make the data accessible to the application program. The data can include videos, images, audios, dialed and received calls, browsing history and bookmarks, phone books, etc.

[0125] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build an application program. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0126] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of the call state (including call connection, call hang-up, etc.).

[0127] The resource manager provides various resources for the application program, such as localized strings, icons, pictures, layout files, video files, etc.

[0128] The notification manager enables the application program to display notification information in the status bar, which can be used to convey a type of message that can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform the completion of the download, message reminders, etc. The notification manager can also be a notification in the form of a chart or a scroll bar text appearing in the top status bar of the system, such as a notification of an application running in the background, and can also be a notification in the form of a dialogue window appearing on the screen. For example, prompting text information in the status bar, issuing a prompt sound, the electronic device vibrating, the indicator light flashing, etc.

[0129] The Android runtime includes a core library and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0130] The core library includes two parts: one part is the function function required by the java language to call, and the other part is the core library of Android.

[0131] The application program layer and the application framework layer run in a virtual machine. The virtual machine executes the java files of the application program layer and the application framework layer as binary files. The virtual machine is used to perform functions such as management of the object life cycle, stack management, thread management, management of security and exceptions, and garbage collection.

[0132] The system library can include a plurality of functional modules. For example, a surface manager, media libraries, a three-dimensional graphics processing library (for example, OpenGL ES), a 2D graphics engine (for example, SGL), and the like.

[0133] The surface manager is used to manage the display subsystem and provides fusion of 2D and 3D layers for a plurality of application programs.

[0134] The media library supports playback and recording of a plurality of commonly used audio, video formats, and static image files. The media library can support a plurality of audio and video encoding formats, for example, MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, and the like.

[0135] The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing, and the like.

[0136] The 2D graphics engine is a drawing engine for 2D drawing.

[0137] The kernel layer is a layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0138] It is emphasized that, Figure 2B only for illustrative purposes; the software structure of the electronic device 100 provided by the embodiments of the present application can also use other software architectures, such as the software architecture of a Linux or other operating system.

[0139] Based on the structure of the electronic device 100 shown in Figure 2A and Figure 2B the following describes in detail the short text translation method provided by the embodiments of the present application.

[0140] The short text translation method provided by the embodiments of the present application can be applied to various scenarios with translation needs.

[0141] For example, in a photographing scenario, the electronic device 100 can translate a short text in a photographed photo.

[0142] For another example, in a reading scenario, the electronic device 100 can translate a short text displayed on a display screen.

[0143] For another example, the electronic device 100 may translate short text in an image captured as a screenshot, downloaded from the Internet, or obtained in other ways.

[0144] For another example, in an AR scenario, the electronic device 100 can translate the short text in the currently acquired image.

[0145] The electronic device 100 is not limited to static images and can also translate short text in videos or dynamic images, which is not limited in the present embodiment. The short text translation method provided in the following embodiments can also be applied to other scenarios, which is not limited here.

[0146] The following describes a short text translation method provided by an embodiment of the present application in conjunction with an exemplary user interface implemented on the electronic device 100 .

[0147] Figures 3A-3E This is a set of user interfaces implemented on the electronic device 100 in the photo translation scenario exemplified in this application. Figures 3A-3E The illustrated set of user interfaces may be provided by a camera application. A camera application is an image capture application on electronic devices such as smartphones and tablets that allows users to take pictures, videos, etc. This application does not impose any restrictions on the name of the application.

[0148] Figure 3A The figure shows a user interface 31 provided by the camera application. This user interface 31 may be displayed in response to the electronic device 100 detecting a touch operation (e.g., a click operation) on the camera icon on the home screen (not shown) of the electronic device 100. Without limitation, the user may also open this user interface 31 in other applications. For example, the user may open this user interface 31 by inputting a left swipe gesture on the display screen on the lock screen.

[0149] like Figure 3A As shown, the user interface 31 may include: an area 301, a shooting mode list 302, and a series of controls 303-311.

[0150] Area 301 can be referred to as a preview frame 301. Preview frame 301 can be used to display images captured in real time by camera 193. Electronic device 100 can refresh the displayed content in real time to facilitate user preview of the image currently captured by camera 193. The content captured in real time by camera 193, i.e., the content in preview frame 301, will be cached in the RAM of the electronic device. Here, camera 193 can be a rear-facing camera or a front-facing camera. Figure 3A The preview frame 301 displays a menu.

[0151] One or more shooting mode options can be displayed in the shooting mode list 302. The one or more shooting mode options can include a night scene mode option, a portrait mode option, a photo mode option, a video mode option, and more shooting mode options, and the like. The one or more shooting mode options can be in the form of text information, icons, or other forms of interactive elements (IEs) on the interface. In some embodiments, the electronic device 100 can default to select the photo mode option, Figure 3A The arrow below the photo mode option in the shooting mode list 302 can indicate that the photo mode option has been selected.

[0152] The control 303 can be used to listen to a user operation that triggers shooting (photographing or video recording). The electronic device can detect a user operation (e.g., a click operation on the control 303) acting on the control 303 and save the image in the preview frame 301 to the ROM in response to the operation. The saved image can be a picture or a video. In addition, the electronic device 100 can also display a thumbnail of the saved image in the control 305. That is, the user can click the control 303 to trigger shooting. The control 303 can be a button or other forms of controls. In this application, the control 303 can be referred to as a shooting control.

[0153] The control 304 can be used to listen to a user operation that triggers switching the camera. The electronic device 100 can detect a user operation acting on the control 304 and switch the camera in response to the operation, such as switching the rear camera to the front camera or switching the front camera to the rear camera.

[0154] The control 305 can be used to listen to a user operation that triggers opening the “gallery”. The electronic device 100 can detect a user operation (e.g., a click operation on the control 305) acting on the control 305 and display a user interface of the “gallery” in response to the operation, in which the user interface can display pictures saved by the electronic device 100. Here, the “gallery” is an application program for managing pictures on a smart phone, a tablet computer, or the like, which can also be referred to as an “album”. The name of the application program is not limited in this embodiment. The “gallery” can support various operations of the user on the pictures stored on the electronic device, such as browsing, editing, deleting, selecting, and the like.

[0155] The controls 303-311 are controls for controlling the current shooting effect. The control 306 is used to start the AR real-time translation function, and the function of the control will be described in detail later. The controls 307-311 are respectively used to control the flash, start AI photographing, set a color filter, open a shooting setting item, and adjust the focal length.

[0156] As Figure 3AAs shown, the electronic device 100 can detect a user operation (e.g., a click operation, a touch operation, etc.) on the control 303, and save the display content in the preview box 301 in response to the user operation. The display content can be stored in ROM. Afterwards, the control 305 can display the display content in the saved preview box 301.

[0157] Figure 3B The user interface 32 provided by the gallery in the electronic device 100 is shown. The user interface 32 may be a user interface provided by the electronic device 100 in response to the Figure 3A The interface displayed in response to an operation (e.g., a click operation, a touch operation, etc.) detected on the control 305 shown may also be a user interface displayed in response to a touch operation (e.g., a click operation) on an icon in a gallery on the home screen (not shown) of the electronic device 100 and selection of the image 312. Without limitation, the electronic device 100 may also display the user interface 32 in other scenarios, which are not limited here.

[0158] like Figure 3B As shown, the user interface 32 displays a picture 312 . The original picture 312 may be stored in the ROM of the electronic device 100 .

[0159] like Figure 3B As shown, the electronic device 100 can detect a user operation on the image 312 in the user interface 32, such as a two-finger press operation, a two-finger long press operation, etc., and display the image 312 in response to the user operation. Figure 3C Menu bar 313 is displayed in the menu bar 313. One or more options may include: a translation option 313a, a text recognition option 313b, a delete option 313c, a copy option 313d, and a share option 313e. These one or more options may be displayed on the interface as text messages, icons, or other interactive elements. These one or more options can be used to perform corresponding operations on the image 312, such as translating the text in the image, recognizing the text in the image, deleting the image, copying the image, sharing the image, etc.

[0160] The electronic device 100 can detect a user operation (such as a click operation, a touch operation, etc.) on the control 313a, and in response to the user operation, translate the short text in the image 312 using the translation method provided in the embodiment of the present application and display the translation result. Figure 3D The following contents are: return control 314, language bar 315, more controls 316. Among them:

[0161] The return control 314 can be used to return to the previous page, that is, to return to the previous page. Figure 3B The user interface 32 is shown. More controls 316 can be used to display more operational controls.

[0162] The language bar 315 is used to indicate the language before translation and the language after translation of the short text in the current page. The language bar 315 displays: a language identifier 315a before translation, and a language identifier 315b after translation. The language identifiers 315a and 315b can be expressed as text information, icons or other forms of interactive elements. The language identifiers 315a and 315b can be set in advance by the user or set by default by the electronic device 100. For example, the user can click on the language identifier 315a to select the language into which the text needs to be translated, and click on the language identifier 315b to select the language into which the text is translated. Figure 3C The language identifier 315a shown in the figure indicates that the electronic device 100 will automatically detect the language type of the short text in the picture 312, and the language identifier 315b indicates that the electronic device 100 will translate the short text in the picture 312 into Chinese. The original short text in the picture 312 can also be called the original text.

[0163] In response to Figure 3C As shown, the user operates on the control 313 a , and the electronic device 100 translates the short text in the picture 312 into Chinese corresponding to the language identifier 315 a .

[0164] refer to Figure 3E , Figure 3E FIG. 3 shows a user interface 32 displayed after the electronic device 100 translates the short text in the picture 312 into Chinese. Figure 3E As shown, the user interface 32 displays the translation of the short text in the picture 312, such as Figure 3E The text 317 in the image 312 is translated as "Fried Cod".

[0165] In some embodiments, the translated text can be displayed over the original text in the image 312, covering the original text, such as Figure 3E As shown. From the user's perspective, only the translated text can be seen in the picture 312 of the user interface 32. In some embodiments, the user can switch between the translated text and the original text by clicking on the picture 312, for example, Figure 3E In the user interface 32 shown, when the electronic device 100 detects a click operation, it can stop displaying the translation 317 and directly display the image 312. If the click operation is detected again, the translation 317 can be displayed again on top of the image 312. This allows switching between the original text and the translation, making it easier for the user to view.

[0166] In other embodiments, the translated text can be displayed below or around the original text without affecting the display of the original text. From the user's perspective, the translated text and the original text can be seen simultaneously in the image 312 of the user interface 32. This makes it convenient for the user to directly compare the original text and the translated text. Similarly, the user can also display or not display the translated text by clicking on the image 312.

[0167] Understandable, Figure 3D The user interface 32 shown is optional, and the electronic device 100 responds to Figure 3C When the user operation on the control 313a shown in FIG. 1 is performed at a faster speed, the electronic device 100 can directly Figure 3C Jump to display Figure 3E In this way, the user can intuitively feel that the electronic device 100 quickly responds to the user operation and translates the short text in the picture 312.

[0168] exist Figures 3A-3E In the photo translation scenario shown, the specific implementation process of the electronic device 100 translating the text in the picture 312 can be referred to the detailed description of the subsequent method embodiments and will not be repeated here.

[0169] Figures 4A-4D A set of user interfaces implemented on the electronic device 100 in the AR translation scenario exemplified in this application. Figures 4A-4D The illustrated set of user interfaces may be provided by a camera application.

[0170] Figure 4A 4 shows a user interface 41 provided by a camera application. Figure 4A As shown, the user interface 41 includes: an area 301, a shooting mode list 302 and a series of controls 303-311. Among them, the area 301, the shooting mode list 302 and a series of controls 303-311 can refer to Figure 3A The relevant descriptions in the user interface 31 are not repeated here.

[0171] like Figure 4A As shown, the electronic device 100 can detect a user operation (such as a click operation, a touch operation, etc.) acting on the control 306. In response to the user operation, the electronic device 100 can start AR real-time translation. The control 306 can be called an AR translation control.

[0172] like Figure 4B As shown, the electronic device 100 can display the user interface 42 in response to a user operation on the control 306. The user interface 42 includes: a preview box 301 and a menu bar 401.

[0173] Preview box 301 and Figure 3AThe preview box 301 in the figure is the same as that in the image captured by the camera 193 in real time. The electronic device 100 can refresh the displayed content in real time to facilitate the user to preview the image currently captured by the camera 193. The content captured by the camera 193 in real time, that is, the content in the preview box 301 will be cached in the RAM of the electronic device. Here, the camera 193 can be a rear camera or a front camera. The preview box 301 in the figure shows an entry card. It is understandable that the electronic device 100 displays the user interface 41 and the user interface 42 at different time points, and the content displayed in the preview box 301 of the two is refreshed.

[0174] In the embodiments of this application, Figure 4B The preview box 301 in the, and, Figure 3A The content captured in real time by the camera 193 and displayed in the preview box 301 may be the original image captured by the camera 193 or an image obtained by cropping the original image, and there is no limitation here. Figure 4B The preview box 301 in the, and, Figure 3A The sizes of the preview boxes 301 in the example can be the same or different, and this embodiment of the application does not limit this.

[0175] One or more options are displayed in the menu bar 401, which may include: a scan option 401a, a translation option 401b, a shopping option 401c, a calorie identification option 401d, and an object identification option 401e. These one or more options may be displayed as text information, icons, or other forms of interactive elements on the interface. These one or more options may be used to perform corresponding operations on the displayed content in the preview box 301, such as scanning a QR code in the displayed content, translating text in the displayed content, jumping to purchase items in the displayed content, calculating the calories of objects in the displayed content, identifying objects contained in the displayed content, and the like. Figure 4B As shown, the display status of the Scan option 401a in the menu bar 401 indicates that the option 401a is currently selected. The display status can be, for example, the shadow of the option 401a, such as the font of the option 401a becomes larger, the option 401a is framed, the option 401a is underlined, the color is darkened, etc.

[0176] like Figure 4B As shown, the electronic device 100 can detect a user operation (such as a click operation, a touch operation, etc.) acting on the scan option 401a. In response to the user operation, the electronic device 100 uses the translation method provided in the embodiment of the present application to translate the short text in the preview box 301 and displays it. Figure 4C The following contents: control 402, language bar 403, more controls 404. Among them, language bar 403 and more controls 404 can refer to Figure 3DThe language bar 315 and the more controls 316 in the image gallery are shown in FIG. The control 402 can be used to monitor the operation of opening the gallery. Figure 4C The display state of the translation control 401b indicates that the translation control 401b is selected.

[0177] In response to Figure 4B As shown, the user operates on the translation option 401 b , and the electronic device 100 translates the short text in the preview box 301 into Chinese corresponding to the language identifier 403 b .

[0178] refer to Figure 4D , Figure 4D FIG4 shows a user interface 42 displayed after the electronic device 100 translates the short text in the preview box 301 into Chinese. Figure 4D As shown, the user interface 42 displays the translation of the short text in the preview box 301, such as Figure 4D The text in the preview box 301 is translated as "For Agency Use Only".

[0179] and Figure 3E similar, Figure 4D The translation obtained after the electronic device 100 translates the original text in the preview box 301 can be displayed on top of the original text in the preview box 301, covering the original text; or it can be displayed below or around the original text without affecting the display of the original text.

[0180] Understandable, Figure 4C The user interface 42 shown is optional, and the electronic device 100 responds to Figure 4B When the user operation on the translation option 401b shown in FIG. 1 is fast, the electronic device 100 can directly translate the selected item 401b. Figure 4B Jump to display Figure 4D In this way, the user can intuitively feel that the electronic device 100 quickly responds to the user operation and translates the short text displayed in the preview box 301.

[0181] exist Figures 4A-4D In the AR translation scenario shown, the specific implementation process of the electronic device 100 translating the text in the preview box 301 can be referred to the detailed description of the subsequent method embodiments and will not be repeated here.

[0182] Not limited to the above Figures 3A-3E The photo translation scene shown, Figures 4A-4D In the AR translation scenario shown, the method provided in the embodiment of the present application can also be applied to other scenarios, such as reading scenarios, screenshot scenarios, etc.

[0183] refer to Figure 5 , Figure 5The flow of the short text translation method provided in the embodiment of the present application is exemplified.

[0184] like Figure 5 As shown, the method may include the following steps:

[0185] S101, the electronic device 100 displays a first image.

[0186] In some embodiments, the first image may be an image stored in the ROM of the electronic device 100, including but not limited to: an image captured and saved by the electronic device 100 through a camera, such as Figure 3B The picture 312 in the user interface 32 shown, the image downloaded from the network by the electronic device 100, or the image shared by other devices, etc. Here, the electronic device 100 can display the first image on the display screen in response to the operation of opening the gallery and selecting the image, which can be referred to as Figure 3B The electronic device 100 may also display the first image on the display screen in response to other operations, which is not limited here. Here, when the first image is an image captured and saved by the electronic device 100 through a camera, examples of the first image may include Figure 3A The image in the preview box 301 of the user interface 31 is shown, Figure 3A The user interface 31 shown can be called a first shooting preview interface. Figure 3A The preview box 301 in the image may be referred to as a first preview box.

[0187] In other embodiments, the first image may also be an image stored in the RAM of the electronic device 100, including but not limited to: an image captured in real time by the camera 193 in the preview frame of the shooting scene, such as Figure 4B The image in the preview box 301 in the user interface 42 is shown. It is understandable that in the shooting scene, the first image is refreshed in real time. Here, the electronic device 100 can display the image on the display screen in response to the operation of opening the camera application. Figure 4A The user interface 41 shown then responds to the Figure 4A User action on control 306 in displays Figure 4B The user interface 42 shown, that is, the display of the first image, can refer to the above description of the UI embodiment. Here, Figure 4A The user interface 41 shown may be referred to as a first shooting preview interface. Figure 4A The control 306 in may be referred to as a first control, Figure 4B The user interface 42 shown may be referred to as a second shooting preview interface. Figure 4B The preview box 301 in the image may be referred to as a second preview box.

[0188] In some embodiments, the first image can also be an image that is not stored in the electronic device 100 but is directly drawn by the electronic device 100, such as the screen content directly drawn on the display screen when the electronic device 100 runs a reading application or a news application.

[0189] The first image can be a static image, such as Figure 3B The first image can also be a dynamic image, such as Figure 4A The first image can also be a dynamic image, such as

[0190] S102, the electronic device 100 receives a first operation for triggering translation.

[0191] The first operation can be an operation on the first image. For example, it can be Figure 3B and Figure 3C The double-finger pressing operation or the double-finger long-pressing operation on the picture 312 and the user operation (such as a click operation, a touch operation, etc.) on the translation option 313a can be referred to as a second operation on the first image, and the translation option 313a can be referred to as a first option. Figure 3B and Figure 3C The double-finger pressing operation or the double-finger long-pressing operation on the picture 312 and the user operation (such as a click operation, a touch operation, etc.) on the translation option 313a can be referred to as a second operation on the first image, and the translation option 313a can be referred to as a first option.

[0192] The first operation can also be other operations, for example, it can be Figure 4A The user operation (such as a click operation, a touch operation, etc.) detected on the translation option 401b in the user interface 42 shown in FIG. 4B after the AR translation control 306 in the user interface 41 shown in FIG. 4A is displayed. Figure 4B The user operation (such as a click operation, a touch operation, etc.) detected on the translation option 401b in the user interface 42 shown in FIG. 4B after the AR translation control 306 in the user interface 41 shown in FIG. 4A is displayed.

[0193] The first operation is not limited to the above examples, and can also be implemented in other forms, such as a voice instruction, a shaking gesture, etc., which are not limited in the present application.

[0194] S103-S105, the electronic device 100 translates the text information in the first image.

[0195] After the electronic device 100 receives the first operation for triggering translation in S102, it will respond to the first operation and translate the text information in the first image from one language to another language. The language before translation can be referred to as the first language, and the language after translation can be referred to as the second language.

[0196] In some embodiments, the first language and the second language can be set by the user autonomously.

[0197] The user can set the first language and the second language after inputting the first operation.

[0198] For example, referring to Figure 3D , after detecting the first operation (i.e. the user operation on the translation option 313a in Figure 3C , the electronic device 100 can display the language bar 315. After the user clicks the pre-translation language identifier 315a in the language bar 315, the electronic device 100 can display a plurality of language options, from which the user can select one as the first language. Similarly, the user can select one as the second language from the plurality of language options by clicking the post-translation language identifier 315b.

[0199] For another example, referring to Figure 4C , after detecting the first operation (i.e. the user operation on the translation option 401b in Figure 4B , the electronic device 100 can display the language bar 403. The user can set the first language and the second language through the language bar 403.

[0200] The user can also set the first language and the second language in advance. For example, the user can select the first language and the second language in advance in a page provided by the setting application of the electronic device 100.

[0201] In other embodiments, the first language and the second language can also be set by default by the electronic device 100. For example, the electronic device 100 can set the first language as English and the second language as Chinese by default, etc. without limitation.

[0202] In a special case, the first language can include multiple language types. For example, Figure 3D the language identifier 315a in Figure 4C , the first language indicated by the language identifier 403a in does not limit the language type of the pre-translation text. No matter what language type the text in the first image belongs to, the electronic device 100 will translate it into the second language. In some embodiments, the first image can include text information of different language types, and the electronic device 100 can translate the text information of different language types into the second language in response to the first operation.

[0203] The process of translating the text information in the first image is described below.

[0204] S103, in response to the first operation, the electronic device 100 identifies the short text in the first image.

[0205] The electronic device 100 can obtain the first image from a RAM or a ROM, or directly from the screen content drawn by the electronic device 100. Details can be referred to the foregoing description.

[0206] If the first language set by the user or the electronic device 100 is a single language, for example, the first language is set to English, Russian, Korean, Japanese, the electronic device 100 can only recognize the text belonging to the first language in the first image, and then recognize the short text in the text belonging to the first language.

[0207] If the first language set by the user or the electronic device 100 contains multiple language types, for example Figure 3D the language identifier 315a in the first image, Figure 4C the first language indicated by the language identifier 403a in the first image, the electronic device 100 can recognize the text belonging to the multiple language types in the first image, and then recognize the short text in the text. For example, the electronic device 100 can recognize the short text belonging to English in the first image, and also recognize the short text belonging to Korean and Japanese in the first image, and so on.

[0208] In a specific implementation, the electronic device 100 can use optical character recognition (OCR) or other text recognition technology to extract the text information in the first image.

[0209] In the embodiment of the present application, the electronic device 100 can recognize the short text in the extracted text information of the first image. The short text refers to the text with a relatively short length, sparse content, and concise expression, and usually does not exceed a certain number of characters, for example, the text composed of 2-5 words or the text within 160 characters. The short text usually forms a paragraph by itself, and thus there is no context or limited context. The short text can include, but is not limited to, for example, dish name, road sign, table, microblog, chat information, news theme, opinion review, mobile phone message, or literature abstract, and so on.

[0210] Therefore, the electronic device 100 can recognize the text in the text information of the first image that meets the characteristics of the short text as the short text according to the characteristics of the short text. For example, the electronic device 100 can recognize the text that is independent as a paragraph and has a length of no more than 5 words in the extracted text information of the first image as the short text.

[0211] For example, the electronic device 100 can recognize the short text "BATTERED WHITING" in the first image (i.e., the picture 312) shown in Figure 3B Figure 4A ​the short text "For Agency Use Only", "Declaration Card", etc. in the first image (i.e. the image displayed in the preview frame 301) shown.

[0212] The first image can include other types of text in addition to the short text, such as long text, etc. without limitation.

[0213] In S104, the electronic device 100 generates the context of the recognized short text.

[0214] Here, the context of the short text generated by the electronic device does not refer to the text before or after the short text in the same paragraph, but refers to the text having the same effect as the actual context. The context generated by the electronic device can reflect the context of the short text, which is some factors that have an impact on the short text, such as time, place, scene, degree of formality of speaking, topic or register, etc.

[0215] In the embodiments of the present application, the electronic device 100 can generate the context of the short text in the following ways:

[0216] (1) The electronic device 100 generates the context using the supplementary information of the short text in the first image.

[0217] In some cases, although the short text in the first image does not have context or has limited context, there can be some supplementary information after or before the short text. These supplementary information can be a further supplementary description of the short text. For example, the name of a dish is usually followed by the ingredients and cooking methods of the dish, the location of the corresponding place is usually below the road sign, the explanatory description of the content in the table is usually below the table, etc.

[0218] Therefore, if there is supplementary information of the short text in the first image, the context can be generated using the supplementary information.

[0219] In a specific implementation, the electronic device 100 can determine the text information in the first image and within a second value from the short text as the supplementary information of the short text. The second value can be pre-set. For example, the short text can be the text one line after the short text, or the text one line before the short text, etc. Illustratively, the electronic device 100 can determine the text "Battered whiting fillets served with thick cut chips" in the first image (i.e. the picture 312) as the supplementary information of the short text "BATTERED WHITING". Figure 3B

[0220] ​After determining the supplementary information, the electronic device 100 can extract, as the context of the short text, part information in the supplementary information whose relevance to the short text exceeds a certain value. The value can be set in advance, for example, can be set to a first value.

[0221] In some embodiments, the electronic device 100 can use an attention mechanism to calculate an attention score between each word in the supplementary information and each word in the short text. The attention score is the relevance or similarity between each information in the supplementary information and each information in the short text, and the higher the attention score, the higher the relevance or similarity. The following describes possible ways to calculate the attention score.

[0222] In the attention mechanism, the information obtained by the electronic device 100 (including the supplementary information and the short text) can be regarded as a series of <Key, Value> data pairs, and then any one of the following ways is used to calculate the similarity or relevance of Query and each Key:

[0223] similarity(query, key i ) = query · key i Equation 1

[0224]

[0225] similarity(query, key i ) = MLP(query, key i ) Equation 3

[0226] In the above three equations, similarity(query, key i ) is the similarity or relevance between a certain Query and key i , Query is the result of semantic encoding of a certain word in the supplementary information (i.e., the converted feature vector), and key i is the result of semantic encoding of the i-th word in the short text.

[0227] Among them, Equation 1 calculates the similarity by dot product. Equation 2 calculates the similarity by Cosine. Equation 3 calculates the similarity by introducing an additional neural network, which can be a recurrent neural network, a convolutional neural network, etc., without limitation.

[0228] After that, a SoftMax-like calculation method can also be introduced to convert the attention score obtained by the above equation into a numerical value, and to normalize it. The original each query and keyi The calculated scores between each of the pieces of supplementary information and each of the pieces of short text are arranged into a probability distribution with the sum of the weights of all elements being 1.

[0229] The method of calculating the correlation or similarity between each piece of supplementary information and each piece of short text is not limited to the above-mentioned formulas, and the electronic device 100 in the embodiments of the present application can also calculate the correlation or similarity in other ways, which are not limited here.

[0230] After calculating the attention scores, the electronic device 100 can extract the part of the supplementary information that has a correlation with the short text exceeding a certain value as the context of the short text. There can be multiple strategies for extracting the context, including but not limited to the following strategies: 1. Extracting the words in the supplementary information that have a correlation with any word in the short text exceeding a threshold value as the context. 2. Extracting the words in the supplementary information that have a correlation with each word in the short text exceeding a threshold value as the context. The threshold value can be set in advance, for example, if the attention scores are normalized, the threshold value can be set to 0.6 or 0.7.

[0231] For example, referring to Figure 6 , Figure 6 shows the attention between the supplementary information and the short text in the first image (i.e., picture 312) in Figure 1 .

[0232] As shown in Figure 6 , the two neurons in the top layer represent the two words in the short text "BATTERED WHITING": BATTERED and WHITING. The 8 neurons in the bottom layer represent the 8 words in the supplementary information "Battered whiting fillets served with thick cut chips": Battered, whiting, fillets, served, with, thick, cut, and chips. The connection between any two neurons in the top and bottom layers represents the attention between the corresponding two words, and the thicker the connection, the greater the attention and the greater the correlation.

[0233] Based on the attention shown in Figure 6 , the electronic device 100 can determine the three words Battered, whiting, and chips as the supplementary information of the short text "BATTERED WHITING".

[0234] (2) The electronic device 100 uses external supplementary information to generate the context of the short text.

[0235] In some cases, the short text in the first image contains an entity. The entity refers to a proper name and a meaningful phrase appearing in the short text, such as an organization name, a person name, a place name, a food name, and the like. The entity can also be referred to as a term. The entity usually has a term explanation information corresponding to the entity in an external knowledge base, which is a further supplementary description of the entity in the short text. Therefore, the term explanation information of the entity extended by the external knowledge base can be used as supplementary information of the short text to generate the context of the short text.

[0236] Here, the entity contained in the short text in the first image can also be referred to as a first entity. The term explanation information corresponding to the entity can be referred to as first term explanation information.

[0237] In a specific implementation, the electronic device 100 can first extract the entity in the short text by using deep learning, and then find the term explanation information corresponding to the entity in the external knowledge base. The external knowledge base is stored in a network and is a database containing a large amount of semantic information and entity information. The external knowledge base can include but is not limited to Wikipedia, Wordnet, and the like. For example, the electronic device 100 can identify the entity fish and chips in the short text "BATTERED WHITING" in the first image (i.e., the picture 312) and find the term explanation information corresponding to the entity in the external knowledge base. Figure 3B The entity fish and chips in the short text "BATTERED WHITING" in the first image (i.e., the picture 312) is shown, and the term explanation information corresponding to the entity in the external knowledge base is found. The term explanation information of the entity fish and chips is as follows: Fish and chips is a hot dish consisting of fried fish in batter served with chips. The dish originated in England and is an example of culinary fusion, as its two main ingredients were introduced by immigrants.

[0238] After obtaining the term explanation information, the electronic device 100 can extract part of the information in the term explanation information that has a relevance to the short text exceeding a certain value as the context of the short text. The value can be pre-set, for example, it can be set to a first value. Here, the specific implementation of extracting part of the information in the term explanation information that has a relevance to the short text exceeding a certain value as the context can refer to the description of extracting the context in the supplementary information in the method (1) described above, which will not be repeated here.

[0239] The supplementary information mentioned in the above (1) and (2) can also be referred to as the first text.

[0240] Any word in the first text can be referred to as a first word, and the result of semantic encoding of the first word can be referred to as a first feature vector. Any word in the short text can be referred to as a second word, and the result of semantic encoding of the second word can be referred to as a second feature vector.

[0241] (3) The electronic device 100 directly generates the context of the short text through the trained generation network.

[0242] In some cases, the electronic device 100 can directly generate the context of the short text through the trained generation network. In this way, even if there is no supplementary information for some short text in the first image, or there is a lack of external knowledge base, the electronic device 100 can generate the context of the short text.

[0243] In a specific implementation, the electronic device 100 can use the external knowledge base to train the generation network.

[0244] In some embodiments, the electronic device 100 can use the entries and entry explanation information in the external knowledge base as a corpus pair, such as the corpus pair composed of Official and entry explanation information An official is someone who holds an office in an organization or government, to train the generation network, which can be a sequence to sequence (Seq2Seq) network. Wherein, the entry explanation information is the input of the Seq2Seq network, and the entry is the output of the Seq2Seq, so as to train the Seq2Seq network. Reference Figure 7 , Figure 7 The structure of the Seq2Seq network provided by the embodiments of the present application is shown. The Seq2Seq network is implemented by an Encoder-Decoder (encoding-decoding) framework, and the Encoder and Decoder parts can adopt CNN, RNN, LSTM, GRU, BLSTM, etc. The left side Encoder encodes the input sequence converted from the entry explanation information into a fixed length vector encoding, and the right side Decoder decodes the previously generated fixed vector into an output sequence, which is converted into natural language after the entry.

[0245] In other embodiments, the electronic device 100 can use pre-training and fine-tuning to train the generative network. Specifically, the electronic device 100 can use a generative pre-training (GPT) model to train the generative network, which specifically includes two stages. In the first stage, a language model is pre-trained using a large amount of unlabeled corpus, and then, in the second stage, the pre-trained language model is fine-tuned using the collected corpus pairs. The collected corpus pairs can be, for example, terms and term explanation information, such as the corpus pair consisting of Official and Anofficial is someone who holds an office in an organization or government. In this way, the pre-trained language model can be migrated to the supervised context generation task, ensuring that the input of the trained GPT model is terms and the output is term explanation information, which can be used as the context of the short text. Training the generative network by pre-training and fine-tuning can improve the efficiency of training the generative network.

[0246] After training the generative network (e.g., Seq2Seq network or GPT model), the electronic device 100 can input the short text in the first image into the generative network and output it as the context of the short text. Figure 4A The short text in the preview box 301 (e.g., "For Agency Use Only") is input into the trained generation network, and its output (e.g., "designation used by governments") is used as the context of the short text.

[0247] It is understandable that the above three methods are only examples. In the embodiment of the present application, the electronic device 100 can also generate the context of the short text in other ways, for example, it can also generate the context through network keyword search and other methods, etc., which is not limited here.

[0248] S105 : The electronic device 100 translates the short text in the first image using the generated context to obtain a translation of the short text.

[0249] In the embodiment of the present application, the electronic device 100 can use a transformer model based on a self-attention mechanism to take a short text and context as input, and output a translation result. The following describes in detail how to translate a short text based on the transformer model.

[0250] refer to Figure 8A ,Figure 8A The structure of the transformer model provided by the embodiments of the present application is shown.

[0251] As shown in Figure 8A , the transformer model is composed of an encoder and a decoder. The encoder is used to process the input short text and the context, and the decoder is used to predict the translation result (i.e., output the translation result).

[0252] The encoder is composed of Nx encoding blocks (blocks), which refer to the encoder in Figure 8A . Nx may be 6 or other values, which are not limited here. Each encoding block in the encoder is composed of a self-attention network, a feed forward neural network (FFNN).

[0253] The decoder is also composed of Nx decoding blocks (blocks), which refer to the dncoder in Figure 8A . Nx may be 6 or other values, which are not limited here. Each decoding block in the decoder is composed of a self-attention network, an encoder-decoder attention network, and a feed forward neural network (FFNN).

[0254] The output of the encoder will be used as the input of the decoder.

[0255] Figure 8B The structure of the transformer model is shown in more detail.

[0256] The process of the electronic device 100 using the generated context to translate the short text in the embodiments of the present application is described below. Figure 8B The translation process can include the following steps:

[0257] 1. First, the short text and the context are respectively processed by word embedding to obtain the feature vector corresponding to the short text, and the feature vector corresponding to the context.

[0258] Word embedding processing refers to representing each word in the short text with a high-dimensional feature vector. The numerical value of the feature vector can be understood as the abstract meaning of the corresponding word. By representing the short text with a feature vector, various algebraic operations can be performed on the vector in the subsequent process, and the operation results can be used to quantitatively represent the semantics of the sentence, and finally realize machine translation.

[0259] The electronic device 100 can convert the short text into a feature vector using a word embedding method, which can include, for example, One-hot encoding, Word2vec, and the like, without limitation. The dimension of the word embedding is not limited, and can be, for example, 512 or 768, or the like.

[0260] If the short text has been semantically encoded in the process of generating the context in S104 described above, the process of word embedding the short text again can be omitted here.

[0261] In practice, each word of the short text can be subjected to word embedding processing, and the feature vectors corresponding to each word are combined into a matrix. That is, after the short text is subjected to word embedding processing, a matrix for abstractly representing the short text can be obtained. The vector corresponding to a single word in the short text can be denoted by x, and the matrix corresponding to the short text can be denoted by X. X can also be referred to as a second feature vector matrix.

[0262] Similarly, the context can also be subjected to word embedding processing to obtain a matrix for abstractly representing the context. The vector corresponding to a single word in the context can be denoted by y, and the matrix corresponding to the context can be denoted by Y. Y can also be referred to as a first feature vector matrix.

[0263] 2. The encoder encodes the feature vector corresponding to the short text.

[0264] The encoding process includes the processing of Nx encoding blocks. The feature vector converted from the short text and the context will be input into the first encoding block of the encoder, the output of the first encoding block will be input into the next encoding block, and so on, until all the encoding blocks are processed, i.e., the encoding of the feature vector is completed. The processing of each encoding block is similar, and includes the following two steps: (1) processing of a self-attention network; and (2) processing of an FFNN.

[0265] The processing of the first encoding block will be explained below as an example.

[0266] (1) Processing of a self-attention network.

[0267] Self-attention network is used to calculate the mutual relationship of each word in a sentence with all the words in the sentence, which reflects the relevance and importance of different words in the sentence to some extent. Therefore, using these mutual relationships to adjust the importance (weight) of each word can obtain a new expression of each word. This new representation not only contains the word itself, but also contains the relationship between other words and this word, so it is a more global expression compared with the pure word vector. That is, the self-attention network is used to learn a weight for each word.

[0268] In the attention mechanism, each word has three different vectors, namely the query vector q, the key vector k and the value vector v. These three vectors are obtained by multiplying the feature vector of the word by three different weight matrices, as shown in the following formulas:

[0269] q=y×W Q Formula 4

[0270] k=x×W K Formula 5

[0271] v=x×W V Formula 6

[0272] wherein W Q , W K , W V The sizes of the three matrices are the same. W Q , W K , W V are parameter matrices obtained by pre-training, and the parameter matrix can be represented as dxd, where d represents the dimension of the input hidden state, which can be 512 or 1024, etc. Here, it is not limited. Of course, in actual application, the parameter matrix can also be represented as d1xd2, such as 512x64.

[0273] Then, a score value is calculated for each vector using the following formula:

[0274] score=q·k Formula 7

[0275] The score value score is a similarity calculation used to calculate the similarity of q and k.

[0276] Then, in order to stabilize the gradient, the score of each vector is normalized, that is, score is divided by

[0277] Then, the score is processed using the softmax activation function.

[0278] Then, multiply the processed score by v to get the weighted score v of each feature vector.

[0279] Finally, the scores v of each vector are added together to obtain the output z of the self-attention network, where z = ∑v.

[0280] In the actual process, for the convenience of calculation, a matrix-based calculation method is adopted, which can be summarized as the following formula:

[0281]

[0282] Among them, Q, K, and V are the request vector matrix corresponding to the context, the key vector matrix corresponding to the short text, and the value vector matrix corresponding to the short text, respectively.

[0283] Z is the output of the self-attention network, which is a weighted eigenvector matrix. This eigenvector matrix will be fed into the FFNN.

[0284] Q=Y×W Q Formula 9

[0285] K=X×W K Formula 10

[0286] V=X×W V Formula 11

[0287] (2) FFNN processing.

[0288] FFNN is a fully connected network consisting of two layers. The activation function of the first layer is ReLU, and the second layer is a linear activation function. The output of FFNN can be expressed as:

[0289] FFN(Z)=max(0,ZW1+b1)W2+b2 Formula 12

[0290] After processing multiple encoding blocks, the input text segment can be continuously overlapped with such attention mechanism layers and ordinary nonlinear layers to obtain the final text representation.

[0291] 3. The decoder processes the output of the encoder and finally outputs the translation result.

[0292] The decoding process involves processing Nx decoding blocks. The encoder's output in step 2 is used as the input to the first decoding block in the decoder. The output of the first decoding block is used as the input to the next decoding block, and so on. This process continues until all decoding blocks have been processed, completing the decoding process.

[0293] The processing of each decoding block is similar, and includes the following two steps: (1) processing of a self-attention network; (2) processing of an encoder-decoder attention network; and (3) processing of an FFNN. The self-attention network is used to output the relationship between the current translation and the previously translated context. The encoder-decoder attention network is used to output the relationship between the current translation and the encoded feature vector.

[0294] The input of each decoding block is: the output of the encoder, and the output of the decoding block at the i-1 position; and the output is the probability distribution of the output word at the i position.

[0295] In the encoder-decoder attention network, Q is from the previous output of the decoder, and K and V are from the output of the encoder, which are calculated in the same way as the above formulas 9, 10 and 11.

[0296] After the decoder completes the decoding, the decoded feature vector is input into a fully connected layer with a softmax activation function to obtain an output vector reflecting the probability of each word. After the output vector is processed by a reverse word embedding, the translated text is obtained.

[0297] The translation is not limited to the transformer model, and other neural network models can also be used for translation, which is not limited herein.

[0298] S106, the electronic device 100 displays the translated text of the short text on the display screen.

[0299] In some embodiments of the present application, in response to the first operation in S103, in addition to translating the short text in the first image by the method in S104 and S105, the electronic device 100 can also translate other text in the first image in response to the first operation. For example, the electronic device 100 can translate the other text in the first image that belongs to the first language and is other than the short text into the second language. Here, the method used by the electronic device 100 to translate the other text in the first image that belongs to the first language and is other than the short text is not limited in the embodiments of the present application. For example, the electronic device 100 can use the method in S104 and S105 to translate the other text, or use other translation methods to translate the other text, which is not limited herein.

[0300] Specifically, the electronic device 100 can display the translated text of the short text on the display screen in the following manners:Figure 3E , Figure 4D For example, referring to Figure 3E , the short text "BATTERED WHITING" is translated to "fried whiting". Another example, referring to Figure 4D , the short text "For Agency Use Only" is translated to "For official use only".

[0301] In some other embodiments, Figure 5 One or more of the steps S103, S104, and S105 in can be executed by the server. That is, the server can be used to execute the process of translating the text information in the first image. For example, after the electronic device 100 obtains the above S102, it can send the first image to the server. The server can identify the text information in the first image and return the translated text obtained after translating the text information in the first image to the electronic device 100. Another example, the electronic device 100 can send the short text of the identified first image to the server after the above S103. The server can translate the short text and return the translated text obtained to the electronic device 100. In this way, part of the operations can be executed on the server side, and the processing resources of the server can be used to translate the short text, which can reduce the burden on the electronic device 100 side and can speed up the translation efficiency, enabling the user to see the translated text faster and improving the user experience.

[0302] Implementing the short text translation method provided in the above embodiments, for short texts without context or with limited context, the electronic device can generate the context of the short text, greatly supplement the short text, and use the context to translate the short text. This can improve the translation accuracy of short texts and improve the user experience. Further, when the electronic device uses the transformer model to translate short texts, it can avoid the sequence dependence problem existing in other translation models, can be more easily parallel translated, and has higher efficiency.

[0303] Not limited to the translation method, the present application can also provide natural language processing such as intention recognition, machine reading comprehension, speech recognition, automatic summarization, and question answering for short texts. Specifically, the electronic device can identify the short text and use the context generation method provided in the above embodiments to generate the context of the short text, and then use the short text and the generated short text to perform corresponding natural language processing, such as intention recognition, machine reading comprehension, speech recognition, etc. This can improve the efficiency of various natural language processing for short texts.

[0304] The embodiments of the present application also provide a method for generating the context of the short text, which can refer to the third method for generating the context of the short text in S104 of the above-mentioned method. Figure 5 Through the method, the electronic device 100 can directly generate the context of the short text through the trained generation network. In this way, even if there is no supplementary information for some short text in the first image, or there is a lack of external knowledge base, the electronic device 100 can generate the context of the short text.

[0305] The embodiments of the present application can be combined in any way to achieve different technical effects.

[0306] In the above embodiments, all or part of the embodiments can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of the embodiments can be realized in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, all or part of the computer program instructions generate the processes or functions described in the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk (SSD)) and the like.

[0307] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program to instruct the relevant hardware to complete, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes ROM or random storage memory RAM, magnetic disk or optical disk and various program code storage media.

[0308] In summary, the above only describes the embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present application shall be included in the protection scope of the present application.

Claims

1. A method of translating short texts, characterized by, The method comprises: The electronic device displays a first image; After the electronic device receives a first operation, the electronic device identifies short text in the first image, and the short text does not include context; determining a first text, wherein, in the case that there is supplementary information in the first image, the first text includes the supplementary information, the supplementary information includes text in the first image and in a different paragraph from the short text and within a second value from the short text, and / or, in the case that a knowledge base contains first entity corresponding entry explanation information, the first text includes the first entity corresponding entry explanation information in the knowledge base, the knowledge base contains a plurality of entity corresponding entry explanation information, and the first entity is an entity in the short text; determining the context of the short text according to the first text, wherein the context of the short text is a word in the first text that is more than a first value related to a word in the short text, or the context of the short text is a word in the first text that is more than the first value related to each word in the short text; Or, if the electronic device fails to determine the first text, the electronic device takes the short text as the input of a generation network, and determines the output of the generation network as the context of the short text, wherein the generation network is a network obtained by training a sequence to sequence (Seq2Seq) network with entities in the knowledge base as input and entry explanation information corresponding to the entities as reference output, or the generation network is a network obtained by pre-training a language model with unlabeled corpus and then fine-tuning the language model with collected corpus pairs, wherein the collected corpus pairs contain entities and entry explanation information corresponding to the entities; The electronic device translates the short text using the context to obtain a translation of the short text.

2. The method of claim 1, wherein, Before determining the context of the short text according to the first text, the method further comprises: performing semantic encoding on a first word in the first text to obtain a first feature vector corresponding to the first word; and obtaining a request vector according to the first feature vector; performing semantic encoding on a second word in the short text to obtain a second feature vector corresponding to the second word; and obtaining a key vector according to the second feature vector; determining the relevance of the first word and the second word according to the request vector and the key vector; wherein the first word includes one word in the first text, and the second word includes one word in the short text.

3. The method of claim 1, wherein, The electronic device translates the short text using the context to obtain a translation of the short text, specifically comprising: performing semantic encoding on the context to obtain a first feature vector matrix corresponding to the context; and obtaining a request vector matrix according to the first feature vector matrix; performing semantic encoding on the short text to obtain a second feature vector matrix corresponding to the short text; and obtaining a key vector matrix according to the second feature vector matrix; determining relevance of the context and the short text according to the request vector matrix and the key vector matrix; performing encoding processing on the short text according to the relevance of the context and the short text, to obtain an encoding result of the short text; performing decoding processing on the encoding result of the short text, to obtain a translation corresponding to the short text.

4. The method of claim 1, wherein, Before the electronic device displays the first image and before the electronic device receives the first operation, the method further includes: The electronic device receives a second operation acting on the first image. The electronic device displays a first option. The first operation includes a user operation acting on the first option.

5. The method of claim 4, wherein, Before the electronic device displays the first image, the method further includes: The electronic device displays a first shooting preview interface, and the first shooting preview interface includes a first preview frame, and the first preview frame displays a first image collected by a camera of the electronic device in real time. The electronic device saves the first image in the first preview frame.

6. The method of claim 4, wherein, The first image includes an image downloaded by the electronic device from a network or an image received by the electronic device from another electronic device.

7. The method of claim 1, wherein, Before the electronic device displays the first image, the method further includes: The electronic device displays a first shooting preview interface, and the first shooting preview interface includes a first preview frame and a first control, and the first preview frame displays an image collected by a camera of the electronic device in real time. The electronic device displays the first image, specifically includes: The electronic device receives a user operation acting on the first control. The electronic device displays a second shooting preview interface, and the second shooting preview interface includes a second preview frame and a first option, and the second preview frame displays the first image collected by the camera in real time. The first operation includes a user operation acting on the first option.

8. The method according to any one of claims 1 to 7, characterized in that, After the electronic device translates the short text using the context to obtain a translation of the short text, the method further includes: The electronic device displays the translation of the short text on the first image.

9. An electronic device, comprising: includes: a display screen, a memory, and one or more processors; the display screen, the memory, and the one or more processors are coupled, the memory is configured to store computer program code, the computer program code includes computer instructions, and the one or more processors invoke the computer instructions to cause the electronic device to perform: displaying a first image; after receiving a first operation, identifying a short text in the first image, and the short text does not include a context; determining first text, wherein, in the case that supplementary information exists in the first image, the first text comprises the supplementary information, the supplementary information being text in the first image and in a different paragraph from the short text and having a distance from the short text within a second value, and / or, in the case that a knowledge base comprises first explanation information corresponding to a first entity, the first text comprises the first explanation information corresponding to the first entity in the knowledge base, the knowledge base comprising explanation information corresponding to a plurality of entities, and the first entity being an entity in the short text; determining context of the short text according to the first text, wherein the context of the short text is a word in the first text having a relevance to a word in the short text exceeding a first value, or the context of the short text is a word in the first text having a relevance to each word in the short text exceeding the first value; Or, if the electronic device fails to determine the first text, the electronic device takes the short text as input of a generation network, and determines output of the generation network as the context of the short text, wherein the generation network is a network obtained by training a sequence-to-sequence (Seq2Seq) network with entities in the knowledge base as input and explanation information corresponding to the entities as reference output, or the generation network is a network obtained by pre-training a language model with unlabeled corpus and then fine-tuning the language model with collected corpus pairs, the collected corpus pairs comprising entities and explanation information corresponding to the entities; translating the short text by using the context to obtain a translation of the short text.

10. The electronic device of claim 9, wherein, The one or more processors are further configured to invoke the computer instructions to cause the electronic device to perform: before determining, as the context of the short text, text in the first text having a relevance to the short text exceeding a first value, performing semantic encoding on a first word in the first text to obtain a first feature vector corresponding to the first word; and obtaining a request vector according to the first feature vector; performing semantic encoding on a second word in the short text to obtain a second feature vector corresponding to the second word; and obtaining a key vector according to the second feature vector; determining the relevance of the first word and the second word according to the request vector and the key vector. The first word comprises a word in the first text, and the second word comprises a word in the short text.

11. The electronic device of claim 9, wherein, The one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to perform: performing semantic encoding on the context to obtain a first feature vector matrix corresponding to the context; and obtaining a request vector matrix according to the first feature vector matrix; performing semantic encoding on the short text to obtain a second feature vector matrix corresponding to the short text; and obtaining a key vector matrix according to the second feature vector matrix; determining the relevance of the context and the short text according to the request vector matrix and the key vector matrix. According to the context and the relevance of the short text, the short text is encoded to obtain an encoding result of the short text; The encoding result of the short text is decoded to obtain a translation corresponding to the short text.

12. The electronic device of claim 9, wherein, The one or more processors are further configured to invoke the computer instructions to cause the electronic device to perform: receiving a second operation acting on the first image; displaying a first option; The first operation includes a user operation acting on the first option.

13. The electronic device of claim 12, wherein, The one or more processors are further configured to invoke the computer instructions to cause the electronic device to perform: Before displaying the first image, a first shooting preview interface is displayed, the first shooting preview interface including a first preview frame, the first preview frame displaying a first image obtained by the camera of the electronic device in real time; Saving the first image in the first preview frame.

14. The electronic device of claim 12, wherein, The first image includes an image downloaded by the electronic device from a network, or an image received by the electronic device from another electronic device.

15. The electronic device of claim 9, wherein, The one or more processors are further configured to invoke the computer instructions to cause the electronic device to perform: Before displaying the first image, a first shooting preview interface is displayed, the first shooting preview interface including a first preview frame and a first control, the first preview frame displaying an image obtained by the camera of the electronic device in real time; The one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to perform: the electronic device receives a user operation acting on the first control; the electronic device displays a second shooting preview interface, the second shooting preview interface including a second preview frame and a first option, the second preview frame displaying the first image obtained by the camera in real time; The first operation includes a user operation acting on the first option.

16. The electronic device of any of claims 9-15, wherein, The one or more processors are further configured to invoke the computer instructions to cause the electronic device to perform: After translating the short text using the context to obtain a translation of the short text, displaying the translation of the short text on the first image.

17. A computer-readable storage medium comprising instructions, wherein: When the instructions run on an electronic device, the electronic device performs the method of any one of claims 1-8.

18. A computer program product, characterised in that, When the computer program product runs on a computer, the computer performs the method of any one of claims 1-8.

Images