AR translation processing method and electronic device

Abstract

A processing method and electronic device for AR translation, relating to the field of communication technology. Through the present application scheme, in the scenario where an electronic device is used for AR translation, the posture change of the electronic device can be detected in real time, and feature matching can be performed on the continuous multiple frames of images captured by the camera. Then, according to the posture change of the electronic device and the feature matching, it can be determined whether the text to be translated needs to be fully translated, partially translated, or not translated, so as to select the corresponding translation trigger strategy. This can effectively avoid repeated translation, thereby saving computing resources in the AR translation process and improving translation efficiency to a certain extent.

Description

[0001] This application is a divisional application of the Chinese patent application submitted to the State Intellectual Property Office on September 8, 2021, with application number 202111051366.8 and application name "AR translation processing method and electronic device". Technical Field

[0002] The present application relates to the field of augmented reality (AR) technology, and in particular to a processing method and electronic device for AR translation. Background Art

[0003] At present, AR technology is increasingly used in online shopping, education, medical treatment, games, navigation, translation and other fields. Among them, AR technology is a technology that superimposes virtual digital content (such as three-dimensional models) and the real physical world.

[0004] For example, taking AR translation technology (AR translation for short) as an example, the mobile phone can capture the image of the target to be translated through the camera, and then translate the text content in the image, and superimpose the virtual digital content obtained by the translation with the preview image captured by the mobile phone camera to present a 3D translation effect on the display screen.

[0005] At present, the AR translation method is usually to translate and display the captured images frame by frame in real time. That is to say, the entire translation process needs to be performed for each frame of the image captured by the AR device, which requires the use of a large amount of computing resources, resulting in low translation efficiency. Therefore, there is an obvious lag when displaying the translation results, which affects the user experience. Summary of the invention

[0006] The present application provides an AR translation processing method and electronic device, which solves the problem that a large amount of computing resources are needed during AR translation and the translation result display is stuck.

[0007] In order to achieve the above purpose, this application adopts the following technical solutions:

[0008] In a first aspect, the present application provides a method for processing AR translation, the method comprising:

[0009] Capturing a first image through a camera of an electronic device, wherein the first image includes a first text to be translated;

[0010] Translating a first text to be translated to obtain a first translation result;

[0011] Displaying a first image, and displaying a first virtual image superimposed on the first image, wherein the first virtual image includes a first translation result;

[0012] Capturing a second image through a camera, wherein the second image includes a second text to be translated;

[0013] If the posture change of the electronic device is less than a preset posture threshold, and the feature similarity between the second image and the first image is greater than or equal to a preset similarity threshold, then obtaining the first translation result as the translation result of the second text to be translated; or,

[0014] If the change in the posture of the electronic device is less than a preset posture threshold, and the feature similarity is less than a preset similarity threshold, the second text to be translated is translated; or,

[0015] If the change in the posture of the electronic device is greater than or equal to a preset posture threshold, and the feature similarity is less than a preset similarity threshold, translating part or all of the second text to be translated;

[0016] The second image is displayed, and a second virtual image is superimposed and displayed on the second image, where the second virtual image includes a translation result of the second text to be translated.

[0017] Through the processing method of AR translation provided by the embodiment of the present application, in the scenario of using electronic equipment for AR translation, the posture change of the electronic equipment can be detected in real time, and the feature matching of the continuous multiple frames of images collected by the camera can be performed, and then according to the posture change of the electronic equipment and the feature matching, it can be judged whether the text to be translated needs to be fully translated, partially translated, or not translated, so as to select the corresponding translation trigger strategy. This can effectively avoid repeated translation, thus saving computing resources in the AR translation process and improving translation efficiency to a certain extent.

[0018] For example, in a scenario where the mobile phone does not move and the reading content does not change, it can be determined that the posture changes little and the text to be translated does not change. Therefore, there is no need to translate it again, and the previous translation result can be directly obtained, which greatly saves computing resources.

[0019] As another example, for a scenario where the mobile phone does not move and the pages are turned for reading, it can be determined that the posture changes slightly, but the text to be translated changes significantly, so all text in the current frame image needs to be translated.

[0020] As another example, for a scene where the mobile phone moves with a large amplitude (such as translation, rotation, stretching, shaking, etc.), it can be determined that the posture has changed greatly and the content to be translated has changed, so all or part of the text in the current frame image needs to be translated.

[0021] When the posture change of the electronic device is greater than or equal to the preset posture threshold, and the feature similarity is less than the preset similarity threshold, there may be the following two situations, and there are two corresponding implementation methods:

[0022] Case 1: In a possible implementation, the translating of part or all of the second text to be translated includes:

[0023] When the first part of the second text to be translated is the same as the first text to be translated, obtaining a first translation result as the translation result of the first part of the text;

[0024] Translating the second part of the text to obtain a translation result of the second part of the text, where the second part of the text is the text other than the first part of the text in the second text to be translated;

[0025] The translation result of the second text to be translated includes the translation result of the first part of the text and the translation result of the second part of the text.

[0026] In the above situation 1, if the text content of the current frame image and the previous frame image partially overlaps or repeats, the text of the overlapping or repeated part does not need to be translated.

[0027] Case 2: In another possible implementation, the translating of part or all of the second text to be translated includes:

[0028] In the case that the second text to be translated does not have the same text as the first text to be translated, all texts in the second text to be translated are translated to obtain a translation result of the second text to be translated.

[0029] In the above situation 2, if there is no overlap or duplication of text content between the current frame image and the previous frame image, all texts in the current frame image need to be translated.

[0030] In some embodiments, the electronic device may call a local translation engine to translate the text to be translated and obtain a translation result, wherein the translation engine may be a translation application installed in the electronic device for text translation.

[0031] In some embodiments, when the electronic device is connected to a network, the electronic device can send the text to be translated to a server for text translation, and the server translates the text to be translated and sends the translation result to the electronic device, so that the electronic device obtains the translation result.

[0032] In a possible implementation, the method further includes:

[0033] Extracting feature points in the first image and feature points in the second image;

[0034] The feature points in the second image are compared with the feature points in the first image to obtain the feature similarity between the second image and the first image.

[0035] Through the above scheme, the features of the collected continuous multiple frames of images can be extracted and matched to determine the number of feature matches or feature similarity between the continuous multiple frames of images, so as to judge whether the text in the current image needs to be fully translated, partially translated, or not translated according to the feature similarity in the following text.

[0036] In a possible implementation, the method further includes:

[0037] The SLAM method of simultaneous positioning and map construction is used to generate SLAM maps;

[0038] The position change of the electronic device is determined according to the second image, the measurement data of the target sensor in the electronic device and the SLAM map.

[0039] Among them, the above-mentioned target sensor includes an inertial measurement unit IMU; of course, the target sensor can also include any other sensor that meets actual usage requirements, such as a gyroscope sensor or an acceleration sensor, which is not limited in this application.

[0040] In a possible implementation, the above-mentioned posture change includes a position change and a posture change.

[0041] By using the SLAM method to track the position and posture of electronic devices in real time, the position and posture changes of electronic devices can be accurately calculated.

[0042] In a possible implementation, the method further includes:

[0043] A target virtual plane for displaying AR digital content is determined by using a SLAM method, where the target virtual plane is located above the plane where the captured image is located;

[0044] The above-mentioned superimposing and displaying the second virtual image on the second image includes:

[0045] The second virtual image is displayed on the target virtual plane.

[0046] It can be seen from the above scheme that the AR scene can include a plane where the captured image (including the text to be translated) is located and a plane where the virtual image (including the translation result) is located. These two images are superimposed and displayed to achieve a three-dimensional display effect.

[0047] In a possible implementation, the method further includes:

[0048] Determining a target projection area of ​​the second text to be translated on the target virtual plane;

[0049] The step of displaying the second virtual image on the target virtual plane includes:

[0050] A second virtual image is displayed in the target projection area.

[0051] It can be seen from the above scheme that by real-time tracking and mapping the position of the text to be translated on the virtual plane, the position for displaying the translation result can be accurately located, which can ensure that the text to be translated and the translation result correspond in position in real time, thereby improving the AR display effect and facilitating user reading.

[0052] In a possible implementation, the step of determining a target projection area of ​​the second text to be translated on the target virtual plane includes:

[0053] Determine a first rectangular area occupied by a second text to be translated in the second image;

[0054] According to two endpoints on the diagonal line of the first rectangular area, determine two anchor points mapped to the two endpoints on the target virtual plane;

[0055] Determine a second rectangular area on the target virtual plane with the two anchor points as diagonals;

[0056] The second rectangular area is a target projection area of ​​the second text to be translated on the target virtual plane.

[0057] It can be seen from the above scheme that by mapping the rectangular area where the text to be translated is located onto the virtual plane, the rectangular display area on the virtual plane for displaying the translation result can be determined. This ensures that the text to be translated and the translation result correspond in position in real time, thereby improving the AR display effect and facilitating user reading.

[0058] In a possible implementation, the method further includes: setting the transparency of the target projection area to be less than or equal to a preset transparency threshold.

[0059] Through the above scheme, the transparency of the target projection area can be reduced, making the target projection area opaque or semi-transparent, so that the translation result can be displayed above the corresponding text to be translated, and the text to be translated can be covered or blocked, so as to avoid the text to be translated from causing visual interference or influence on the virtual displayed translation result, thereby making it easier for users to read and improving the user experience.

[0060] In a possible implementation, the method further includes:

[0061] Performing text recognition on the first image to obtain a first text to be translated;

[0062] And, performing text recognition on the second image to obtain a second text to be translated.

[0063] Exemplarily, optical character recognition (OCR) technology may be used for text recognition.

[0064] In a possible implementation, the method further includes:

[0065] Rendering the translation result of the first text to be translated to obtain a first virtual image; and,

[0066] The translation result of the second text to be translated is rendered to obtain a second virtual image.

[0067] Through the above scheme, the translation result of the text to be translated can be rendered into virtual digital content, so that the virtual digital content can be displayed on a virtual plane, thereby presenting an AR display effect.

[0068] In a second aspect, the present application provides an AR translation processing device, the device comprising a unit for executing the method in the first aspect. The device may correspond to executing the method described in the first aspect. For the relevant description of the units in the device, please refer to the description of the first aspect above, and for the sake of brevity, it will not be repeated here.

[0069] The method described in the first aspect above can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a processing module or unit, a display module or unit, etc.

[0070] In a third aspect, the present application provides an electronic device comprising a processor, the processor being coupled to a memory, the memory being used to store computer programs or instructions, and the processor being used to execute the computer programs or instructions stored in the memory, so that the method in the first aspect is executed.

[0071] For example, the processor is used to execute the computer program or instructions stored in the memory, so that the device performs the method in the first aspect.

[0072] In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon a computer program (also referred to as instructions or codes) for implementing the method in the first aspect.

[0073] For example, when the computer program is executed by a computer, the computer can perform the method in the first aspect.

[0074] In a fifth aspect, the present application provides a chip, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof.

[0075] Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0076] In a sixth aspect, the present application provides a chip system, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof.

[0077] Optionally, the chip system also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0078] In a seventh aspect, the present application provides a computer program product, comprising a computer program (also referred to as instructions or codes), wherein when the computer program is executed by a computer, the computer implements the method in the first aspect.

[0079] It can be understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a system architecture diagram of the AR translation scenario involved in the embodiments of the present application;

[0081] Figure 2 A flowchart of a processing method for AR translation provided in an embodiment of the present application;

[0082] Figure 3 A schematic diagram of an interface for enabling an AR translation function in an AR translation processing method provided in an embodiment of the present application;

[0083] Figure 4 A schematic diagram of a method for processing AR translation provided in an embodiment of the present application, in which a coordinate system is combined to represent a posture change of an electronic device;

[0084] Figure 5 A schematic diagram of the overall process of an AR translation processing method provided in an embodiment of the present application;

[0085] Figure 6 A schematic diagram of an interactive interface in a scenario where an electronic device captures images and extracts text for translation during AR translation in an embodiment of the present application is shown;

[0086] Figure 7 A schematic diagram of an interactive interface in a scenario where the image content changes and the content is repeated after the posture of the electronic device changes during AR translation in an embodiment of the present application is shown;

[0087] Figure 8A schematic diagram of an interactive interface in a scenario where the posture of the electronic device changes little during AR translation and the image content changes and the content is not repeated in an embodiment of the present application is shown;

[0088] Fig. 9 A schematic diagram of an interactive interface in a scenario where the posture of the electronic device changes little and the image content remains unchanged during AR translation in an embodiment of the present application is shown;

[0089] Fig.10 The following schematically shows an interface diagram for AR translation using the solution of the present application;

[0090] Fig.11 A schematic diagram of the structure of an AR translation processing device provided in an embodiment of the present application;

[0091] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0092] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0093] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0094] The terms "first", "second", etc. in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of objects. For example, a first image and a second image are used to distinguish different images rather than to describe a specific order of images.

[0095] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0096] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0097] To facilitate understanding of the embodiments of the present application, some terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0098] (1) SLAM: Simultaneous localization and mapping (SLAM), a visual positioning and tracking method that can complete the construction and tracking of the scene based on the spatial position information (such as posture information) collected by the camera of the electronic device. Among them, feature points in the environment can be extracted from the image for mapping, and the position of objects in the environment (such as the target or text to be translated) can be determined in the environment and tracked in real time.

[0099] (2) AR: Augmented reality (AR) technology can superimpose computer-generated virtual objects, scenes, or system prompt information onto real scenes, integrating virtual scenes with real scenes, thereby achieving “enhancement” of reality.

[0100] Specifically in the AR translation scenario, after the real scene (such as the target to be translated) and the virtual scene (such as the translation result) overlap, they can exist simultaneously in the same screen and space, which can enhance the user's reading experience.

[0101] Among them, the AR algorithm can provide SLAM tracking capabilities, provide the ability to recognize planes in the real environment and track the real position and posture of the current device camera. For example, the AR electronic device can be used to collect the spatial position information corresponding to the scene, and then the SLAM tracking and positioning method can be used to complete the construction and tracking of the scene based on the spatial position information collected by the AR device.

[0102] With the rapid development of electronic device performance, visual positioning tracking and AR technology, AR devices can be used to capture images of the text to be translated for translation, and present an intuitive three-dimensional translation effect on the image of the text to be translated, becoming an important means to assist people in solving cross-language text comprehension problems.

[0103] However, the current AR translation method is usually to translate and display the captured images frame by frame in real time, which means that the entire translation process needs to be performed for each frame of the image captured by the AR device, which requires a large amount of computing resources. Among them, processes such as character extraction and translation usually need to be uploaded to the cloud to complete, which requires the use of cloud computing resources and has problems such as traffic loss, display freezes of translation results, and increased device heat and power consumption.

[0104] On the other hand, when the user's posture of holding or wearing the AR device changes (such as moving, rotating, stretching or shaking), the translation results may drift and become inconsistent, and the font format and size may jump.

[0105] In view of this, the embodiment of the present application provides a processing method for AR translation. In the scenario of using electronic equipment for AR translation, the posture change of the electronic equipment can be detected in real time, and the continuous multi-frame images captured by the camera can be feature matched. Then, according to the posture change of the electronic equipment and the feature matching, it can be judged whether the text to be translated needs to be fully translated, partially translated, or not translated, so as to select the corresponding translation trigger strategy. This can effectively avoid repeated translation, thus saving computing resources in the AR translation process and improving translation efficiency to a certain extent.

[0106] Figure 1 Schematic diagram of system architecture involved in various exemplary embodiments of the present application is shown. Figure 1 As shown, the system architecture includes an electronic device 1 and a target 2 to be translated.

[0107] like Figure 1 As shown in (a), the electronic device 1 includes a camera 11, which can be used to capture an image of a target to be translated 2. After performing text recognition, extraction, translation and other processes on the image, the electronic device 1 generates virtual digital content.

[0108] like Figure 1 As shown in (b), the electronic device 1 also includes a display screen 12, such as a touch screen, a flexible screen, a curved screen, etc. The display screen 12 can be used to display an image of the target 2 to be translated, and to superimpose virtual digital content on the image of the target 2 to achieve a three-dimensional display effect.

[0109] It should be noted that the electronic device in the embodiments of the present application may be a handheld terminal (such as a mobile phone), or may be an AR pair of glasses, or any other electronic device that meets actual usage requirements, and the embodiments of the present application are not specifically limited.

[0110] In an embodiment of the present application, during AR translation, the relative position change between the electronic device and the target to be translated can be determined based on visual positioning tracking, and the collected image information can be extracted for text, translated into the target language, and the position of the text in the image coordinate system can be determined. Then, the translation result is rendered into AR digital content and superimposed on the corresponding position of the original image. Then, the relative position of the current device and the photographed object, such as angle and position, is continuously tracked and located. If changes occur and exceed a preset threshold, the entire translation and display process needs to be repeated. Otherwise, the previously rendered augmented reality / digital content is adjusted accordingly according to the changed position and display content so that it accurately fits the content corresponding to the current image.

[0111] The execution subject of the processing method of AR translation provided in the embodiment of the present application can be the above-mentioned electronic device, or it can be a functional module and / or functional entity in the electronic device that can implement the processing method of AR translation, and the present application scheme can be implemented by hardware and / or software, which can be determined according to actual use requirements, and the embodiment of the present application is not limited. Taking an electronic device as an example, the processing method of AR translation provided in the embodiment of the present application is exemplarily described in combination with the accompanying drawings.

[0112] Figure 2 is a flowchart of a processing method for AR translation provided in an embodiment of the present application. Figure 2 As shown, the method includes the following steps S101-S106.

[0113] S101, collecting a first image through a camera of an electronic device, where the first image includes a first text to be translated.

[0114] In the embodiment of the present application, when AR translation is performed through an electronic device, the electronic device first needs to turn on the AR translation function, turn on the camera, and collect images. Figure 3 An exemplary description is given.

[0115] like Figure 3 As shown in (a), in response to the user's triggering operation, the electronic device 1 can start the translation application and display the interface 20 of the translation application. The source language option and the target language option are displayed in the interface 20, and the user can select the source language type and the target language type to be translated according to the actual use requirements. For the sake of convenience, the translation between English and Chinese is used as an example for exemplary explanation. A control 21 is also displayed in the interface 20, which can be used to trigger the start of the AR translation function.

[0116] The user's triggering operation may be to open the translation application by clicking, touching, sliding, or shaking, or by voice control or other means, which is not limited in this application. For example, after the electronic device detects the user's touch operation, the translation function in the application is activated and the camera is activated.

[0117] like Figure 3 As shown in (a), in response to a user operation on the control 21 (eg, a click operation), the electronic device may turn on the camera to capture images and turn on the AR translation function. Figure 3 (b) schematically shows a display interface 22 after the AR translation function is turned on. The display interface 22 displays the language type of the AR translation application, English and Chinese translation. The display interface 22 also displays a viewfinder 23, and the viewfinder 23 displays a prompt message: Please aim at the text, to prompt the user to adjust the position of the electronic device's camera so that the text to be translated is located in the viewfinder 23 to facilitate image capture.

[0118] Optionally, in an embodiment of the present application, the electronic device can capture images at a preset frequency. For example, the preset frequency may be 30 frames of images per second, or 10 frames of images per second, or other frequencies that meet the requirements. The specific frequency of capturing images can be determined based on actual usage requirements and is not limited in the embodiment of the present application.

[0119] S102: Translate a first text to be translated to obtain a first translation result.

[0120] In the embodiment of the present application, since the text content in the first image appears for the first time, the text content that appears for the first time needs to be translated.

[0121] Optionally, in some embodiments, the electronic device may call a local translation engine to translate the text to be translated and obtain a translation result. Here, the translation engine may be a translation application for text translation installed in the electronic device. In other embodiments, when the electronic device is connected to the Internet, the electronic device may send the text to be translated to a server for text translation, and the server translates the text to be translated and sends the translation result to the electronic device, so that the electronic device obtains the translation result. It should be noted that, for the sake of ease of description, the following is an exemplary description using an electronic device calling a local translation engine to translate a text as an example.

[0122] It is understandable that in this step, the language type of the text to be translated can be translated into another language type using translation software to obtain a translation result. For the specific method of translating the text to be translated in this step, it can be implemented in a manner identical or similar to the language translation method in the prior art. For example, when the text information to be translated is an English word, the English word can be input into the translation software, and the translation software is used to obtain the Chinese interpretation and English pronunciation corresponding to the English word, i.e., the translation result. As long as the language type of the text information to be translated can be translated into the target language type required by the user, and the translation result is obtained, the embodiment of the present application does not impose any restrictions on this.

[0123] In an embodiment of the present application, after the electronic device captures the first image, the electronic device may first perform text recognition on the first image to obtain a first text to be translated. Further, the electronic device may call a translation engine to translate the first text to be translated to obtain a first translation result. Further, the electronic device may render the translation result of the first text to be translated to obtain a first virtual image.

[0124] Exemplarily, the electronic device may use OCR technology to perform text recognition on the collected image. Among them, OCR technology can generally include a text detection step and a text recognition step, wherein the text detection step is mainly used to locate the position of the text, and the text recognition step is mainly used to recognize the specific content of the text. In other words, text detection can be performed first to locate the position of the text, and then text recognition can be performed to obtain the specific content of the text. The detailed steps of text recognition using OCR technology can be referred to the implementation steps in the prior art, which will not be repeated here.

[0125] That is to say, text is extracted from the input information stream through OCR, and then the translation engine is called to translate it into target text information to generate digital content.

[0126] S103: Display a first image on a display screen of an electronic device, and display a first virtual image superimposed on the first image, where the first virtual image includes a first translation result.

[0127] In this way, AR technology is used to superimpose the corresponding translation results on the text to be translated, presenting a three-dimensional display effect.

[0128] S104: Capture a second image through a camera, where the second image contains a second text to be translated.

[0129] In the embodiment of the present application, when an image is collected again, the feature similarity between the currently collected image and the previously collected image is first compared with the preset similarity threshold, and the posture change of the electronic device is compared with the preset posture threshold, and then according to the comparison results of the feature similarity and the comparison results of the posture change, it is determined whether the text in the currently collected image needs to be fully translated, partially translated, or not translated, so that different translation trigger strategies are selected to complete the translation process. The specific translation trigger strategy can be found in the following description of S105A, S105B and S105C.

[0130] It should be noted that steps S105A, S105B and S105C are performed one by one.

[0131] S105A, if the posture change of the electronic device is less than the preset posture threshold, and the feature similarity between the second image and the first image is greater than or equal to the preset similarity threshold, obtain the first translation result as the translation result of the second text to be translated.

[0132] For example, in a scenario where the mobile phone does not move and the reading content does not change, it can be determined that the posture change is small and the text to be translated has not changed. In other words, the text has been translated before and has corresponding translation results. Therefore, the text does not need to be translated again using the translation engine, and the previous translation results can be directly obtained, which can greatly save computing resources.

[0133] In this case, the text in the second image does not need to be translated, there is no need to perform text recognition on the second image, there is no need to call a translation engine for translation, and there is no need to render the translation result.

[0134] S105B, if the posture change of the electronic device is less than a preset posture threshold, and the feature similarity is less than a preset similarity threshold, calling a translation engine to translate the second text to be translated.

[0135] For example, in a scenario where the phone does not move and pages are turned for reading, it can be determined that the posture changes slightly, but the text to be translated changes significantly, that is, the text has not been translated before and has no corresponding translation result, so all text in the current frame image needs to be translated.

[0136] In this case, the text in the second image needs to be translated, and the electronic device can use OCR technology to perform text recognition on the second image to obtain a second text to be translated. Further, the electronic device can call a translation engine to translate the second text to be translated to obtain a second translation result. Further, after the electronic device obtains the translation result of the second text to be translated, the electronic device can render the translation result of the second text to be translated to obtain a second virtual image.

[0137] S105C: If the posture change of the electronic device is greater than or equal to the preset posture threshold, and the feature similarity is less than the preset similarity threshold, calling the translation engine to translate part or all of the second text to be translated.

[0138] For example, for a scene where the mobile phone moves greatly (such as translation, rotation, stretching, shaking, etc.), it can be determined that the posture has changed greatly and the content to be translated has changed, which may be all changed or only part of it. Therefore, all or part of the text in the current frame image needs to be translated.

[0139] When the posture change of the electronic device is greater than or equal to the preset posture threshold, and the feature similarity is less than the preset similarity threshold, there may be the following two situations, and there are two corresponding implementation methods:

[0140] Case 1: When the first part of the second text to be translated is the same as the first text to be translated, the first translation result is obtained as the translation result of the first part of the text; and the translation engine is called to translate the second part of the text to obtain the translation result of the second part of the text, which is the text in the second text to be translated except the first part of the text. The translation result of the second text to be translated includes the translation result of the first part of the text and the translation result of the second part of the text.

[0141] In the above situation 1, if the current frame image and the previous frame image have some overlapping or repeated text content, the text of the overlapping or repeated part does not need to be translated, and the translation result does not need to be rendered.

[0142] Case 2: When the second text to be translated does not have the same text as the first text to be translated, the translation engine is called to translate all the text in the second text to be translated to obtain the translation result of the second text to be translated.

[0143] In the above situation 2, if there is no overlap or duplication of text content between the current frame image and the previous frame image, all texts in the current frame image need to be translated.

[0144] Optionally, in an embodiment of the present application, the electronic device may extract feature points in the first image and extract feature points in the second image. Then, the electronic device performs feature comparison on the feature points in the second image and the feature points in the first image to obtain feature similarity between the second image and the first image.

[0145] In this way, feature extraction and matching are performed on the collected continuous multiple frames of images to determine the number of feature matches or feature similarity between the continuous multiple frames of images. The more feature matches there are or the greater the feature similarity is, the more similar the text contents of the two frames of images are, and the same text content may not need to be translated.

[0146] Optionally, in an embodiment of the present application, the electronic device may use a SLAM method to generate a SLAM map, and determine the posture information and posture change of the electronic device based on the second image, the measurement data of the target sensor in the electronic device, and the SLAM map. Among them, the posture information includes position information and attitude information, and the posture change includes position change and attitude change. By tracking the position and attitude of the electronic device in real time through the SLAM method, the posture change of the electronic device can be accurately calculated.

[0147] Optionally, in the embodiment of the present application, the target sensor can be an inertial measurement unit (IMU), an acceleration sensor, a gyroscope sensor, or any other sensor that meets the usage requirements, such as a proximity light sensor or a Hall sensor. The specific sensor can be determined based on actual usage requirements and is not limited in the embodiment of the present application.

[0148] For example, Figure 4 As shown, the target sensor can determine the position and position change of the electronic device 1 by detecting the translation distance in the three directions of x, y, and z in the coordinate system. The target sensor can determine the posture and posture change of the electronic device 1 by detecting the rotation angle in the three directions of x, y, and z in the coordinate system.

[0149] In actual implementation, when the electronic device moves, the SLAM system in the electronic device creates a SLAM map, and the sensor in the electronic device collects the motion data of the electronic device in real time. The electronic device can jointly determine the final posture and posture change of the electronic device based on the posture estimated by the SLAM map and the posture estimated by the motion data collected by the sensor.

[0150] It should be noted that when the electronic device detects that the position change is less than the preset position threshold and the posture change is less than the preset position threshold, it can be considered that the posture change of the electronic device is less than the preset posture threshold.

[0151] It should also be noted that when the electronic device detects that the position change is greater than or equal to the preset position threshold, and the posture change is greater than or equal to the preset position threshold, it can be considered that the position change of the electronic device is greater than or equal to the preset posture threshold. Alternatively, when the electronic device detects that the position change is greater than or equal to the preset position threshold, it can also be considered that the position change of the electronic device is greater than or equal to the preset posture threshold. Alternatively, when the electronic device detects that the posture change is greater than or equal to the preset posture threshold, it can also be considered that the position change of the electronic device is greater than or equal to the preset posture threshold.

[0152] As can be seen from the above, the embodiment of the present application provides a reasonable translation triggering strategy: a comprehensive analysis is performed based on the results of the SLAM tracking and positioning of the electronic device and the content similarity comparison results of the collected images to determine whether the text in the current frame image needs to be fully translated, partially translated, or not translated. This can effectively avoid repeated translations, thereby saving computing resources in the AR translation process and improving translation efficiency to a certain extent.

[0153] S106, displaying the second image on the display screen, and displaying a second virtual image superimposed on the second image, wherein the second virtual image includes a translation result of the second text to be translated.

[0154] Optionally, in an embodiment of the present application, the electronic device may use a SLAM method to determine a target virtual plane for displaying AR digital content, where the target virtual plane is located above the plane where the captured image is located. In this case, the electronic device may display a second virtual image on the target virtual plane, thereby realizing superimposing the second virtual image on the second image.

[0155] For example, taking a mobile phone as an example, a mobile phone can usually analyze the position and posture of the mobile phone in space relative to the target to be translated, and build a three-dimensional model of the spatial environment. Even if the position and posture of the mobile phone change, the AR digital content can be accurately superimposed and displayed on the image corresponding to the target to be translated.

[0156] In the embodiment of the present application, the specific implementation of step S106 is: the electronic device can update the first image and the first virtual image displayed on the display screen to a second image and a second virtual image, and the second virtual image is superimposed on the second image.

[0157] It should be noted that, in the case of S105A above, when it is detected that the position change of the electronic device is small and the content to be translated has not changed, there is no need to translate again, and the previous translation result can be directly obtained. That is to say, in this case, the second image and the first image displayed on the display screen are substantially images with the same content, and the second virtual image superimposed on the second image can be the same image as the first virtual image.

[0158] It can be seen from the above scheme that the AR scene can include a plane where the captured image (including the text to be translated) is located and a plane where the virtual image (including the translation result) is located. These two images are superimposed and displayed to achieve a three-dimensional display effect.

[0159] Optionally, in the embodiment of the present application, the electronic device may determine a target projection area of ​​the second text to be translated on the target virtual plane, so that the electronic device may display the second virtual image in the target projection area of ​​the target virtual plane.

[0160] In this way, by real-time tracking and mapping the position of the text to be translated on the virtual plane, the position for displaying the translation result can be accurately located, which can ensure that the text to be translated and the translation result correspond in position in real time, thereby improving the AR display effect and facilitating user reading.

[0161] Optionally, in an embodiment of the present application, the electronic device may determine a first rectangular area occupied by the second text to be translated in the second image, and determine two anchor points mapped to the target virtual plane based on two endpoints on the diagonal of the first rectangular area, and determine a second rectangular area on the target virtual plane with the two anchor points as diagonals. The second rectangular area is the target projection area of ​​the second text to be translated on the target virtual plane.

[0162] It can be seen from the above scheme that by mapping the rectangular area where the text to be translated is located onto the virtual plane, the rectangular display area on the virtual plane for displaying the translation result can be determined. This ensures that the text to be translated and the translation result correspond in position in real time, thereby improving the AR display effect and facilitating user reading.

[0163] Optionally, in an embodiment of the present application, the electronic device can set the transparency of the target projection area to be less than or equal to a preset transparency threshold. Transparency is also called transmittance, and the smaller the transparency, the more obvious the shielding effect. The preset transparency threshold can be 30%, 10%, or 0%, and 0% represents complete opacity. It can be determined according to actual use requirements and is not limited in the embodiment of the present application.

[0164] In actual implementation, the transparency of the target projection area is set to a smaller value, making the target projection area opaque or semi-transparent, so that the translation result can be displayed above the corresponding text to be translated, and the translation result can cover or block the text to be translated, avoiding the visual interference or impact of the text to be translated on the virtual displayed translation result, which is more convenient for users to read and improves the user experience.

[0165] Through the above scheme, the translation result of the text to be translated can be rendered into virtual digital content, so that the virtual digital content can be displayed on a virtual plane, thereby presenting an AR display effect.

[0166] The embodiment of the present application adds a positioning tracking function to accurately match the rendered translation result with the original position, and presents the content under the perspective as the relative position changes. When the user subsequently rotates the electronic device, zooms in or out, and performs other posture changes, it can present accurate display effects, such as font size, fit, and the difference between the smeared area and the background.

[0167] In order to have a more overall and intuitive understanding, let's combine Figure 5 The system block diagram shown provides an overall and detailed description of the processing method for AR translation provided in the embodiment of the present application.

[0168] like Figure 5 As shown, the system block diagram includes the following steps S201-S214, which can be divided into five parts: initializing SLAM and OCR engines, SLAM posture tracking, translation triggering strategy, text extraction / translation / generation of digital content, and digital content anchoring display.

[0169] S201, initialize SLAM and OCR engines.

[0170] Assuming that the electronic device has a SLAM function and an OCR function, when the AR translation function is turned on, the electronic device starts the SLAM function and the OCR function. In addition, the electronic device turns on a camera to collect images.

[0171] S202, tracking the position and posture of the electronic device through SLAM.

[0172] Among them, in the SLAM posture tracking process, feature extraction and matching, posture tracking and map construction are realized respectively.

[0173] Through feature extraction and matching, the feature matching number (denoted as N1) of the collected previous and next frame images can be obtained.

[0174] Through posture tracking, the posture information of the electronic device can be obtained, and the posture information includes the current position and posture of the electronic device, as well as the position change and posture change.

[0175] Optionally, in an embodiment of the present application, the posture change information of the electronic device can be estimated by extracting feature points and matching the feature points. Specifically, firstly, the feature points of the two key frames before and after are extracted, and then the feature points are matched. After matching the feature points, two one-to-one corresponding pixel point sets can be obtained. Next, the posture change information of the electronic device is determined based on the two sets of matched pixel point sets.

[0176] Through map construction, a virtual digital plane can be generated to display the translation results.

[0177] In this way, feature extraction and matching are performed through the collected images and data collected by the IMU sensor. Through the tracking data at the front end and the map data at the back end, the position and posture of the current frame image are calculated, and a virtual digital plane is generated.

[0178] Then, the translation trigger strategy is executed, and the position information, virtual digital plane, and feature matching number obtained above are used for discriminant analysis to determine whether to translate all, part, or no translation is required. The specific judgment process is shown in the following steps S203-S207.

[0179] S203, determine whether the feature matching number N1 is less than a threshold value (denoted as N), and whether the posture change (denoted as P1) is less than a threshold value (denoted as P).

[0180] Among them, when the position change and the posture change are both less than the corresponding threshold, it can be considered that the posture change is less than the threshold.

[0181] If yes, execute the following S204; if no, execute the following S205.

[0182] S204: If N1 is smaller than N and P1 is smaller than P, it means that the content to be translated in the image has changed, and therefore all the text in the image needs to be translated.

[0183] S205, determine whether N1 is less than N, and whether P1 is greater than or equal to P.

[0184] Among them, when the position change and / or posture change is greater than or equal to the corresponding threshold, it can be considered that the posture change is greater than or equal to the threshold.

[0185] If yes, execute the following S206; if no, execute the following S207.

[0186] S206: If N1 is less than N and P1 is greater than or equal to P, it means that the posture of the electronic device has changed significantly, so part or all of the text in the image needs to be translated.

[0187] Case 1: If the current image has the same or repeated content as the previously captured image, the repeated content does not need to be translated, but the other parts need to be translated.

[0188] Case 2: If there is no duplicate content, all translation is required.

[0189] S207, if N1 is greater than or equal to N and P1 is less than P, it means that the posture change of the electronic device is small, and the content to be translated has not changed, so the text in this frame image does not need to be translated.

[0190] After S204 and S206 , the following S208 is executed.

[0191] S208, calling the OCR engine and using the OcrLoop program to extract text from the image.

[0192] Among them, the purpose of text extraction can be achieved through steps such as image preprocessing, text detection and text recognition.

[0193] During the image preprocessing process, the collected images may be preprocessed by, for example, binarization, denoising, and tilt angle detection and correction.

[0194] During text detection, the location of the text area can be obtained using the traditional connected region algorithm or a deep learning-based algorithm.

[0195] During the text recognition process, the content in the text area can be obtained based on the deep learning OCR algorithm.

[0196] In combination with the above S204, all the text in the image needs to be extracted. In combination with the above S206, part or all of the text in the image needs to be extracted.

[0197] S209, calling back the onOcrComplete program to complete the text extraction.

[0198] S210, calling a translation engine to translate the recognized text.

[0199] This process mainly involves calling a translation engine to translate the text content extracted in S209 into a target language.

[0200] S211, rendering the translation result into digital content.

[0201] The translation result obtained in S210 is converted to generate digital content that can be virtually displayed. In actual implementation, the digital content can be a virtual image.

[0202] In this way, through S208 - S211 , text extraction, translation and processing of the translation results into digital content are completed.

[0203] S212, obtaining the position of the digitized content on the current image, for example, two endpoints on the diagonal line of the rectangle where the digitized content is located may be selected.

[0204] S213, two rays perpendicular to the rectangular surface are drawn from the two end points respectively, intersecting with the virtual plane to obtain two anchor points. The two anchor points can be used to determine the placement position and display size of the digital content on the virtual plane.

[0205] S214, rendering and displaying the final result on the virtual plane.

[0206] Furthermore, S202 - S214 are repeatedly executed to perform AR translation.

[0207] Through the processing method of AR translation provided by the embodiment of the present application, in the scenario of using electronic equipment for AR translation, the posture change of the electronic equipment can be detected in real time, and the feature matching of the continuous multiple frames of images collected by the camera can be performed, and then according to the posture change of the electronic equipment and the feature matching, it can be judged whether the text to be translated needs to be fully translated, partially translated, or not translated, so as to select the corresponding translation trigger strategy. This can effectively avoid repeated translation, thus saving computing resources in the AR translation process and improving translation efficiency to a certain extent.

[0208] Next, combine Figures 6 to 9 , exemplarily illustrating the translation display results in different situations of the relative posture of the electronic device and the target to be translated in the AR translation scenario of an embodiment of the present application.

[0209] Figure 6 A schematic diagram of the interactive interface in a scenario where an electronic device captures images and extracts text for translation during AR translation is shown.

[0210] like Figure 6 As shown in (a), the electronic device 1 turns on the AR translation function, the camera is turned on, and the image is collected by aiming at the target 1 to be translated. The target 1 to be translated contains the text 24 "Cease to struggle and you cease to live". The electronic device 1 extracts the text from the collected image, and calls the translation engine for translation, and obtains the translation result "Life never stops, struggle never stops". Further, the translation result is rendered into digital content to form a virtual image. Figure 6 As shown in (b), the electronic device 1 displays the collected physical image 25 on the display screen, and overlays and displays a virtual image 26 corresponding to the translation result on the physical image 25.

[0211] The virtual image 26 may cover or block the text area when displayed, and graphics other than the text in the physical image 25 may be displayed normally without being blocked by the virtual image 26 .

[0212] Figure 7The schematic diagram of the interactive interface in a scenario where the image content changes and the content is repeated after the posture of the electronic device changes during AR translation is shown. In this scenario, the translated part of the text to be translated does not need to be translated.

[0213] like Figure 7 As shown in (a), the electronic device 1 captures an image of the target 1 to be translated, and the image of the target 1 to be translated contains text 24. After the above-mentioned text extraction, translation and generation of digital content, a three-dimensional image of the superposition of the physical image 25 and the virtual image 26 is displayed on the display screen of the electronic device 1. The virtual image 26 is the translation result corresponding to the text 24.

[0214] The electronic device 1 continues to collect images. At this time, it is detected that the electronic device 1 is translated in a certain direction and the distance from the target 1 to be translated is increased, and the image content of the target 1 to be translated collected by the electronic device 1 changes, such as Figure 7 As shown in (b), the image of the target to be translated 1 contains not only text 24, but also text 27 "Good luck!". Since text 24 has been translated, it does not need to be translated. At this time, only text 27 needs to be translated. Figure 7 As shown in (b), the electronic device 1 displays the collected physical image 28 on the display screen, and superimposes a virtual image 26 corresponding to the translation result of the text 24 and a virtual image 29 corresponding to the translation result of the text 27 ("Good luck!") on the physical image 28.

[0215] Figure 8 The diagram shows the interactive interface in a scenario where the posture of the electronic device changes little during AR translation, the image content changes, and the content is not repeated. In this scenario, the text to be translated does not contain the translated text, and all of it needs to be translated.

[0216] like Figure 8 As shown in (a), the electronic device 1 captures an image of the target 1 to be translated, and the image of the target 1 to be translated contains text 24. After the above-mentioned text extraction, translation and generation of digital content, a three-dimensional image of the superposition of the physical image 25 and the virtual image 26 is displayed on the display screen of the electronic device 1. The virtual image 26 is the translation result corresponding to the text 24.

[0217] The electronic device 1 continues to collect images. At this time, it is detected that the position of the electronic device 1 remains unchanged or changes slightly, but the image content of the target to be translated 1 collected by the electronic device 1 changes, such as Figure 8 As shown in (b), the image of the target to be translated 1 changes from text 24 to text 30. Since text 30 is content that has not been translated, it needs to be translated. Figure 8As shown in (b), the electronic device 1 displays the collected physical image 31 on the display screen, and superimposes and displays a virtual image 32 corresponding to the translation result of the text 30 on the physical image 31.

[0218] Fig. 9 The schematic diagram of the interactive interface in a scenario where the posture of the electronic device changes little and the image content remains unchanged during AR translation is shown. In this scenario, the text to be translated is already translated and does not need to be translated.

[0219] like Fig. 9 As shown in (a), the electronic device 1 captures an image of the target 1 to be translated, and the image of the target 1 to be translated contains text 24. After the above-mentioned text extraction, translation and generation of digital content, a three-dimensional image of the superposition of the physical image 25 and the virtual image 26 is displayed on the display screen of the electronic device 1. The virtual image 26 is the translation result corresponding to the text 24.

[0220] The electronic device 1 continues to collect images. At this time, it is detected that the posture of the electronic device 1 remains unchanged or changes slightly, and the image content of the target 1 to be translated collected by the electronic device 1 does not change. Fig. 9 As shown in (b), the image of the target to be translated 1 contains text 24. Since text 24 has been translated, it does not need to be translated. Fig. 9 As shown in (b), the electronic device 1 still displays the collected physical image 25 on the display screen, and superimposes and displays the virtual image 26 corresponding to the translation result of the text 24 on the physical image 25.

[0221] It should be noted that the embodiments of the present application solve the following problems by tracking and locating the device through SLAM and setting a reasonable translation trigger strategy: (1) the translation of digital content drifts and does not fit, and the font format and size jump; and (1) repeated translation causes large terminal algorithm scheduling overhead, increased traffic loss and cloud computing resource usage.

[0222] Regarding the above problem (1): In the prior art, during the AR translation process, when the user's posture of holding or wearing the AR device changes (such as moving, rotating, stretching or shaking, etc.), the translation result may drift and not fit, and the font format and size may jump. Therefore, the 3D translation effect is poor, affecting the user experience.

[0223] This application solution renders the translation result into digital content and directly overlays it on the corresponding position of the original content. When the position of the translation result and the original content changes, the translation result is not mechanically pasted; instead, the position changes of the two are tracked and located. When the relative position change of the two exceeds the preset threshold, the translation is re-done and the new translation result is displayed; when the relative position change of the two is small, the positioning tracking technology is used to accurately overlay the previous translation result on the corresponding position of the current image.

[0224] Compared with the prior art, the embodiments of the present application can accurately locate the display position of the translation result by anchoring the digital content to the SLAM virtual plane, real-time tracking of SLAM, and combining the position / posture of the electronic device, thereby avoiding the drift and mismatch of the translation result and the jump of font format and size to a certain extent.

[0225] Regarding the above question (2): Compared with the real-time frame-by-frame translation method of the prior art, the solution of the present application first analyzes the collected image to determine whether the content in the previous and next frame images has changed, and determines whether a full translation is required. For example, for images with small content changes, translation may not be required. This can reduce the algorithm scheduling overhead and save computing resources.

[0226] The present application scheme locates and tracks the relative posture information of the camera and the photographed object in real time. When the posture change is less than a preset threshold, the first translation result is dynamically superimposed on the current image and displayed without the need for an additional translation process. When the posture change is greater than the preset threshold, image recognition and translation are performed again, and the new translation content is superimposed on the current image.

[0227] Fig.10 The following schematically shows an interface diagram for AR translation using the solution of the present application. Fig.10 As shown, after the AR translation function is turned on in the electronic device, the electronic device captures an image (called the original image), extracts the text content "math" in the image, and translates the text content to obtain a translation result "math"; then, the electronic device superimposes the translation result of the text on the original image to present an AR display effect.

[0228] Based on the above-mentioned embodiments, it can be seen that when AR translation is performed through the present application solution, when the user holds the mobile phone unsteadily, such as when it is shaken and rotated, the accurate translation font size, good fit effect, and small difference between the smeared area and the background can be displayed, and the translation target can be matched in real time. In addition, the algorithm scheduling overhead can be reduced, thereby reducing freezes, time consumption, and power consumption.

[0229] It should also be noted that, in the embodiments of the present application, "greater than" can be replaced by "greater than or equal to", "less than or equal to" can be replaced by "less than", or "greater than or equal to" can be replaced by "greater than", and "less than" can be replaced by "less than or equal to".

[0230] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the protection scope of this application.

[0231] It can be understood that the methods and operations implemented by electronic devices in the above-mentioned various method embodiments can also be implemented by components (such as chips or circuits) that can be used in electronic devices.

[0232] The above describes the method embodiment provided by the present application, and the following describes the device embodiment provided by the present application. It should be understood that the description of the device embodiment corresponds to the description of the method embodiment, so the content not described in detail can be referred to the method embodiment above, and for the sake of brevity, it will not be repeated here.

[0233] The above mainly describes the solution provided by the embodiment of the present application from the perspective of method steps. It is understandable that, in order to achieve the above functions, the electronic device implementing the method includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should be aware that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of the present application.

[0234] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other feasible division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.

[0235] Fig.11 A schematic block diagram of an AR translation processing device 800 provided in an embodiment of the present application. The device 800 can be used to perform the actions performed by the electronic device in the above method embodiment. The device 800 includes an image acquisition unit 810, a processing unit 820 and a display unit 830.

[0236] An image acquisition unit 810 is configured to acquire a first image, wherein the first image includes a first text to be translated;

[0237] The processing unit 820 is used to call a translation engine to translate the first text to be translated to obtain a first translation result;

[0238] A display unit 830 is configured to display the first image on a display screen of the device 800, and to display a first virtual image superimposed on the first image, wherein the first virtual image includes the first translation result;

[0239] The image acquisition unit 810 is further configured to acquire a second image through the camera, wherein the second image includes a second text to be translated;

[0240] The processing unit 820 is further configured to: if the change amount of the posture of the device 800 is less than a preset posture threshold, and the feature similarity between the second image and the first image is greater than or equal to a preset similarity threshold, obtain the first translation result as the translation result of the second text to be translated; or, if the change amount of the posture of the device 800 is less than the preset posture threshold, and the feature similarity is less than the preset similarity threshold, call the translation engine to translate the second text to be translated; or, if the change amount of the posture of the device 800 is greater than or equal to the preset posture threshold, and the feature similarity is less than the preset similarity threshold, call the translation engine to translate part or all of the text in the second text to be translated;

[0241] The display unit 830 is further configured to display the second image on the display screen, and to display a second virtual image superimposed on the second image, wherein the second virtual image includes a translation result of the second text to be translated.

[0242] Through the solution provided by the embodiment of the present application, in the scenario of using electronic devices for AR translation, the posture change of the electronic device can be detected in real time, and the feature matching of the continuous multiple frames of images collected by the camera can be performed, and then according to the posture change of the electronic device and the feature matching, it can be judged whether the text to be translated needs to be fully translated, partially translated, or not translated, so as to select the corresponding translation trigger strategy. This can effectively avoid repeated translation, thereby saving computing resources in the AR translation process and improving translation efficiency to a certain extent.

[0243] For example, in a scenario where the mobile phone does not move and the reading content does not change, it can be determined that the posture changes little and the text to be translated does not change. Therefore, there is no need to translate it again, and the previous translation result can be directly obtained, which greatly saves computing resources.

[0244] As another example, for a scenario where the mobile phone does not move and the pages are turned for reading, it can be determined that the posture changes slightly, but the text to be translated changes significantly, so all text in the current frame image needs to be translated.

[0245] As another example, for a scene where the mobile phone moves with a large amplitude (such as translation, rotation, stretching, shaking, etc.), it can be determined that the posture has changed greatly and the content to be translated has changed, so all or part of the text in the current frame image needs to be translated.

[0246] When the posture change of the electronic device is greater than or equal to the preset posture threshold, and the feature similarity is less than the preset similarity threshold, there may be the following two situations, and there are two corresponding implementation methods:

[0247] In a possible implementation, calling the translation engine to translate part or all of the second text to be translated includes:

[0248] When the first part of the second text to be translated is the same as the first text to be translated, obtaining a first translation result as the translation result of the first part of the text;

[0249] Invoking a translation engine to translate the second part of the text to obtain a translation result of the second part of the text, where the second part of the text is the text other than the first part of the text in the second text to be translated;

[0250] The translation result of the second text to be translated includes the translation result of the first part of the text and the translation result of the second part of the text.

[0251] In the above situation 1, if the text content of the current frame image and the previous frame image partially overlaps or repeats, the text of the overlapping or repeated part does not need to be translated.

[0252] In another possible implementation, calling the translation engine to translate part or all of the second text to be translated includes:

[0253] In the case that the second text to be translated does not have the same text as the first text to be translated, the translation engine is called to translate all the text in the second text to be translated to obtain a translation result of the second text to be translated.

[0254] In the above situation 2, if there is no overlap or duplication of text content between the current frame image and the previous frame image, all texts in the current frame image need to be translated.

[0255] In a possible implementation, the processing unit 820 is further configured to:

[0256] Extracting feature points in the first image and feature points in the second image;

[0257] The feature points in the second image are compared with the feature points in the first image to obtain the feature similarity between the second image and the first image.

[0258] Through the above scheme, the features of the collected continuous multiple frames of images can be extracted and matched to determine the number of feature matches or feature similarity between the continuous multiple frames of images, so as to judge whether the text in the current image needs to be fully translated, partially translated, or not translated according to the feature similarity in the following text.

[0259] In a possible implementation, the processing unit 820 is further configured to:

[0260] The SLAM method of simultaneous positioning and map construction is used to generate SLAM maps;

[0261] The position change of the electronic device is determined according to the second image, the measurement data of the target sensor in the electronic device and the SLAM map.

[0262] Among them, the above-mentioned target sensor includes an IMU sensor; of course, the target sensor can also include any other sensor that meets actual usage requirements, such as a gyroscope sensor or an acceleration sensor, which is not limited in this application.

[0263] In a possible implementation, the above-mentioned posture change includes a position change and a posture change.

[0264] By using the SLAM method to track the position and posture of electronic devices in real time, the position and posture changes of electronic devices can be accurately calculated.

[0265] In a possible implementation, the processing unit 820 is further configured to:

[0266] A target virtual plane for displaying AR digital content is determined by using a SLAM method, where the target virtual plane is located above the plane where the captured image is located;

[0267] The above-mentioned superimposing and displaying the second virtual image on the second image includes:

[0268] The second virtual image is displayed on the target virtual plane.

[0269] It can be seen from the above scheme that the AR scene can include a plane where the captured image (including the text to be translated) is located and a plane where the virtual image (including the translation result) is located. These two images are superimposed and displayed to achieve a three-dimensional display effect.

[0270] In a possible implementation, the processing unit 820 is further configured to:

[0271] Determining a target projection area of ​​the second text to be translated on the target virtual plane;

[0272] The step of displaying the second virtual image on the target virtual plane includes:

[0273] A second virtual image is displayed in the target projection area.

[0274] It can be seen from the above scheme that by real-time tracking and mapping the position of the text to be translated on the virtual plane, the position for displaying the translation result can be accurately located, which can ensure that the text to be translated and the translation result correspond in position in real time, thereby improving the AR display effect and facilitating user reading.

[0275] In a possible implementation, the step of determining a target projection area of ​​the second text to be translated on the target virtual plane includes:

[0276] Determine a first rectangular area occupied by a second text to be translated in the second image;

[0277] According to two endpoints on the diagonal line of the first rectangular area, determine two anchor points mapped to the two endpoints on the target virtual plane;

[0278] Determine a second rectangular area on the target virtual plane with the two anchor points as diagonals;

[0279] The second rectangular area is a target projection area of ​​the second text to be translated on the target virtual plane.

[0280] It can be seen from the above scheme that by mapping the rectangular area where the text to be translated is located onto the virtual plane, the rectangular display area on the virtual plane for displaying the translation result can be determined. This ensures that the text to be translated and the translation result correspond in position in real time, thereby improving the AR display effect and facilitating user reading.

[0281] In a possible implementation, the processing unit 820 is further configured to set the transparency of the target projection area to be less than or equal to a preset transparency threshold.

[0282] Through the above scheme, the transparency of the target projection area can be reduced, making the target projection area opaque or semi-transparent, so that the translation result can be displayed above the corresponding text to be translated, and the text to be translated can be covered or blocked, so as to avoid the text to be translated from causing visual interference or influence on the virtual displayed translation result, thereby making it easier for users to read and improving the user experience.

[0283] In a possible implementation, the processing unit 820 is further configured to:

[0284] Performing text recognition on the first image to obtain a first text to be translated; and performing text recognition on the second image to obtain a second text to be translated.

[0285] For example, OCR technology may be used for text recognition.

[0286] In a possible implementation, the processing unit 820 is further configured to:

[0287] The translation result of the first text to be translated is rendered to obtain a first virtual image; and the translation result of the second text to be translated is rendered to obtain a second virtual image.

[0288] Through the above scheme, the translation result of the text to be translated can be rendered into virtual digital content, so that the virtual digital content can be displayed on a virtual plane, thereby presenting an AR display effect.

[0289] According to the embodiment of the present application, the device 800 may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of the units in the device 800 are respectively for implementing the corresponding processes of the method, which will not be repeated here for the sake of brevity.

[0290] Fig.12 9 is a structural schematic diagram of an electronic device 900 provided in an embodiment of the present application. The electronic device 900 may include a processor 910, an external memory interface 920, an internal memory 921, a universal serial bus (USB) interface 930, a charging management module 940, a power management unit 941, a battery 942, an antenna 1, an antenna 2, a mobile communication module 950, a wireless communication module 960, an audio module 970, a speaker 970A, a receiver 970B, a microphone 970C, an earphone interface 970D, a sensor module 980, a button 990, a motor 991, an indicator 992, a camera 993, a display screen 994, and a subscriber identification module (SIM) card interface 995, etc. The sensor module 980 may include a pressure sensor 980A, a gyroscope sensor 980B, an air pressure sensor 980C, a magnetic sensor 980D, an acceleration sensor 980E, a distance sensor 980F, a proximity light sensor 980G, a fingerprint sensor 980H, a temperature sensor 980I, a touch sensor 980J, an ambient light sensor 980K and a bone conduction sensor 980L, etc.

[0291] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 900. In other embodiments of the present application, the electronic device 900 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0292] The processor 910 may include one or more processing units, for example: the processor 910 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. Among them, the controller can be the nerve center and command center of the electronic device 900. The controller can generate an operation control signal (such as a control signal for image acquisition, a control signal for text extraction, a control signal for text translation, a control signal for image rendering and display, etc.) according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0293] A memory may also be provided in the processor 910 for storing instructions and data, for example, for storing image data collected during the AR translation process and text translation results. In some embodiments, the memory in the processor 910 is a cache memory. The memory may store instructions or data that the processor 910 has just used or circulated. If the processor 910 needs to use the instruction or data again, it may be called directly from the memory. Repeated access is avoided, the waiting time of the processor 910 is reduced, and the efficiency of the system is improved.

[0294] The processor 910 can be used to execute the above program code and call related modules to implement the AR translation function of the electronic device in the embodiment of the present application.

[0295] In some embodiments, the processor 910 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a general-purpose input / output (GPIO) interface, and / or a universal serial bus (USB) interface, etc. It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 900. In other embodiments of the present application, the electronic device 900 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0296] The electronic device 900 implements the display function through a GPU, a display screen 994, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 994 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 910 may include one or more GPUs, which execute program instructions to generate or change display information.

[0297] The display screen 994 is used to display images or videos, for example, for displaying AR translation results. The display screen 994 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 900 may include 1 or N display screens 994, where N is a positive integer greater than 1.

[0298] The electronic device 900 can realize the shooting function through the ISP, the camera 993, the video codec, the GPU, the display screen 994 and the application processor.

[0299] ISP is used to process data fed back by camera 993. For example, when collecting AR images, the shutter is opened, and light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, and converts it into a real-life image and a virtual image visible to the naked eye. ISP can also perform algorithmic optimization on the noise, brightness, and skin color of the image. ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, ISP can be set in camera 993.

[0300] The camera 993 is used to capture static images or videos, for example, it is used to collect images of text to be translated during AR translation. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 900 may include 1 or N cameras 993, where N is a positive integer greater than 1.

[0301] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. The NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. The NPU can realize applications such as intelligent cognition of the electronic device 900, such as image recognition, text recognition, and text understanding.

[0302] The external memory interface 920 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 900. The external memory card communicates with the processor 910 through the external memory interface 920 to implement a data storage function. For example, the image data collected during the AR translation process and the text translation results and other files are saved in the external memory card.

[0303] The internal memory 921 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 910 executes various functional applications and data processing of the electronic device 900 by running the instructions stored in the internal memory 921. The internal memory 921 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as an AR translation function, etc.), etc. The data storage area may store data created during the use of the electronic device 900 (such as AR-related image data, etc.), etc. In addition, the internal memory 921 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0304] The pressure sensor 980A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 980A can be set on the display screen 994. There are many types of pressure sensors 980A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 980A, the capacitance between the electrodes changes. The electronic device 900 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 994, the electronic device 900 detects the touch operation intensity according to the pressure sensor 980A. The electronic device 900 can also calculate the position of the touch based on the detection signal of the pressure sensor 980A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the AR translation application icon, an instruction to open the AR translation application is executed.

[0305] The gyroscope sensor 980B can be used to determine the motion posture of the electronic device 900. In some embodiments, the angular velocity of the electronic device 900 around three axes (such as the x, y, and z axes) can be determined by the gyroscope sensor 980B to determine the posture of the electronic device 900. The gyroscope sensor 980B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 980B detects the angle of the electronic device 900 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 900 through reverse movement to achieve anti-shake. The gyroscope sensor 980B can also be used for navigation and somatosensory game scenes.

[0306] The acceleration sensor 980E can detect the magnitude of acceleration in all directions (generally three axes) of the electronic device 900. When the electronic device 900 is stationary, it can detect the magnitude and direction of gravity, and can also be used to detect the posture of the electronic device, which is applied to applications such as horizontal and vertical screen switching.

[0307] The distance sensor 980F is used to measure the distance. The electronic device 900 can measure the distance by infrared or laser. In some embodiments, in the image acquisition scene during AR translation, the electronic device 900 can use the distance sensor 980F to measure the distance to achieve fast focusing.

[0308] The proximity light sensor 980G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode may be an infrared light-emitting diode. The electronic device 900 emits infrared light outward through the light-emitting diode. The electronic device 900 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 900. When insufficient reflected light is detected, the electronic device 900 can determine that there is no object near the electronic device 900. The electronic device 900 can use the proximity light sensor 980G to detect the distance between the electronic device 900 and the target to be translated and the distance change.

[0309] The ambient light sensor 980K is used to sense the brightness of the ambient light. The electronic device 900 can adaptively adjust the brightness of the display screen 994 according to the perceived brightness of the ambient light. The ambient light sensor 980K can also be used to automatically adjust the white balance during image acquisition.

[0310] The magnetic sensor 980D includes a Hall sensor. The electronic device 900 can use the magnetic sensor 980D to detect the displacement of the electronic device 900. In some embodiments, the Hall sensor can use a magnet to form a linear trapezoidal magnetic field (or called a ramp magnetic field). The displacement change of the Hall plate in the linear magnetic field is consistent with the change in magnetic field strength. The Hall potential formed is proportional to the displacement. The electronic device 900 obtains the Hall potential and can measure the displacement of the electronic device 900.

[0311] The touch sensor 980J is also called a "touch panel". The touch sensor 980J can be set on the display screen 994, and the touch sensor 980J and the display screen 994 form a touch screen, also called a "touch screen". The touch sensor 980J is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 994. In other embodiments, the touch sensor 980J can also be set on the surface of the electronic device 900, which is different from the position of the display screen 994.

[0312] The button 990 includes a power button, a volume button, an AR translation control, etc. The button 990 may be a mechanical button. It may also be a touch button. The electronic device 900 may receive a button input and generate a key signal input related to the user settings and function control of the electronic device 900. For example, when the electronic device receives a user input to an AR translation control, the electronic device 900 may generate an instruction to trigger the camera to turn on and enable the AR translation function.

[0313] Motor 991 can generate vibration prompts. Motor 991 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as AR translation applications, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 994, motor 991 can also correspond to different vibration feedback effects. Different application scenarios can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0314] Optionally, the electronic device 900 may be a mobile terminal or a non-mobile terminal. Exemplarily, the electronic device 900 may be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), a wireless headset, a wireless bracelet, wireless smart glasses, a wireless watch, an augmented reality (AR) / virtual reality (VR) device, such as an AR helmet or AR glasses, etc. The embodiment of the present application does not specifically limit the device type of the electronic device 900.

[0315] It should be understood that Fig.12 The electronic device 900 shown may correspond to Fig.11 The device 800 shown. Fig.12 The processor 910, the display screen 994, and the camera 993 in the electronic device 900 shown in FIG. 9 may correspond to Fig.11 The processing unit 820, the display unit 830, and the image acquisition unit 810 in the device 800.

[0316] In actual implementation, when the electronic device 900 is running, the processor 910 executes the computer execution instructions in the memory 921 to perform the operation steps of the above method through the electronic device 900.

[0317] Optionally, in some embodiments, the present application provides a chip, which is coupled to a memory, and the chip is used to read and execute computer programs or instructions stored in the memory to execute the methods in the above embodiments.

[0318] Optionally, in some embodiments, the present application provides an electronic device, which includes a chip, and the chip is used to read and execute a computer program or instruction stored in a memory, so that the method in each embodiment is executed.

[0319] Optionally, in some embodiments, the embodiments of the present application further provide a computer-readable storage medium, which stores program code. When the computer program code runs on a computer, the computer executes the methods in the above embodiments.

[0320] Optionally, in some embodiments, the embodiments of the present application further provide a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes the methods in the above embodiments.

[0321] In an embodiment of the present application, the electronic device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. Among them, the hardware layer may include hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system of the operating system layer can be any one or more computer operating systems that implement business processing through processes, such as Linux operating system, Unix operating system, Android operating system, iOS operating system, or Windows operating system. The application layer may include applications such as browsers, address books, word processing software, and instant messaging software.

[0322] The embodiments of the present application do not specifically limit the specific structure of the execution subject of the method provided in the embodiments of the present application, as long as it is possible to communicate according to the method provided in the embodiments of the present application by running a program that records the code of the method provided in the embodiments of the present application. For example, the execution subject of the method provided in the embodiments of the present application may be an electronic device, or a functional module in the electronic device that can call and execute a program.

[0323] Various aspects or features of the present application may be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" as used herein may encompass a computer program accessible from any computer-readable device, carrier, or medium. For example, a computer-readable medium may include, but is not limited to, a magnetic storage device (e.g., a hard disk, a floppy disk, or a magnetic tape, etc.), an optical disk (e.g., a compact disc (CD), a digital versatile disc (DVD), etc.), a smart card, and a flash memory device (e.g., an erasable programmable read-only memory (EPROM), a card, a stick, or a key drive, etc.).

[0324] The various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0325] It should be understood that the processor mentioned in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0326] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM). For example, RAM can be used as an external cache. By way of example and not limitation, RAM may include the following forms: static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0327] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0328] It should also be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0329] Those of ordinary skill in the art will appreciate that the units and steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of this application.

[0330] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0331] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0332] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0333] In addition, each functional unit in each embodiment of the present application may be integrated into one unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0334] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer software product, which is stored in a storage medium. The computer software product includes several instructions that enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium may include, but is not limited to: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0335] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0336] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A processing method for augmented reality (AR) translation, characterized in that: include: Capturing a first image through a camera of an electronic device, wherein the first image includes a first text to be translated; Translating the first text to be translated to obtain a first translation result; Displaying the first image, and displaying a first virtual image superimposed on the first image, wherein the first virtual image includes the first translation result; Capturing a second image through the camera, wherein the second image includes a second text to be translated; Real-time positioning and tracking of the position and posture of the electronic device, and calculation of the position and posture change of the electronic device relative to the photographed object, wherein the position and posture change includes the change of movement, rotation, distance stretching or shaking; If the change in the posture of the electronic device is less than a preset posture threshold, and the feature similarity between the second image and the first image is greater than or equal to a preset similarity threshold, obtaining the first translation result as the translation result of the second text to be translated; If the change in the posture of the electronic device is less than the preset posture threshold, and the feature similarity is less than the preset similarity threshold, translating the second text to be translated; If the change in the posture of the electronic device is greater than or equal to the preset posture threshold, and the feature similarity is less than the preset similarity threshold, translating part or all of the second text to be translated; Displaying the second image, and displaying a second virtual image superimposed on the second image, wherein the second virtual image includes a translation result of the second text to be translated; The translating of part or all of the second text to be translated includes: In the case where the first part of the second text to be translated is the same as the first text to be translated, obtaining the first translation result as the translation result of the first part of the text; translating the second part of the text to obtain the translation result of the second part of the text, wherein the second part of the text is the text in the second text to be translated except the first part of the text; wherein the translation result of the second text to be translated includes the translation result of the first part of the text and the translation result of the second part of the text; In the case that the second text to be translated does not have the same text as the first text to be translated, all texts in the second text to be translated are translated to obtain a translation result of the second text to be translated.

2. The method according to claim 1, characterized in that The method further comprises: Extracting feature points in the first image and feature points in the second image; Feature points in the second image are compared with feature points in the first image to obtain feature similarity between the second image and the first image.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The SLAM method of simultaneous positioning and map construction is used to generate SLAM maps; Determine a change in the position and posture of the electronic device according to the second image, measurement data of the target sensor in the electronic device, and the SLAM map; Wherein, the target sensor includes an inertial measurement unit IMU.

4. The method according to any one of claims 1 to 3, characterized in that The posture change includes a position change and a posture change.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: A target virtual plane for displaying AR digital content is determined by using a SLAM method, wherein the target virtual plane is located above the plane where the captured image is located; The step of superimposing and displaying the second virtual image on the second image includes: The second virtual image is displayed on the target virtual plane.

6. The method according to claim 5, characterized in that The method further comprises: Determining a target projection area of ​​the second text to be translated on the target virtual plane; Wherein, displaying the second virtual image on the target virtual plane includes: The second virtual image is displayed in the target projection area.

7. The method according to claim 6, characterized in that The determining a target projection area of ​​the second text to be translated on the target virtual plane includes: Determine a first rectangular area occupied by the second text to be translated in the second image; According to two endpoints on a diagonal line of the first rectangular area, determining two anchor points to which the two endpoints are mapped on the target virtual plane; Determine a second rectangular area on the target virtual plane with the two anchor points as diagonals; The second rectangular area is a target projection area of ​​the second text to be translated on the target virtual plane.

8. The method according to claim 6, characterized in that The method further comprises: The transparency of the target projection area is set to be less than or equal to a preset transparency threshold.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Performing text recognition on the first image to obtain the first text to be translated; and Perform text recognition on the second image to obtain the second text to be translated.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: Rendering the translation result of the first text to be translated to obtain the first virtual image; and, The translation result of the second text to be translated is rendered to obtain the second virtual image.

11. An electronic device, characterized in that: The electronic device comprises a display screen, a camera and a processor, wherein the processor is coupled to a memory, and the processor is used to execute a computer program or instruction stored in the memory, so that the electronic device implements the method as claimed in any one of claims 1 to 10.

12. A chip system, characterized in that: The chip system is coupled to a memory, and the chip system is used to read and execute a computer program stored in the memory to implement the method as claimed in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 10.

Images