AR-based translation method and AR device

Abstract

The present invention provides an AR-based translation method and an AR device, which can realize partial translation of a whole-page document, better meet the translation needs of users, and improve user experience. The translation method includes: the AR device determines the target gesture operation input by the user, determines the first area to be translated according to the target gesture operation, and translates the text included in the first area to be translated, and outputs the translation result; wherein the first area to be translated is a part of the first area or the whole area, and the first area is the whole area of ​​the first document.

Description

Technical Field

[0001] The present invention belongs to the technical field of augmented reality, and in particular relates to a translation-assisted measure. Background Art

[0002] There are many translation softwares available, which mainly translate the whole document or the whole page document. However, sometimes users just don't know the meaning of some words in the whole page document, in which case the whole page document does not need to be translated. Therefore, the demand for partial translation is proposed. How to achieve partial translation is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present invention provides an AR-based translation method and an AR device, which can realize partial translation of a whole-page document, better meet the translation needs of users, and improve user experience.

[0004] In the first aspect, an embodiment of the present invention provides an AR-based translation method, which can be executed by an AR device or by an AR chip set in the device. The method includes: the AR device determines the target gesture operation input by the user, determines the first area to be translated according to the target gesture operation, and translates the text included in the first area to be translated, and outputs the translation result; wherein the first area to be translated is a partial area or the entire area of ​​the first area, and the first area is the entire area of ​​the first document.

[0005] Optionally, before the AR device determines the target gesture operation input by the user, the method further includes:

[0006] Periodically collecting a plurality of first images, wherein some or all of the plurality of first images include a target gesture operation performed by a user;

[0007] The multiple first images are matched with multiple preset gesture operations respectively, and at least one target image is selected from the multiple first images, wherein the target gesture operation included in the target image has a matching degree with the preset gesture operation greater than or equal to a first preset threshold.

[0008] Optionally, the AR device determines the first area to be translated according to the target gesture operation, including:

[0009] Extracting a first area in the target image where the target gesture operation is located;

[0010] Determine whether the working mode of the AR device is an association mode;

[0011] If the working mode of the AR device is the association mode, it is determined that the first area to be translated is larger than the first area; otherwise, it is determined that the first area to be translated is the first area.

[0012] Optionally, after the AR device determines the first area to be translated according to the target gesture operation, the method further includes:

[0013] Outputting and displaying the first region to be translated;

[0014] receiving a zoom operation of the user on the first area to be translated;

[0015] According to the scaling operation, a second region to be translated is determined, wherein the second region to be translated is obtained by scaling the first region to be translated.

[0016] Optionally, before determining whether the working mode of the AR device is the association mode, the method further includes: receiving a selection instruction input by a user, where the selection instruction is used to select to enter the association mode.

[0017] In a second aspect, an AR device is provided, wherein the AR device includes a camera, and the AR device further includes:

[0018] A first determination module, used to determine a target gesture operation input by a user;

[0019] A second determination module is used to determine a first area to be translated according to the target gesture operation, wherein the first area to be translated is a part of or the whole of the first area, and the first area is the whole area of ​​the first document;

[0020] The translation module is used to translate the text included in the first area to be translated and output the translation result.

[0021] Optionally, the camera is used to periodically capture a plurality of first images, wherein some or all of the plurality of first images include a target gesture operation performed by a user;

[0022] The first determination module is specifically used to match the multiple first images with multiple pre-set gesture operations respectively, and select at least one target image from the multiple first images, wherein the target gesture operation included in the target image has a matching degree with the pre-set gesture operation greater than or equal to a first preset threshold.

[0023] Optionally, the second determining module is specifically configured to:

[0024] Extracting a first area in the target image where the target gesture operation is located;

[0025] Determine whether the working mode of the AR device is an association mode;

[0026] If the working mode of the AR device is the association mode, it is determined that the first area to be translated is larger than the first area; otherwise, it is determined that the first area to be translated is the first area.

[0027] Optionally, the second determining module is further used for:

[0028] Outputting and displaying the first region to be translated;

[0029] receiving a zoom operation of the user on the first area to be translated;

[0030] According to the scaling operation, a second region to be translated is determined, wherein the second region to be translated is obtained by scaling the first region to be translated.

[0031] Optionally, the first determining module is further used to: receive a selection instruction input by a user, where the selection instruction is used to select to enter the association mode.

[0032] The technical effects of the AR device provided in the second aspect can refer to the technical effects of the various implementation methods of the first aspect mentioned above, and will not be repeated here.

[0033] In a third aspect, an AR device is provided, the AR device comprising:

[0034] at least one processor, and

[0035] a memory coupled to the at least one processor;

[0036] The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method as described in any one of the first aspects by executing the instructions stored in the memory.

[0037] The technical effects of the AR device provided by the embodiment of the present invention can refer to the technical effects of the various implementation methods of the first aspect mentioned above, and will not be repeated here.

[0038] According to a fourth aspect, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method according to any one of the first aspects is implemented.

[0039] The embodiment of the present invention can translate the content to be translated based on the positioning of the AR device, that is, perform local translation, which can meet the actual translation needs of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a translation method based on an AR device provided in an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of a gesture description provided by an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of a gesture description provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of a gesture description provided by an embodiment of the present invention;

[0046] Figure 6 A schematic diagram of the structure of an AR device provided in an embodiment of the present invention;

[0047] Figure 7 Another structural schematic diagram of an AR device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0049] With the emergence of more and more translation software, it has provided convenience for users to read documents. However, most translation software translates the entire document or the entire page of the document, and cannot achieve partial translation according to the user's needs. For example, when a user is reading a document, there may be only a small number of words or sentences that affect the user's reading. At this time, full-text translation is obviously a waste of time and unnecessary. Or after the full text is translated, the user needs to search for the part that the user needs to translate from the translated document, which requires further screening, which is cumbersome.

[0050] Furthermore, current translation software requires users to input the content to be translated into a specific location (also called a translation box) on the interface provided by the translation software for partial translation. However, if the content to be translated is on paper, users need to manually input the content to be translated into the translation box, or scan the paper document and convert it into an electronic file for the user through other electronic devices, which is also cumbersome.

[0051] In view of this, an embodiment of the present invention proposes a translation method based on AR technology, which can translate the content required for translation according to the user's location, thereby meeting the user's actual translation needs.

[0052] The technical solution provided by the embodiments of the present invention is described below in conjunction with the accompanying drawings.

[0053] See also Figure 1 , which is an application scenario applicable to the translation method provided in an embodiment of the present invention. The scenario includes an AR device and a carrier of a document to be translated. Among them, the AR device can scan the document to be translated, and can scan the entire area of ​​the document to be translated, or scan a partial area of ​​the document to be translated. It should be understood that the AR device can scan the corresponding area of ​​the document to be translated, and from this perspective, the area of ​​the document to be translated scanned by the AR device can be called the area to be translated, and the area to be translated can be the entire area of ​​the document to be translated, or it can be a partial area of ​​the document to be translated.

[0054] In some embodiments, the AR device may be an AR glasses, an AR helmet, or other devices. In the following description, the AR device is an AR glasses as an example. It should be noted that the AR device in the embodiments of the present invention is not limited to AR glasses, as long as the user can view the document to be translated through the AR device. It should be understood that the AR device may be provided with an image acquisition device, such as a camera, and may also be provided with a display screen to display the document to be translated.

[0055] See also Figure 2 , which is a flow chart of a translation method provided in an embodiment of the present invention. The method can be executed by AR glasses. The specific flow of the method is described as follows.

[0056] S201: An AR device receives a gesture operation input by a user, where the gesture operation can be used to indicate a translation area in a first document.

[0057] In an embodiment of the present invention, the translation area may be the entire area of ​​the first document or a partial area of ​​the first document. The AR glasses can scan the first document so that the user can view the entire first document through the AR glasses. If the user needs to translate part of the content in the first document, the user can perform gesture operations on the first document to specify the content to be translated (translation area) from the first document.

[0058] Specifically, the user can turn on the AR device to view the document to be translated. When viewing the first document, if the user wants to select a translation area from it, the user can perform a gesture operation on the first document through the AR device, and the gesture operation can be used to indicate the translation area. The AR device generates a recognition instruction based on the gesture operation performed by the user, and executes the recognition instruction to recognize the translation area corresponding to the gesture operation.

[0059] There are many ways that the user can use gestures to select the translation area. For example, see Figure 3 , users can use their thumb and other fingers to directly circle a certain area ( Figure 3 The area where the area is located can be considered as the translation area, and the user's operation of circling a certain area is the gesture operation of selecting the translation area. Alternatively, see Figure 4 , the user can also use two fingers to perform a relatively far sliding operation, then the sliding track formed by the two fingers ( Figure 3 If the area where the translation is located is within the translation area, the sliding operation performed by the user is a gesture operation to select the translation area. Alternatively, the user can also perform a sliding operation with a certain finger, and the sliding track formed by the sliding operation can circle a certain area ( Figure 5 The area is within the translation area (indicated by a bold cloud circle). The sliding operation performed by the user is a gesture operation for selecting the translation area.

[0060] For another example, the user may click on a certain position of the first document. If the time the user stays on the certain position is greater than or equal to the preset time, it can be considered that the user needs to translate the entire page of the first document. At this time, the translation area is the entire area of ​​the entire page of the first document, and the click operation performed by the user is the gesture operation of selecting the translation area. The preset time can be a preset fixed value, such as 5S, or can also be other possible values.

[0061] S202: The AR device determines a target gesture operation input by the user.

[0062] It should be understood that when the user views the first document through the AR device, the user may perform various gesture operations, that is, in addition to the translation gesture operation, the user may also perform other gesture operations, such as page turning gesture operations.

[0063] To this end, in an embodiment of the present invention, the AR device may periodically collect multiple first images, and some of the multiple first images may include gesture operations for translation, and some of the first images may include gesture operations other than translation. For ease of description, in the text, the gesture operation for translation is referred to as the target gesture operation. The AR device may select an image with the target gesture operation from the multiple first images.

[0064] Exemplarily, the embodiment of the present invention may pre-set multiple gesture operations, and these multiple gesture operations may be considered as gesture operations for translation, for example Figure 3-Figure 5 The bold cloud circle shown indicates the gesture operation. The AR device can extract the gesture operation included in each first image and match the gesture operation with the preset gesture operation. If the matching degree is greater than or equal to the preset threshold, then the gesture operation included in the first image can be considered to be the target gesture operation. On the contrary, if the matching degree is less than the preset threshold, then the gesture operation included in the first image is not the target gesture operation. In this way, the AR device can filter out the first image including the target gesture operation from multiple first images.

[0065] S203: Locate the area to be translated in the first document according to the target gesture operation.

[0066] It should be understood that the area to be translated is usually a regular area, such as a rectangular area, etc. However, the gesture operation performed by the user usually delimits an irregular area. To this end, in an embodiment of the present invention, the AR device can locate the area to be translated in the first document according to the recognized target gesture operation.

[0067] For example, see Figure 3-Figure 5 , the area circled by the target gesture operation performed by the user is the area indicated by the bold cloud circle, and the AR device can determine that the area to be translated is greater than or equal to the area indicated by the bold cloud circle, so as to avoid losing the information to be translated.

[0068] As an example, the horizontal increment and the vertical increment may be pre-equalized, wherein the horizontal increment is the increment along the first direction, the vertical increment is the increment along the second direction, and the first direction and the second direction are perpendicular. The AR device may extract the motion trajectory of the target gesture operation and delineate the first area according to the motion trajectory. On the basis of the first area, the horizontal increment is expanded along the first direction, and the vertical increment is expanded along the second direction, and the first area after expansion in the first direction and the second direction is determined as the area to be translated.

[0069] Furthermore, the AR device may output and display the area to be translated. Specifically, the AR device may output the area to be translated in the area where the target gesture operation is located, and the user may determine whether the area to be translated is appropriate. If the area to be translated is not appropriate, the user may perform a zoom operation on the area to be translated. The zoom operation may be, for example, an operation of sliding in a certain direction, etc. The AR device receives the zoom operation, zooms the area to be translated, and determines the final area to be translated. For ease of description, in this article, the area to be translated determined by the AR device through the target gesture operation is referred to as the first area to be translated, and the finally determined area to be translated is referred to as the second area to be translated.

[0070] Or, generally speaking, the first area to be translated is small, so some content that needs to be translated may be omitted. To this end, an embodiment of the present invention may set an association mode, which can be considered as the content located around the first area to be translated in addition to the content of the area to be translated. For example, multiple ranges can be set in advance, and each range can be determined according to the distance from the center of the first area to be translated. After the AR device outputs the first area to be translated determined according to the target gesture operation, the user can input a selection instruction to the AR device to enter the association mode. The AR device receives the selection instruction and can output multiple ranges, and the user selects a range from these multiple ranges. It can also be understood that when the AR device determines the area to be translated according to the target gesture operation, if it is detected that the working mode of the AR device is the association mode, then the AR device can determine the final second area to be translated according to the range selected by the association mode and the initially determined first area to be translated.

[0071] S204: Translate the content of the second region to be translated, and output the translation result.

[0072] After the AR device determines the second area to be translated, it can translate the content of the second area to be translated. Specifically, the AR device can intercept the image to be translated corresponding to the second area to be translated, and transmit the acquired image to the optical character recognition (OCR) module set in the AR device. It should be understood that the OCR module can process the image to be translated to obtain the text and other content included in the image to be translated. The OCR module sends the acquired text content to the translation module, and the translation module can translate the received text content.

[0073] Exemplarily, before translating the received text content, the translation module may output prompt information to prompt the user to select a target language for translation, such as English-Chinese translation, Chinese-English translation, English-French translation, English-German translation, etc. After the user selects the target language, the translation module translates the received text content according to the user's selection and outputs the translation result.

[0074] The AR device may display the translation result. Specifically, the AR device may display the translation result in a preset target area. It should be understood that the preset target area may avoid blocking the viewing area as much as possible. For example, the preset target area may be located in a corner area of ​​the display screen of the AR device, which may minimize blocking of the first document and improve the user experience.

[0075] In the embodiment of the present invention, the content to be translated can be translated based on the positioning of the AR device, that is, local translation can be performed, which can meet the actual translation needs of the user.

[0076] The device provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0077] See also Figure 6 Based on the same inventive concept, an embodiment of the present invention provides an AR device, the AR device includes a sensor such as a camera and an OCR module, and the VR device may include a first determination module 601, a second determination module 602 and a translation module 603. The first determination module 601, the second determination module 602 and the translation module 603 may implement related functional units through a hardware processor.

[0078] A first determining module 601 is used to determine a target gesture operation input by a user;

[0079] A second determination module 602 is used to determine a first area to be translated according to the target gesture operation, wherein the first area to be translated is a part or the whole area of ​​the first area, and the first area is the whole area of ​​the first document;

[0080] The translation module 603 is used to translate the text included in the first region to be translated and output the translation result.

[0081] Optionally, the camera is used to periodically capture a plurality of first images, wherein some or all of the plurality of first images include a target gesture operation performed by a user;

[0082] The first determination module 601 is specifically used to match the multiple first images with multiple pre-set gesture operations respectively, and select at least one target image from the multiple first images, wherein the target gesture operation included in the target image has a matching degree with the pre-set gesture operation greater than or equal to a first preset threshold.

[0083] Optionally, the second determining module 602 is specifically configured to:

[0084] Extracting a first area in the target image where the target gesture operation is located;

[0085] Determine whether the working mode of the AR device is an association mode;

[0086] If the working mode of the AR device is the association mode, it is determined that the first area to be translated is larger than the first area; otherwise, it is determined that the first area to be translated is the first area.

[0087] Optionally, the second determining module 602 is further configured to:

[0088] Outputting and displaying the first region to be translated;

[0089] receiving a zoom operation of the user on the first area to be translated;

[0090] According to the scaling operation, a second region to be translated is determined, wherein the second region to be translated is obtained by scaling the first region to be translated.

[0091] Optionally, the first determining module 601 is further used to: receive a selection instruction input by a user, where the selection instruction is used to select to enter the association mode.

[0092] The AR device can be used to perform Figure 2 The method provided in the embodiment shown in the figure, therefore, the functions that can be implemented by each functional unit in the AR device can be referred to. Figure 2 The relevant introduction in the illustrated embodiment will not be repeated in detail.

[0093] See also Figure 7 Based on the same inventive concept, an embodiment of the present invention provides an AR device, which may include: at least one processor 701, the processor 701 is used to execute the computer program stored in the memory to implement the embodiment of the present invention. Figure 2 The steps of the method are shown.

[0094] The processor 701 may be a central processing unit, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling program execution, and is used to control and manage the actions of the AR device, and support the AR device to perform operations such as Figure 2 The method steps are shown.

[0095] Optionally, the AR device further includes a memory 702 connected to at least one processor, and the memory 702 may include a read-only memory (ROM), a random access memory (RAM), and a disk memory. The memory 702 is used to store data required by the processor 701 when it is running, that is, it stores instructions that can be executed by at least one processor 701. At least one processor 701 executes the instructions stored in the memory 702 to execute the following. Figure 2 The method shown. The number of the memory 702 is one or more. The memory 702 is Figure 7 However, it should be noted that the memory 702 is not a mandatory functional module, so Figure 7 Shown by dotted lines.

[0096] The physical devices corresponding to the first determination module 601, the second determination module 602 and the translation module 603 may all be the aforementioned processor 701. The cooking appliance may be used to execute Figure 2 Therefore, for the functions that can be realized by each functional module in the device, please refer to Figure 2 The corresponding description in the illustrated embodiment will not be repeated here.

[0097] The embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes the following Figure 1 The method described.

[0098] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0099] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a read-only memory (ROM) / random access memory (RAM), a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A translation method based on AR, characterized in that: include: The AR device determines the target gesture operation input by the user; The AR device determines a first area to be translated according to the target gesture operation, wherein the first area to be translated is a part or the whole area of ​​the first area, and the first area is the whole area of ​​the first document; In order to avoid missing translation content, an association mode is set, and the association mode additionally translates content located around the first area to be translated; multiple ranges are pre-set, and each range is determined according to the distance from the center of the first area to be translated; The user inputs a selection instruction to the AR device to enter the association mode; the AR device receives the selection instruction and outputs multiple ranges, and the user selects a range from the multiple ranges; The AR device determines a final second area to be translated according to the range selected by the association mode and the determined first area to be translated; The AR device translates the content of the second area to be translated and outputs the translation result.

2. The method according to claim 1, characterized in that Before the AR device determines the target gesture operation input by the user, the method further includes: Periodically collecting a plurality of first images, wherein some or all of the plurality of first images include a target gesture operation performed by a user; The multiple first images are matched with multiple preset gesture operations respectively, and at least one target image is screened out from the multiple first images, wherein the target gesture operation included in the target image has a matching degree with the preset gesture operation greater than or equal to a first preset threshold.

3. An AR device, comprising a camera, characterized in that: The AR device further includes: A first determination module, used to determine a target gesture operation input by a user; A second determination module is used to determine a first area to be translated according to the target gesture operation, wherein the first area to be translated is a part of or the whole of the first area, and the first area is the whole area of ​​the first document; In order to avoid missing translation content, an association mode is set, and the association mode additionally translates content located around the first area to be translated; multiple ranges are pre-set, and each range is determined according to the distance from the center of the first area to be translated; The user inputs a selection instruction to the AR device to enter the association mode; the AR device receives the selection instruction and outputs multiple ranges, and the user selects a range from the multiple ranges; The AR device determines a final second area to be translated according to the range selected by the association mode and the determined first area to be translated; The translation module is used to translate the content of the second region to be translated and output the translation result.

4. The AR device according to claim 3, characterized in that: The camera is used to periodically capture a plurality of first images, wherein some or all of the plurality of first images include a target gesture operation performed by a user; The first determination module is specifically used to match the multiple first images with multiple pre-set gesture operations respectively, and filter out at least one target image from the multiple first images, wherein the target gesture operation included in the target image has a matching degree with the pre-set gesture operation greater than or equal to a first preset threshold.

Images