Text translation method, device, electronic device and storage medium

Abstract

The present invention provides a text translation method, device, electronic device and storage medium, the method comprising: obtaining a target text to be translated and a regional image of the target text; performing text translation on the target text to obtain a translation text expressed in the target language; performing semantic understanding on the regional image to obtain an image description text expressed in the target language; and determining a translation result of the target text based on the semantic information of the translation text and the semantic information of the image description text. The text translation method, device, electronic device and storage medium provided by the present invention can further improve translation accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a text translation method, device, electronic device and storage medium. Background Art

[0002] Image-to-text translation is the process of automatically translating the source language text contained in an image or video into the target language text using a computer system. Image-to-text translation technology can help people translate and understand the text content in images and videos quickly and effectively.

[0003] At present, images containing text to be translated are usually obtained by scanning devices, and then optical character recognition (OCR) is performed on the images to obtain the text to be translated, and the source language text is translated (such as Chinese-English translation) to obtain the translation result. However, the above translation results are usually separated from the semantic content of the image in the area where the text to be translated is located, resulting in low accuracy of the translation results. Summary of the invention

[0004] The present invention provides a text translation method, device, electronic device and storage medium, which are used to solve the defect of low text translation accuracy in the prior art.

[0005] The present invention provides a text translation method, comprising:

[0006] Acquire a target text to be translated and a region image of a region where the target text is located;

[0007] Performing text translation on the target text to obtain a translated text expressed in the target language;

[0008] Performing semantic understanding on the regional image to obtain image description text expressed in the target language;

[0009] Based on the semantic information of the translation text and the semantic information of the image description text, a translation result of the target text is determined.

[0010] According to a text translation method provided by the present invention, the determining of the translation result of the target text based on the semantic information of the translation text and the semantic information of the image description text includes:

[0011] Determining the semantic similarity between the translated text and the image description text based on the semantic information of the translated text and the semantic information of the image description text;

[0012] When the semantic similarity is greater than a threshold, the translation text is corrected based on the image description text to obtain a translation result of the target text.

[0013] According to a text translation method provided by the present invention, the translation text is corrected based on the image description text to obtain the translation result of the target text, including:

[0014] Matching the image description text with the translation text, and determining the to-be-corrected participle in the translation text and the correction participle in the image description text corresponding to the to-be-corrected participle;

[0015] The corrected segmented words are used to replace the segmented words to be corrected in the translation text to obtain a translation result of the target text.

[0016] According to a text translation method provided by the present invention, the determining of the semantic similarity between the translated text and the image description text further includes:

[0017] When the semantic similarity is less than or equal to a threshold, the translated text is used as the translation result.

[0018] According to a text translation method provided by the present invention, the step of obtaining a target text to be translated and a regional image of a region where the target text is located includes:

[0019] Acquire a global image including the target text and the regional image;

[0020] Performing text recognition on the global image to obtain the target text;

[0021] The image corresponding to the target text is deleted from the global image to obtain the regional image.

[0022] According to a text translation method provided by the present invention, the step of obtaining a target text to be translated and a regional image of a region where the target text is located includes:

[0023] Based on the first scanning element, collecting a text image of the target text;

[0024] Performing text recognition on the text image to obtain the target text;

[0025] Based on the second scanning element, a regional image of the area where the target text is located is collected.

[0026] According to a text translation method provided by the present invention, after determining the translation result of the target text, the method further includes: displaying the translation result.

[0027] The present invention also provides a text translation device, comprising:

[0028] A data acquisition unit, used to acquire a target text to be translated and a region image of a region where the target text is located;

[0029] An initial translation unit, used for performing text translation on the target text to obtain a translation text expressed in a target language;

[0030] An image description unit, used for performing semantic understanding on the regional image to obtain an image description text expressed in the target language;

[0031] The text translation unit is used to determine the translation result of the target text based on the semantic information of the translation text and the semantic information of the image description text.

[0032] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned text translation methods are implemented.

[0033] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned text translation methods are implemented.

[0034] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the above-mentioned text translation methods are implemented.

[0035] The text translation method, device, electronic device and storage medium provided by the present invention perform semantic understanding on the regional image of the area where the target text is located, so as to obtain an image description text that fits the semantic content of the regional image, and determine the translation result of the target text based on the semantic information of the translation text and the semantic information of the image description text, so that the translation result can fit the semantic content of the regional image, thereby improving the text translation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a flowchart of the text translation method provided by the present invention;

[0038] Figure 2 is a flowchart of an implementation of step 140 in the text translation method provided by the present invention;

[0039] Figure 3 is a flowchart of an implementation of step 142 in the text translation method provided by the present invention;

[0040] Figure 4 It is one of the flowcharts of the implementation of step 110 in the text translation method provided by the present invention;

[0041] Figure 5 This is the second flowchart diagram of the implementation of step 110 in the text translation method provided by the present invention;

[0042] Figure 6 is a structural schematic diagram of a text translation device provided by the present invention;

[0043] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] At present, images containing text to be translated are usually obtained by scanning devices, and then OCR recognition is performed on the images to obtain the text to be translated, and the source language text is translated (such as Chinese-English translation) to obtain the translation result. However, the text to be translated is usually the text in the image, and the above translation result is obtained by translating the image of the area where the text to be translated is located, so that the translation result is separated from the semantic content of the area image, which reduces the accuracy of the translation result.

[0046] For example, in the image, there is a person wearing blue trousers, a blue shirt, a blue hat, a green mask, and a black eye mask, and the image contains the text "I'm wearing blue trousers, a blue shirt, a blue hat and a black mask. Who am I?". In the traditional method, the text content in the image is usually scanned by a scanner, and then OCR recognition is performed to obtain the text, and then the text is translated. The translation result obtained is "I am wearing blue trousers, a blue shirt, a blue hat and a black mask. Who am I?". It can be seen that the "black mask" in the translation result is obviously inconsistent with the "black eye mask" in the image. This is because "mask" as a name interpretation has the meaning of both mask and eye mask. Therefore, when the text is translated alone, the translation result obtained is inconsistent with the content expressed in the image, which in turn affects the accuracy of the translation result.

[0047] To this end, the present invention provides a text translation method. Figure 1 It is a flowchart of the text translation method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0048] Step 110: Obtain the target text to be translated and the region image of the region where the target text is located.

[0049] Specifically, the target text is the text to be translated, and the regional image is the image of the region where the target text is located. The target text here can be directly input by the user, or can be obtained by transcribing the collected audio, or can be obtained by collecting images through image collection devices such as scanners, mobile phones, cameras, etc., and performing OCR text recognition on the images, which is not specifically limited in the embodiments of the present invention.

[0050] Among them, the target text may be a text used to describe the regional image, that is, the semantic content of the target text is highly correlated with the semantic content expressed by the regional image. For example, the semantic content expressed by the regional image is "a person wearing blue trousers, a blue shirt, a blue hat, a green mask, and a black eye mask", and the target text contained in the regional image is "I'm wearing blue trousers, a blue shirt, a blue hat and a black mask. Who am I?". It can be seen that the semantic content of the target text is basically consistent with the semantic content expressed by the regional image, that is, the semantic content of the two is highly correlated. In addition, the semantic content of the target text may also be less correlated with the semantic content expressed by the regional image. For example, the semantic content expressed by the regional image is "a mouse opens its mouth and shows a surprised expression", and the target text contained in the regional image is "Look at Maskman!". It can be seen that the semantic content of the target text is quite different from the semantic content expressed by the regional image, that is, the semantic content of the two is less correlated.

[0051] Step 120: Translate the target text to obtain a translated text in the target language.

[0052] Specifically, after obtaining the target text, the target text may be input into a text translation model, and the text translation model translates the target text to obtain a translation text expressed in the target language.

[0053] It should be noted that since the target text is translated without the semantic content of the regional image, and some word segments in the target text may have multiple translations, the semantic content of the translated text may deviate from the real semantic content of the target text. For example, the target text is "I'm wearing a black mask". When translating the target text alone, since mask has the meaning of both mask and eye mask when translated as a name, the translated text may be "I'm wearing a black mask". However, in the regional image corresponding to the target text, there is a person wearing a black eye mask, not a black mask. It can be seen that there is a difference between the semantic content expressed by the translated text "I'm wearing a black mask" and the regional image, that is, the translated text obtained after text translation of the target text needs to be corrected.

[0054] Optionally, before translating the target text, a text translation model can be pre-trained, which can be achieved by performing the following steps: first, a large number of sample texts are collected, and their corresponding sample translation texts are determined through manual annotation. Then, the initial model is trained based on the sample texts and their corresponding sample translation texts to obtain a text translation model.

[0055] Step 130: semantically understand the region image to obtain image description text expressed in the target language.

[0056] Specifically, the image description text is used to describe the semantic content expressed by the regional image. After the regional image is obtained, the regional image can be input into the semantic understanding model, and the semantic understanding model performs semantic understanding on the regional image to obtain the image description text expressed in the target language.

[0057] It should be noted that since the image description text is obtained by semantically understanding the regional image, the image description text can be understood as being obtained in combination with the semantic content of the regional image. That is, the image description text can express the semantic content of the regional image more accurately than the translated text obtained by translating without the semantic content of the regional image.

[0058] Optionally, before semantic understanding of the regional image, a semantic understanding model can be pre-trained, which can be implemented by performing the following steps: first, a large number of sample images are collected, and their corresponding sample image description texts are determined through manual annotation. Then, the initial model is trained based on the sample images and their corresponding sample image description texts to obtain a semantic understanding model.

[0059] Step 140: Determine the translation result of the target text based on the semantic information of the translation text and the semantic information of the image description text.

[0060] Specifically, since the semantic content of the regional image is separated when translating the target text, the translated text may not accurately express the actual semantic content of the target text. In this case, the translated text needs to be corrected.

[0061] In this regard, after determining the translation text and the image description text, the semantic similarity between the translation text and the image description text can be determined based on the semantic information of the translation text and the semantic information of the image description text. The greater the semantic similarity, the higher the correlation between the semantic content of the target text and the semantic content of the image description text, that is, the higher the probability that the target text is used to describe the regional image. At this time, the translation text can be corrected based on the image description text that can accurately describe the semantic content of the regional image. For example, the image description text with higher accuracy can be directly used as the translation result of the target text, or the translation text can be matched with the image description text to determine the to-be-corrected participle in the translation text and the correction participle corresponding to the to-be-corrected participle in the image description text, and after replacing the to-be-corrected participle in the translation text with the correction participle, the replaced translation text is used as the translation result.

[0062] In addition, the smaller the semantic similarity between the translated text and the image description text, the lower the correlation between the semantic content of the target text and the semantic content of the image description text, that is, the higher the probability that the target text and the regional image express different contents. At this time, the translation result of the target text is not constrained by the image description text, that is, the translated text can be directly used as the translation result.

[0063] Compared with the traditional method of translating the target text separately from the semantic content of the regional image, the method provided by the embodiment of the present invention combines the image description text used to describe the semantic content of the regional image to determine the translation result of the target text. Therefore, when the target text is used to describe the semantic content of the regional image, the translation result of the target text can be determined in accordance with the semantic content of the regional image itself, thereby improving the accuracy of text translation.

[0064] The text translation method provided by the embodiment of the present invention performs semantic understanding on the regional image of the area where the target text is located, so as to obtain an image description text that fits the semantic content of the regional image, and determines the translation result of the target text based on the semantic information of the translation text and the semantic information of the image description text, so that the translation result can fit the semantic content of the regional image, thereby improving the text translation accuracy.

[0065] Based on the above embodiments, Figure 2 is a flowchart of an implementation of step 140 in the text translation method provided by the present invention, such as Figure 2 As shown, step 140 includes:

[0066] Step 141: Determine the semantic similarity between the translated text and the image description text based on the semantic information of the translated text and the semantic information of the image description text;

[0067] Step 142: When the semantic similarity is greater than a threshold, the translation text is corrected based on the image description text to obtain a translation result of the target text.

[0068] Specifically, after determining the translation text and the image description text, the semantic similarity between the translation text and the image description text can be determined based on the semantic information of the translation text and the semantic information of the image description text. When the semantic similarity is greater than a threshold, it indicates that the semantic content of the target text and the semantic content of the image description text are highly correlated, that is, the target text is highly likely to be used to describe the regional image. At this time, the translation text can be corrected based on the image description text. For example, the image description text with higher accuracy can be directly used as the translation result of the target text, or the translation text can be matched with the image description text to determine the to-be-corrected participle in the translation text and the correction participle corresponding to the to-be-corrected participle in the image description text, and after replacing the to-be-corrected participle in the translation text with the correction participle, the replaced translation text is used as the translation result.

[0069] Optionally, when determining the semantic similarity between the translated text and the image description text, the semantic features of the translated text can be determined based on the semantic information of the translated text, and the semantic features of the image description text can be determined based on the semantic information of the image description text, and then the distance between the semantic features of the translated text and the semantic features of the image description text is calculated, and the semantic similarity is represented by the distance, that is, the greater the distance, the smaller the semantic similarity; the smaller the distance, the greater the semantic similarity.

[0070] It can be seen that the embodiment of the present invention can accurately determine the semantic similarity between the translated text and the image description text based on the semantic information of the two, and correct the translated text based on the image description text when the semantic similarity is greater than a threshold, thereby ensuring that the translation result of the target text can fit the semantic content of the regional image, thereby improving the translation accuracy.

[0071] Based on any of the above embodiments, Figure 3 is a flowchart of an implementation of step 142 in the text translation method provided by the present invention, such as Figure 3 As shown, step 142 includes:

[0072] Step 1421: Match the image description text with the translation text to determine the to-be-corrected segmentation in the translation text and the correction segmentation corresponding to the to-be-corrected segmentation in the image description text;

[0073] Step 1422: Replace the to-be-corrected segmentation in the translation text with the correction segmentation to obtain the translation result of the target text.

[0074] Specifically, the to-be-corrected segmentation refers to the segmentation with translation errors in the translation text, and the correction segmentation refers to the correct translation segmentation corresponding to the to-be-corrected segmentation. The image description text and the translation text are matched. If there is any segmentation in the translation text that is inconsistent with the expression content of the corresponding segmentation in the image description text, it indicates that the corresponding segmentation in the translation text may have a translation error. At this time, the corresponding segmentation in the translation text can be used as the to-be-corrected segmentation, and the corresponding segmentation in the image description text can be used as the correction segmentation, and the to-be-corrected segmentation in the translation text can be replaced with the correction segmentation to obtain the translation result of the target text.

[0075] For example, the translated text is "The doctor put on a yellow mask", and the image description text corresponding to the translated text is "The doctor put on a yellow mask". Then "mask" in the translated text is the participle to be corrected, and "mask" in the image description text is the correction participle. The correction participle "mask" is used to replace the participle to be corrected in the translated text, and the translation result is "The doctor put on a yellow mask".

[0076] Optionally, when correcting the translated text based on the image description text, the image description text and the translated text can be input into the correction model, and the correction model matches the image description text with the translated text, determines the to-be-corrected segmented words in the translated text and the correction segmented words corresponding to the to-be-corrected segmented words in the image description text, and replaces the to-be-corrected segmented words in the translated text with the correction segmented words, and outputs the translation result of the target text. The correction model is trained based on the sample translated text of the sample source text, the sample image description text of the sample area image of the area where the sample source text is located, and the sample translation result corresponding to the sample source text.

[0077] Optionally, when correcting the translated text based on the image description text, entity recognition can be performed on the image description text and the translated text respectively to obtain the entity segmentation of the image description text and the entity segmentation of the translated text, and the entity segmentation of the two are matched. If there are unmatched entity segmentations, it indicates that there may be translation errors in the corresponding entity segmentation in the translated text. At this time, the corresponding entity segmentation in the translated text is used as the segmentation to be corrected, and the corresponding entity segmentation in the image description text is used as the correction segmentation. Then, the segmentation to be corrected in the translated text is replaced by the correction segmentation, and the translated text after the segmentation is replaced is used as the translation result of the target text.

[0078] It can be seen that the embodiment of the present invention can accurately determine the to-be-corrected participles in the translated text and the correction participles corresponding to the to-be-corrected participles in the image description text by matching the image description text with the translated text, and replace the to-be-corrected participles in the translated text with the correction participles, thereby correcting the translated text, so that the translation result of the target text can fit the semantic content of the regional image.

[0079] Based on any of the above embodiments, the semantic similarity between the translated text and the image description text is determined in step 141, and then the following steps are further included:

[0080] When the semantic similarity is less than or equal to the threshold, the translated text is taken as the translation result.

[0081] Specifically, after determining the translation text and the image description text, the semantic similarity between the translation text and the image description text can be determined based on the semantic information of the translation text and the semantic information of the image description text. When the semantic similarity is less than or equal to a threshold, it indicates that the correlation between the semantic content of the target text and the semantic content of the image description text is low, that is, the probability that the content expressed by the target text and the regional image is different is high. At this time, the translation result of the target text is not constrained by the image description text, that is, the translation text can be directly used as the translation result.

[0082] It can be seen that the embodiment of the present invention uses the translated text as the translation result when the semantic similarity is less than or equal to the threshold, so that the translation result of the target text can be accurately determined when the content expressed by the target text and the regional image is different.

[0083] Based on any of the above embodiments, Figure 4 is one of the flowcharts of the implementation of step 110 in the text translation method provided by the present invention, such as Figure 4 As shown, step 110 includes:

[0084] Step 111a, obtaining a global image including the target text and the regional image;

[0085] Step 112a, performing text recognition on the global image to obtain target text;

[0086] Step 113a: Delete the image corresponding to the target text from the global image to obtain a regional image.

[0087] Specifically, the global image refers to an image containing target text information and regional image information, which can be obtained by a scanning element. After obtaining the global image, text recognition is performed on the global image, so that the target text in the global image can be obtained. Then, the image corresponding to the target text is deleted from the global image containing the target text information and regional image information, so that the regional image can be obtained.

[0088] It should be noted that the regional image here does not contain the image of the target text, so as to avoid interference of the target text information on the semantic content of the regional image when semantic understanding of the regional image is performed, and thus obtain an image description text that can accurately express the semantic content of the regional image.

[0089] It can be seen that the embodiment of the present invention first obtains a global image containing the target text and the regional image, and then performs text recognition on the global image, so that the target text can be accurately obtained. After obtaining the target text, the image corresponding to the target text is deleted from the global image, so that the regional image can be accurately obtained.

[0090] Based on any of the above embodiments, Figure 5 FIG. 2 is a flow chart of the implementation of step 110 in the text translation method provided by the present invention. Figure 5 As shown, step 110 includes:

[0091] Step 111b: acquiring a text image of the target text based on the first scanning element;

[0092] Step 112b, performing text recognition on the text image to obtain the target text;

[0093] Step 113b: Based on the second scanning element, a regional image of the area where the target text is located is captured.

[0094] Specifically, the first scanning element is used to capture the text image of the target text, so that text recognition can be performed on the text image to accurately obtain the corresponding target text. The second scanning element is used to capture the regional image of the area where the target text is located, so that semantic understanding can be performed based on the regional image to accurately obtain the image description text.

[0095] It can be understood that the first scanning element and the second scanning element can be two scanning elements located on the body of the dictionary pen, and then when the user translates the target text, the corresponding image can be captured by the first scanning element and the second scanning element on the dictionary pen, and then the captured image can be transmitted to the processor in the dictionary pen, which is used to execute the text translation method described in any of the above embodiments.

[0096] Based on any of the above embodiments, the translation result of the target text is determined in step 140, and then the translation result is displayed.

[0097] Specifically, after obtaining the translation result of the target text, the translation result can be displayed in the form of text on the display screen of the dictionary pen, or in the form of voice playback through the voice playback module on the dictionary pen, or in the form of text display and semantic playback at the same time, so that the user can intuitively and quickly obtain the translation result of the target text.

[0098] Based on any of the above embodiments, the present invention further provides a text translation method, the method comprising:

[0099] The target text to be translated and the regional image of the target text area are obtained, and then the target text is translated to obtain the translated text expressed in the target language, and the regional image is input into the semantic understanding model, and the semantic understanding model performs semantic understanding on the regional image to obtain the image description text expressed in the target language.

[0100] Next, based on the semantic information of the translated text and the semantic information of the image description text, the semantic similarity between the translated text and the image description text is determined. When the semantic similarity is greater than a threshold, the image description text and the translated text are matched to determine the to-be-corrected segmentation in the translated text and the correction segmentation corresponding to the to-be-corrected segmentation in the image description text, and the to-be-corrected segmentation in the translated text is replaced with the correction segmentation to obtain the translation result of the target text. When the semantic similarity is less than or equal to the threshold, the translated text is used as the translation result.

[0101] Finally, after obtaining the translation result of the target text, the translation result of the target text is displayed.

[0102] When obtaining the target text to be translated and the regional image of the area where the target text is located, the same scanning element can be used to capture a global image containing the target text and the regional image, and then the global image is subjected to text recognition to obtain the target text, and then the image corresponding to the target text is deleted from the global image to obtain the regional image. A first scanning element can also be used to capture a text image of the target text, and text recognition is performed on the text image to obtain the target text, and a second scanning element can be used to capture a regional image of the area where the target text is located.

[0103] The text translation device provided by the present invention is described below. The text translation device described below and the text translation method described above can be referred to each other.

[0104] Based on any of the above embodiments, Figure 6 is a schematic diagram of the structure of the text translation device provided by the present invention, such as Figure 6 As shown, the text translation device comprises:

[0105] The data acquisition unit 610 is used to acquire a target text to be translated and a region image of a region where the target text is located;

[0106] An initial translation unit 620 is used to perform text translation on the target text to obtain a translation text expressed in a target language;

[0107] An image description unit 630, configured to perform semantic understanding on the regional image to obtain an image description text expressed in the target language;

[0108] The text translation unit 640 is used to determine the translation result of the target text based on the semantic information of the translation text and the semantic information of the image description text.

[0109] Based on any of the above embodiments, the text translation unit 640 includes:

[0110] a similarity determination unit, configured to determine the semantic similarity between the translation text and the image description text based on the semantic information of the translation text and the semantic information of the image description text;

[0111] The first translation unit is used to correct the translation text based on the image description text when the semantic similarity is greater than a threshold, so as to obtain a translation result of the target text.

[0112] Based on any of the above embodiments, the first translation unit includes:

[0113] A matching unit, configured to match the image description text with the translation text, and determine the to-be-corrected segmentation in the translation text and the correction segmentation corresponding to the to-be-corrected segmentation in the image description text;

[0114] The correction unit is used to replace the to-be-corrected participle in the translation text with the correction participle to obtain the translation result of the target text.

[0115] Based on any of the above embodiments, the device further includes:

[0116] The second translation unit is configured to determine the semantic similarity between the translation text and the image description text, and when the semantic similarity is less than or equal to a threshold, use the translation text as the translation result.

[0117] Based on any of the above embodiments, the data acquisition unit 610 includes:

[0118] A global image acquisition unit, used for acquiring a global image including the target text and the regional image;

[0119] A first recognition unit, configured to perform text recognition on the global image to obtain the target text;

[0120] The image deletion unit is used to delete the image corresponding to the target text from the global image to obtain the regional image.

[0121] Based on any of the above embodiments, the data acquisition unit 610 includes:

[0122] A first scanning unit, configured to capture a text image of the target text based on a first scanning element;

[0123] A second recognition unit, used for performing text recognition on the text image to obtain the target text;

[0124] The second scanning unit is used to collect a regional image of the area where the target text is located based on a second scanning element.

[0125] Based on any of the above embodiments, the device further includes:

[0126] A display unit is used to display the translation result after determining the translation result of the target text.

[0127] Figure 7 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 7 As shown, the electronic device may include: a processor 710, a memory 720, a communications interface 730 and a communications bus 740, wherein the processor 710, the memory 720 and the communications interface 730 communicate with each other through the communications bus 740. The processor 710 may call the logic instructions in the memory 720 to execute the text translation method, which includes: obtaining a target text to be translated and a regional image of the region where the target text is located; performing text translation on the target text to obtain a translation text expressed in the target language; performing semantic understanding on the regional image to obtain an image description text expressed in the target language; and determining the translation result of the target text based on the semantic information of the translation text and the semantic information of the image description text.

[0128] In addition, the logic instructions in the above-mentioned memory 720 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0129] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the text translation method provided by the above-mentioned methods, and the method includes: obtaining a target text to be translated, and a regional image of the area where the target text is located; performing text translation on the target text to obtain a translated text expressed in the target language; performing semantic understanding on the regional image to obtain an image description text expressed in the target language; and determining the translation result of the target text based on the semantic information of the translated text and the semantic information of the image description text.

[0130] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned text translation methods, the methods comprising: obtaining a target text to be translated, and a regional image of the area where the target text is located; performing text translation on the target text to obtain a translated text expressed in a target language; performing semantic understanding on the regional image to obtain an image description text expressed in the target language; and determining a translation result of the target text based on the semantic information of the translated text and the semantic information of the image description text.

[0131] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text translation method, characterized in that: include: Acquire a target text to be translated and a regional image of a region where the target text is located, wherein the regional image and the image of the target text constitute a global image; Performing text translation on the target text to obtain a translated text expressed in the target language; Performing semantic understanding on the regional image to obtain image description text expressed in the target language; Determining a translation result of the target text based on the semantic information of the translation text and the semantic information of the image description text; The determining the translation result of the target text based on the semantic information of the translation text and the semantic information of the image description text includes: Determining the semantic similarity between the translated text and the image description text based on the semantic information of the translated text and the semantic information of the image description text; When the semantic similarity is greater than a threshold, the translation text is corrected based on the image description text to obtain a translation result of the target text.

2. The text translation method according to claim 1, characterized in that: The correcting the translated text based on the image description text to obtain the translation result of the target text includes: Matching the image description text with the translation text, and determining the to-be-corrected participle in the translation text and the correction participle in the image description text corresponding to the to-be-corrected participle; The corrected segmented words are used to replace the segmented words to be corrected in the translation text to obtain a translation result of the target text.

3. The text translation method according to claim 1, characterized in that: The determining of the semantic similarity between the translated text and the image description text further includes: When the semantic similarity is less than or equal to a threshold, the translated text is used as the translation result.

4. The text translation method according to any one of claims 1 to 3, characterized in that: The step of obtaining a target text to be translated and a region image of a region where the target text is located includes: Acquire a global image including the target text and the regional image; Performing text recognition on the global image to obtain the target text; The image corresponding to the target text is deleted from the global image to obtain the regional image.

5. The text translation method according to any one of claims 1 to 3, characterized in that: The step of obtaining a target text to be translated and a region image of a region where the target text is located includes: Based on the first scanning element, collecting a text image of the target text; Performing text recognition on the text image to obtain the target text; Based on the second scanning element, a regional image of the area where the target text is located is collected.

6. The text translation method according to any one of claims 1 to 3, characterized in that: The step of determining the translation result of the target text further includes: displaying the translation result.

7. A text translation device, characterized in that: include: A data acquisition unit, used to acquire a target text to be translated and a regional image of a region where the target text is located, wherein the regional image and the image of the target text constitute a global image; An initial translation unit, used for performing text translation on the target text to obtain a translation text expressed in a target language; An image description unit, used for performing semantic understanding on the regional image to obtain an image description text expressed in the target language; A text translation unit, configured to determine a translation result of the target text based on the semantic information of the translation text and the semantic information of the image description text; The determining the translation result of the target text based on the semantic information of the translation text and the semantic information of the image description text includes: Determining the semantic similarity between the translated text and the image description text based on the semantic information of the translated text and the semantic information of the image description text; When the semantic similarity is greater than a threshold, the translation text is corrected based on the image description text to obtain a translation result of the target text.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the text translation method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the text translation method according to any one of claims 1 to 6 are implemented.

Images