Translation method, device, system, equipment, storage medium and program product

Abstract

The invention provides a translation method, device, system and equipment, a storage medium and a program product, and the method comprises the steps: carrying out semantic analysis based on an input text and an input image in user input, and determining an output reply based on an analysis result; under the condition that the output reply is the input guidance for the user, acquiring supplementary input of the user corresponding to the input guidance; translation is carried out based on user input and supplementary input to obtain a translation result corresponding to the user input, translation is carried out in an interactive mode, when it is determined that an accurate result cannot be obtained currently through analysis, guidance can be provided to enable a user to carry out targeted information supplementation to obtain more information related to a translation target, translation is carried out based on the information, and the translation efficiency is improved. The problem that a translation result is inaccurate, unreliable and unpractical in a traditional scheme is solved, translation flexibility is improved, complex scenes can be better dealt with, ambiguity is reduced, an accurate and reliable translation result is obtained, effective help can be provided for a user, and user experience is optimized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a translation method, device, system, equipment, storage medium and program product. Background Art

[0002] In the current translation technology field, although there are many methods that have realized the function of extracting text from images and translating them, most of these methods are limited to directly converting the text in the image into the corresponding translation text, or filling the translation text in the corresponding position of the original text. However, these methods face many challenges in practical applications, especially in dealing with complex objects, and there are major problems in comprehensiveness, accuracy and practicality.

[0003] Specifically, current translation technology often directly translates based on the limited information provided by the user. Once faced with complex and ambiguous situations, it is impossible to obtain accurate and reliable translation results, and thus it is impossible to provide users with practical help for specific translation scenarios. The accuracy and practicality of the translation are greatly reduced, and the user experience is also worrying. Summary of the invention

[0004] The present invention provides a translation method, apparatus, system, device, storage medium and program product to solve the defects in the prior art that the translation results are inaccurate, unreliable and impractical, resulting in a poor translation experience, achieve comprehensive and accurate content translation, and provide users with reliable language assistance to optimize the user's translation experience.

[0005] The present invention provides a translation method, comprising: Determining user input, wherein the user input includes input text and input image; Performing semantic analysis based on the input text and the input image, and determining an output response based on an analysis result obtained by the semantic analysis; In a case where the output reply is an input guidance to the user, obtaining a supplementary input of the user corresponding to the input guidance; Based on the user input and the supplementary input, a translation is performed to obtain a translation result corresponding to the user input.

[0006] According to a translation method provided by the present invention, the semantic analysis based on the input text and the input image comprises: Performing text encoding and visual encoding based on the input text and the input image, respectively, to obtain text features and image features; A semantic analysis is performed based on the text features and the image features, wherein the semantic analysis includes at least one of polysemous word analysis, cultural field analysis, and language difference analysis.

[0007] According to a translation method provided by the present invention, the step of determining an output response based on an analysis result obtained by semantic analysis comprises: When the analysis result indicates that the translation target in the input image has at least one of a polysemy problem, a cross-domain problem, and a language difference problem, determining that the output reply is an input guidance for the user; When the analysis result indicates that the translation target in the input image does not have polysemy problems, cross-domain problems, and language difference problems, the output reply is determined to be the translation result corresponding to the user input.

[0008] According to a translation method provided by the present invention, the step of performing text encoding and visual encoding based on the input text and the input image to obtain text features and image features includes: Performing object detection based on the input text and the input image; When the detection result obtained by the target detection indicates that the translation target exists in the input image, text encoding and visual encoding are performed based on the input text and the input image respectively to obtain text features and image features.

[0009] According to a translation method provided by the present invention, the translation is performed based on the user input and the supplementary input to obtain a translation result corresponding to the user input, including: Use the supplementary input corresponding to the output reply of the previous round as the user input of the current round; Based on the user input of the current round, and the user input and output reply of the previous round of the current round, translation is performed to obtain the translation result.

[0010] According to a translation method provided by the present invention, the output reply is determined based on the analysis result obtained by semantic analysis, and then further includes: Performing speech synthesis based on the output reply to obtain a reply synthesized speech, and displaying the reply synthesized speech; The step of obtaining the translation result corresponding to the user input further includes: Speech synthesis is performed based on the translation result to obtain result synthesized speech, and the result synthesized speech is displayed.

[0011] The present invention also provides a translation device, comprising: A determination unit, used to determine user input, wherein the user input includes input text and input image; An analysis unit, configured to perform semantic analysis based on the input text and the input image, and determine an output response based on an analysis result obtained by the semantic analysis; An acquiring unit, configured to acquire, when the output reply is an input guidance to the user, a supplementary input of the user corresponding to the input guidance; The translation unit is used to translate based on the user input and the supplementary input to obtain a translation result corresponding to the user input.

[0012] The present invention also provides a translation system, wherein a large multimodal model is deployed on the system; the large multimodal model is constructed on the basis of a large language model and a large visual model; The multimodal large model is used to perform semantic analysis based on input text and input images in user input, determine an output response based on the analysis result obtained by the semantic analysis, and when the output response is an input guidance for the user, obtain the user's supplementary input corresponding to the input guidance, and perform translation based on the user input and the supplementary input to obtain a translation result corresponding to the user input.

[0013] According to a translation system provided by the present invention, a speech synthesis engine and a speech recognition engine are deployed on the system, and a sound pickup device, an image acquisition device and a display device are configured; the sound pickup device and the image acquisition device are used to collect the user's input speech and the input image respectively; The speech recognition engine is used to obtain the input speech collected by the sound pickup device, and perform speech recognition on the input speech to obtain the input text; The speech synthesis engine is used to perform speech synthesis based on the output reply and the translation result, respectively, to obtain a reply synthesized speech and a result synthesized speech; The display device is used to display the output reply and the translation result, and / or the reply synthesized speech and the result synthesized speech.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described translation methods when executing the computer program.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the translation method described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned translation methods.

[0017] The translation method, apparatus, system, device, storage medium and program product provided by the present invention perform semantic analysis according to user input, and determine an output response according to the analysis result, and obtain the user's supplementary input corresponding to the input guidance when the output response is a guidance for the user's input; perform translation based on the user input and the supplementary input to obtain a translation result, and perform translation in an interactive manner. When the analysis determines that an accurate result cannot be obtained at present, guidance can be provided to enable the user to supplement information in a targeted manner to obtain more information about the translation target. Translation is performed based on this, which overcomes the problems of inaccurate, unreliable and impractical translation results in traditional solutions, greatly improves the accuracy and flexibility of translation, can better cope with complex scenarios, reduce ambiguity, obtain accurate and reliable translation results, and can provide effective help to users to optimize user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a schematic diagram of the process of the translation method provided by the present invention; Figure 2 is an example diagram of the translation process provided by the present invention; Figure 3 It is the overall framework diagram of the translation process provided by the present invention; Figure 4 is a schematic diagram of the structure of the translation device provided by the present invention; Figure 5 It is a schematic diagram of the structure of the translation system provided by the present invention; Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Most current translation technologies directly extract text from images and translate them into text, or fill the translated text in the corresponding position of the original text and display it with VR (Virtual Reality) equipment. However, these methods face many challenges in practical applications, especially in terms of the comprehensiveness, accuracy and practicality of translating complex objects.

[0022] A notable problem is that current translation technology cannot effectively deal with complex translation objects, such as short abbreviations, polysemous words, etc. When translating such objects, it is often only possible to translate based on the limited information currently provided by the user, and it is impossible to obtain more information about the translation object. Therefore, it is difficult to obtain accurate and reliable translation results, and it is impossible to provide effective help to users in actual scenarios. The practicality is poor and the user experience is not good.

[0023] In this regard, the present invention provides a translation method, which aims to perform semantic analysis based on user input to determine whether there is a situation where the semantics are complex and a reliable translation result cannot be directly obtained, and a reply can be given according to the analysis result, so that the user can make targeted supplements based on the output reply, thereby obtaining richer and more comprehensive information, and then translation can be performed based on this to obtain a translation result corresponding to the user input. Translation is performed in an interactive manner, which greatly improves the accuracy and flexibility of the translation, can better cope with complex scenarios, reduce ambiguity, and obtain accurate and reliable translation results.

[0024] Figure 1 It is a schematic diagram of the translation method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, determining user input, where the user input includes input text and input image; Step 120, performing semantic analysis based on the input text and the input image, and determining an output response based on an analysis result obtained by the semantic analysis; Step 130, when the output reply is an input guidance for the user, obtaining a supplementary input of the user corresponding to the input guidance; Step 140: perform translation based on the user input and the supplementary input to obtain a translation result corresponding to the user input.

[0025] Here, user input refers to the input information provided by the user during translation, which can be text, voice, image, etc., and the embodiments of the present invention do not specifically limit it. User input is for the translation system. Before translation, the user can input his own needs and the object / target to be translated into the system so that the system can translate it and finally output the translation result corresponding to the user input. Among them, user needs can be presented in text (input text), and the translation object can be presented in the form of an image (input image) as a carrier.

[0026] It should be noted that in the actual translation process, users mostly expect to get immediate feedback when they have translation needs. Therefore, when translating, most of the input is done quickly in the form of voice rather than word-for-word text entry. When the input information includes input voice, in order to facilitate the understanding and semantic analysis of user needs during subsequent translation, the input voice can be converted into text form to obtain the input text.

[0027] The input text may be a sentence, such as "Please help me translate the letters in the picture?", "Can you explain what the English letters in the picture mean?", etc., or a paragraph consisting of multiple sentences, such as "I am now under a building. There is a large blue sign in the square downstairs, which is a few letters. I am a little curious about what these letters mean. Can you help me translate them?", which is not specifically limited in the embodiments of the present invention. The input text may be directly input by the user, or may be obtained by performing voice recognition on the input voice, and the voice recognition may be implemented by a voice recognition engine deployed on the system.

[0028] The input image can be an image directly collected by the user, such as when the user has a translation requirement, the image can be collected by using an image acquisition device set on the system, or it can be an image obtained from a video stream, such as when the user's input is a real-time video stream, the video frame in the real-time video stream can be used as the input image. The input image can be a single image or multiple images, which is not specifically limited in the embodiment of the present invention.

[0029] After determining the input text and input image, considering that the traditional solution directly translates based on the information provided by the user, the problem of being unable to guarantee the accuracy and reliability of the translation results in the face of complex translation objects, in the embodiment of the present invention, the user input is first analyzed to determine whether the currently input translation object has complex semantics such as abbreviations, multiple meanings, cross-domains, language differences, etc., which makes it impossible to directly obtain accurate and reliable translation results, thereby obtaining analysis results. That is, the translation object can be semantically analyzed based on the input text and input image to determine whether the translation object has the above problems in the current translation scenario, thereby obtaining analysis results. Here, specifically, the input text and input image can be directly input into the multimodal large model of the system, so as to perform analysis with the help of this and obtain analysis results.

[0030] It should be noted that the multimodal large model is constructed on the basis of a large language model and a large visual model. Before it is actually translated and put into use, it has been learned through a large amount of text and image data. It has excellent capabilities in language understanding, information search, comparative query, text translation, etc. Therefore, before translation, the powerful capabilities of this model can be used for semantic analysis to obtain reliable analysis results.

[0031] Furthermore, after obtaining the analysis results, the embodiments of the present invention can output targeted reply content according to the analysis results to obtain accurate and reliable translation results corresponding to the user input. That is, the reply corresponding to the current input, that is, the output reply corresponding to the user input, can be determined according to the analysis results obtained by semantic analysis.

[0032] Specifically, when it is determined through semantic analysis that the currently input translation object has complex semantics and cannot directly produce a reliable translation result, a guided reply content can be provided and output so that the user can supplement the information based on the output reply after knowing the output reply, thereby obtaining the user's supplementary input.

[0033] Correspondingly, when the analysis determines that the currently input translation object does not have the problem of complex semantics and cannot directly produce reliable translation results, the translation can be performed directly. Specifically, the translation can be performed using a multimodal large model to translate the translation object / translation target in the input image under the guidance of the input text, and the translation result is output. Since there is no problem of complex semantics, the translation results obtained through the multimodal large model are accurate and reliable, which can provide practical and effective help to users in actual scenarios and help optimize the user experience.

[0034] After obtaining the output response of the multimodal large model, if the output response is a guided response content, for example, "This may be the abbreviation of the name of a corporate organization. In order to better understand the surrounding environment, please slightly turn your phone to take a distant image, or supplement the surrounding information." That is, the output response is an input guide for the user. At this time, the user's reply information based on the input guide can be obtained, which can also be understood as obtaining the input information when the user supplements the input guide. This input information is called the user's supplementary input. The supplementary input here can be new text provided by the user according to the input guide, or a new image, or new text and a new image, which is not specifically limited in the embodiment of the present invention.

[0035] After that, the translation can be performed according to the user's supplementary input and the original user input to obtain the translation result. That is, after obtaining the input information supplemented by the user according to the output reply, in the embodiment of the present invention, the translation can be performed according to the supplemented input information at this time and the previous input information. Specifically, semantic analysis can be performed based on the supplementary input and the user input to determine whether the semantics cannot be determined by the supplemented information after the user supplements the information, and whether there are still semantic complexity problems. If there are still, the above process can be repeated, and the input guidance for the user can be output again to further obtain information about the translation object and translate it; each input guidance can be determined according to the current input information. On the contrary, if there is no semantic complexity problem, the translation object can be translated using a multimodal large model based on the user input and the supplementary input, and finally the translation result corresponding to the user input can be obtained.

[0036] In an embodiment of the present invention, after each user input, semantic analysis is performed based on the latest input information and the previously input information, and when it is determined that there is no semantic complexity problem, translation is performed through a multimodal large model to obtain a translation result; when this problem exists, a guided reply is given, and the user's supplementary input is obtained, and the above process is repeated until there is no semantic complexity and a reliable translation result cannot be directly obtained; through multiple rounds of analysis and supplementary input, the final translation result can be made more accurate and reliable, can better adapt to specific scenarios, and provide users with more practical and reliable language assistance.

[0037] The translation method provided by the present invention performs semantic analysis according to user input, and determines an output response according to the analysis result, and obtains the user's supplementary input corresponding to the input guidance when the output response is a guidance for the user's input; performs translation based on the user input and the supplementary input to obtain a translation result, and performs translation in an interactive manner. When the analysis determines that an accurate result cannot be obtained at present, guidance can be provided to enable the user to supplement information in a targeted manner to obtain more information about the translation target, and the translation is performed based on this, which overcomes the problems of inaccurate, unreliable and impractical translation results in traditional solutions, greatly improves the accuracy and flexibility of translation, can better cope with complex scenarios, reduce ambiguity, obtain accurate and reliable translation results, and can provide effective help to users, thereby optimizing user experience.

[0038] Based on the above embodiment, in step 120, semantic analysis is performed based on the input text and the input image, including: Perform text encoding and visual encoding based on the input text and input image respectively to obtain text features and image features; Semantic analysis is performed based on text features and image features, where the semantic analysis includes at least one of polysemy analysis, cultural field analysis, and language difference analysis.

[0039] Specifically, in step 120, the process of performing semantic analysis based on the input text and the input image may specifically include: Considering that current translation methods cannot effectively deal with complex scenarios, and face challenges in terms of polysemy, cross-cultural and language differences. Specifically, current translation technology cannot deal with the common phenomena of synonyms and polysemy in the translation process, as well as the resulting ambiguity. Different contexts, cultural backgrounds and language habits may lead to multiple possible interpretations of the same word when translating; in addition, cross-cultural differences also make it possible for certain words or expressions to have specific meanings in one language, while it is difficult to find a completely corresponding expression in another language. However, current translation methods are completely unable to deal with such problems and cannot obtain translation results that fit the context / environment and can effectively help users.

[0040] In view of this, in the embodiment of the present invention, it is proposed that after determining the user input, during the specific semantic analysis, specific analysis can be performed on the above problems, such as analyzing whether the currently input translation object has polysemy problems, cross-cultural problems, language differences, etc., so as to obtain analysis results. That is, based on the input text and input image in the user input, one or more of polysemy analysis, cultural field analysis, and language difference analysis can be performed using a multimodal large model to obtain the analysis results of the semantic analysis.

[0041] In detail, after the input text and the input image are input into the multimodal large model, the multimodal large model will first perform feature extraction on the input text and the input image to extract the features that represent the respective contents in the text and the image; that is, the text encoder and the visual encoder in the multimodal large model are used to encode the input text and the input image respectively, so as to obtain text features through encoding by the text encoder and obtain image features through encoding by the visual encoder.

[0042] After that, semantic analysis can be performed based on the text features and image features obtained by encoding to obtain the analysis results of the translation object. Here, specifically, based on the text features and image features respectively encoded by the text encoder and the visual encoder, a semantic analysis can be performed on the currently input translation object through a multimodal large model, that is, based on the text features and image features, at least one of polysemy analysis, cultural field analysis, and language difference analysis is performed on the translation object to analyze whether it has polysemy problems, cross-cultural field problems, language difference problems, etc., so as to obtain the analysis results of the translation object at the above levels; and the analysis results at various levels can be combined to determine the analysis results of the final translation object.

[0043] In an embodiment of the present invention, by performing at least one of polysemy analysis, cultural field analysis and language difference analysis on text features of an input text and image features of an input image, complex scenarios in the translation process can be effectively identified, so that targeted responses can be given based on the analysis results to guide the user to provide more relevant information, thereby effectively improving the accuracy and reliability of the translation results, more effectively coping with complex scenarios, and solving the current problem of being unable to cope with polysemy, cross-cultural fields and obvious language differences in the translation process.

[0044] Based on the above embodiment, in step 120, determining the output reply based on the analysis result obtained by the semantic analysis includes: When the analysis result indicates that the translation target in the input image has at least one of a polysemy problem, a cross-domain problem, and a language difference problem, determining that the output response is an input guidance for the user; When the analysis result indicates that the translation target in the input image does not have polysemy problems, cross-domain problems, and language difference problems, the output reply is determined to be the translation result corresponding to the user input.

[0045] Specifically, the process of determining the output response based on the analysis results obtained by semantic analysis can be divided into the following two cases: First, when the analysis determines that the translation object has complex semantics and cannot directly produce reliable translation results, that is, when the analysis results show that the translation object / translation target in the input image has any one or more of the following problems: polysemy, cross-domain problems, and language differences, at this time, the multimodal large model can output guided reply content to guide the user to add more information about the translation target. That is, in this case, the output reply can be determined based on the user input, and the output reply is a guide to the user's input.

[0046] Secondly, when the analysis determines that the translation object does not have a problem of semantic complexity and cannot directly obtain a reliable translation result, that is, when the analysis results show that the translation object / translation target in the input image does not have polysemy problems, cross-domain problems, and language differences, at this time, the translation object can be directly translated to obtain a reliable translation result. Here, specifically, under the guidance of the input text, the translation object / translation target in the translation input image is translated using a multimodal large model, and the translation result is output. Based on this translation result, the current reply content can be determined, that is, when there is no semantic complexity problem, the output reply is the translation result corresponding to the user input.

[0047] It should be noted that after the multimodal large model obtains the analysis results through semantic analysis, it will not directly output the analysis results, but will give a response directly based on the analysis results. In non-complex scenarios, it will directly provide translation results that meet user needs. In complex scenarios, it will provide guided response content to guide users to supplement information, thereby improving translation accuracy.

[0048] Based on the above embodiment, text encoding and visual encoding are performed based on the input text and the input image respectively to obtain text features and image features, including: Perform object detection based on input text and input image; When the detection result obtained by the target detection indicates that the translation target exists in the input image, text encoding and visual encoding are performed based on the input text and the input image respectively to obtain text features and image features.

[0049] Specifically, the process of performing text encoding and visual encoding on the input text and the input image respectively to obtain text features and image features may specifically include: It is understandable that in the actual translation process, in order to facilitate input and achieve real-time translation in specific scenarios, the user may not input a single image, but a video stream. For example, when a user is on a business trip and is in a foreign place where the language is not fluent, he can input a real-time video stream to achieve real-time translation of the content in the video stream through interaction with the multimodal large model.

[0050] However, not every frame in the input video stream may contain content that needs to be translated. Therefore, in order to improve the translation efficiency of the multimodal large model and obtain translation results more quickly, in an embodiment of the present invention, target detection can also be performed before semantic analysis to detect whether there are translation objects / translation targets in each video frame of the video stream, thereby obtaining detection results.

[0051] Specifically, here, based on the input text, each video frame / input image in the video stream is first subjected to target detection through a multimodal large model to detect whether there is an object / target that needs to be translated mentioned in the input text, thereby obtaining the detection result of each video frame / input image. Here, the translation object / translation target can be a specific text or icon, such as "English", "IATA", etc., or it can be a broadly referred content, such as all English, which is not specifically limited in the embodiment of the present invention.

[0052] Furthermore, when the detection result indicates that the translation object / target exists in the input image, semantic analysis can be performed, that is, text encoding and visual encoding can be performed according to the input text and the input image, respectively, to obtain text features and image features. Conversely, when the detection result reflects that the object / target to be translated indicated by the input text does not exist in the input image, the video frame can be removed from the video stream, or when the input is a single image, a prompt message can be output to prompt the user to adjust the shooting angle and re-capture the image.

[0053] Based on the above embodiment, step 140 includes: Use the supplementary input corresponding to the output reply of the previous round as the user input of the current round; Based on the user input of the current round, as well as the user input and output responses of the previous round of the current round, the translation is performed to obtain a translation result.

[0054] Specifically, in step 140, the process of translating according to the user input and the supplementary input to obtain the translation result corresponding to the user input may specifically include: It is understandable that when users use the translation system to achieve translation goals, the interaction between the two is continuous. After each user input, the system can give a targeted response based on the user input, and at this time, it can be considered that a round of dialogue has ended. When the user provides new information based on the output response, it can be considered that a new round of dialogue has begun. Through multiple rounds of interaction, the translation results corresponding to the user's needs can be finally obtained.

[0055] Based on this, in an embodiment of the present invention, after performing semantic analysis based on the user input of the previous round and determining the output reply based on the analysis result, it can be determined that the previous round of conversation has ended, and the obtained user's supplementary input for the output reply of the previous round can be regarded as the user input of the current round. After that, semantic analysis can be performed based on the user input of the current round, as well as the user input of the round before the current round and the corresponding output reply. If there is a semantically complex problem, a guided reply content is output; conversely, if the above problem does not exist, translation can be directly performed based on the input at this time to obtain a translation result.

[0056] Figure 2 is an example diagram of the translation process provided by the present invention, such as Figure 2 As shown in the figure, after the user's first round of input, the multimodal large model performs semantic analysis based on the first round of input text and input image, and determines through analysis that there is a semantic complexity problem at present, so it outputs a guided reply content, such as "This may be the abbreviation of the name of a corporate organization. In order to better understand the surrounding environment, please slightly turn your phone to take a distant view or supplement the surrounding information." At this time, the first round of dialogue ends.

[0057] After seeing the output response, the user can provide targeted information supplementation and additional input to start the second round of dialogue. At this time, the multimodal large model can perform semantic analysis based on the second round of user input, as well as the first round of user input and corresponding output response. The analysis determines that there are no more semantic complexity issues, so translation can be performed directly. The second round of output response is the translation result corresponding to the user input, which is "This is the International Air Transport Association (IATA)".

[0058] Based on the above embodiment, in step 130, an output reply is determined based on the analysis result obtained by the semantic analysis, and then the following steps are further included: Perform speech synthesis based on the output reply to obtain a reply synthesized speech, and display the reply synthesized speech; In step 140, a translation result corresponding to the user input is obtained, and then the following steps are further included: Speech synthesis is performed based on the translation result to obtain result synthesized speech, and the result synthesized speech is displayed.

[0059] Figure 3 It is the overall framework diagram of the translation process provided by the present invention, such as Figure 3 As shown, after determining the output reply according to the analysis result obtained by semantic analysis, and after translating according to the user input and supplementary input to obtain the translation result, in the embodiment of the present invention, in order to facilitate the user to listen and know the feedback result in time, the output reply of each round can be semantically synthesized to obtain a synthesized speech. Specifically, the speech synthesis engine deployed on the translation system can be used to perform speech synthesis according to the output reply to obtain a reply synthesized speech, and speech synthesis can be performed according to the translation result to obtain a result synthesized speech, and the reply synthesized speech and the result synthesized speech can be played, so that the user can know the system feedback in time, give a targeted reply, or solve the language problem in time to meet their own needs.

[0060] The translation device provided by the present invention is described below. The translation device described below and the translation method described above can be referred to each other.

[0061] Figure 4 is a schematic diagram of the structure of the translation device provided by the present invention, such as Figure 4 As shown, the device comprises: A determination unit 410, configured to determine a user input, wherein the user input includes an input text and an input image; An analysis unit 420, configured to perform semantic analysis based on the input text and the input image, and determine an output response based on an analysis result obtained by the semantic analysis; An acquisition unit 430 is configured to acquire, when the output reply is an input guidance to the user, a supplementary input of the user corresponding to the input guidance; The translation unit 440 is used to perform translation based on the user input and the supplementary input to obtain a translation result corresponding to the user input.

[0062] The translation device provided by the present invention performs semantic analysis according to user input, and determines an output response according to the analysis result. When the output response is a guide to the user's input, the device obtains the user's supplementary input corresponding to the input guide; the device performs translation based on the user input and the supplementary input to obtain a translation result. The device performs translation in an interactive manner. When the analysis determines that an accurate result cannot be obtained at present, the device provides guidance to enable the user to perform targeted information supplementation to obtain more information about the translation target, and performs translation based on the information. The device overcomes the problems of inaccurate, unreliable and impractical translation results in traditional solutions, greatly improves the accuracy and flexibility of translation, can better cope with complex scenarios, reduce ambiguity, obtain accurate and reliable translation results, and can provide effective help to users to optimize user experience.

[0063] Based on the above embodiment, the analysis unit 420 is used for: Performing text encoding and visual encoding based on the input text and the input image, respectively, to obtain text features and image features; A semantic analysis is performed based on the text features and the image features, wherein the semantic analysis includes at least one of polysemous word analysis, cultural field analysis, and language difference analysis.

[0064] Based on the above embodiment, the analysis unit 420 is used for: When the analysis result indicates that the translation target in the input image has at least one of a polysemy problem, a cross-domain problem, and a language difference problem, determining that the output reply is an input guidance for the user; When the analysis result indicates that the translation target in the input image does not have polysemy problems, cross-domain problems, and language difference problems, the output reply is determined to be the translation result corresponding to the user input.

[0065] Based on the above embodiment, the analysis unit 420 is used for: Performing object detection based on the input text and the input image; When the detection result obtained by the target detection indicates that the translation target exists in the input image, text encoding and visual encoding are performed based on the input text and the input image respectively to obtain text features and image features.

[0066] Based on the above embodiment, the translation unit 440 is used to: Use the supplementary input corresponding to the output reply of the previous round as the user input of the current round; Based on the user input of the current round, and the user input and output reply of the previous round of the current round, translation is performed to obtain the translation result.

[0067] Based on the above embodiment, the device further includes a synthetic display unit, which is used for: Performing speech synthesis based on the output reply to obtain a reply synthesized speech, and displaying the reply synthesized speech; Speech synthesis is performed based on the translation result to obtain result synthesized speech, and the result synthesized speech is displayed.

[0068] Figure 5 It is a schematic diagram of the structure of the translation system provided by the present invention. Figure 5 As shown, a multimodal large model is deployed on the system; the multimodal large model is built on the basis of a large language model and a large visual model; The multimodal large model is used to perform semantic analysis based on the input text and input image in the user input, determine the output response based on the analysis results obtained by the semantic analysis, and when the output response is an input guidance for the user, obtain the user's supplementary input corresponding to the input guidance, and translate based on the user input and the supplementary input to obtain a translation result corresponding to the user input.

[0069] Specifically, before translating, users can input their needs and the objects / targets to be translated into the system, so that the system can translate them and finally output the translation results corresponding to the user input. Among them, user needs can be presented in text (input text), and the translation object can be presented in the form of images (input image).

[0070] It should be noted that in the actual translation process, users mostly expect to get immediate feedback when they have translation needs. Therefore, when translating, most of them input quickly in the form of voice rather than entering the text word by word. When the input information includes input voice, in order to facilitate the understanding and semantic analysis of user needs during subsequent translation, the input voice can be converted into text form to obtain input text in the embodiment of the present invention.

[0071] The input text may be a sentence, such as "Please help me translate the letters in the picture?", "Can you explain what the English letters in the picture mean?", etc., or a paragraph consisting of multiple sentences, such as "I am now under a building. There is a large blue sign in the square downstairs, which is a few letters. I am a little curious about what these letters mean. Can you help me translate it?", which is not specifically limited in the embodiment of the present invention. The input text may be directly input by the user, or may be obtained by voice recognition of the input voice, which is not specifically limited in the embodiment of the present invention.

[0072] The input image may be an image directly captured by the user, or may be an image obtained from a real-time recorded video stream or a pre-captured video. The input image may be a single image or multiple images, which is not specifically limited in the embodiment of the present invention.

[0073] After determining the input text and input image, considering the problem that the accuracy and reliability of the translation results cannot be guaranteed in the face of complex translation objects when translating directly based on the information provided by the user in the traditional solution, in the embodiment of the present invention, the user input is first analyzed to determine whether the currently input translation object has complex semantics such as abbreviations, multiple meanings, cross-domains, language differences, etc., which makes it impossible to directly obtain accurate and reliable translation results, and obtain analysis results. That is, the translation object can be semantically analyzed based on the input text and input image to determine whether the translation object has the above problems in the current translation scenario, so as to obtain analysis results. Here, specifically, the input text and input image can be directly input into the multimodal large model of the system, so as to perform analysis with the help of this and obtain analysis results.

[0074] It should be noted that the multimodal large model is constructed on the basis of a large language model and a large visual model. Before it is actually translated and put into use, it has been learned through a large amount of text and image data. It has excellent capabilities in language understanding, information search, comparative query, text translation, etc. Therefore, before translation, the powerful capabilities of this model can be used for semantic analysis to obtain reliable analysis results.

[0075] Furthermore, after obtaining the analysis results, the embodiments of the present invention can output targeted reply content according to the analysis results to obtain accurate and reliable translation results corresponding to the user input. That is, the reply corresponding to the current input, that is, the output reply corresponding to the user input, can be determined according to the analysis results obtained by semantic analysis.

[0076] Specifically, when it is determined through semantic analysis that the currently input translation object has complex semantics and cannot directly produce a reliable translation result, a guided reply content can be provided and output so that the user can supplement the information based on the output reply after knowing the output reply, thereby obtaining the user's supplementary input.

[0077] Correspondingly, when the analysis determines that the currently input translation object does not have the problem of complex semantics and cannot directly produce reliable translation results, the translation can be performed directly. Specifically, under the guidance of the input text, the multimodal large model can be used to translate the translation object / translation target in the input image and output the translation result. Since there is no problem of complex semantics, the translation results obtained through the multimodal large model are accurate and reliable, which can provide practical and effective help to users in actual scenarios and help optimize the user experience.

[0078] After obtaining the output response of the multimodal large model, if the output response is a guided response content, for example, "This may be the abbreviation of the name of a corporate organization. In order to better understand the surrounding environment, please slightly turn your phone to take a distant image, or supplement the surrounding information." That is, the output response is an input guide for the user. At this time, the user's reply information based on the input guide can be obtained, which can also be understood as obtaining the input information when the user supplements the input guide. This input information is called the user's supplementary input. The supplementary input here can be new text provided by the user according to the input guide, or a new image, or new text and a new image, which is not specifically limited in the embodiment of the present invention.

[0079] After that, the translation can be performed according to the user's supplementary input and the original user input to obtain the translation result. That is, after obtaining the input information supplemented by the user according to the output reply, in the embodiment of the present invention, the translation can be performed according to the supplemented input information at this time and the previous input information. Specifically, semantic analysis can be performed based on the supplementary input and the user input to determine whether the semantics cannot be determined by the supplemented information after the user supplements the information, and whether there are still semantic complexity problems. If there are still, the above process can be repeated, and the input guidance for the user can be output again to further obtain information about the translation object and translate it; each input guidance can be determined according to the current input information. On the contrary, if there is no semantic complexity problem, the translation object can be translated using a multimodal large model based on the user input and the supplementary input, and finally the translation result corresponding to the user input can be obtained.

[0080] In an embodiment of the present invention, after each user input, semantic analysis is performed based on the latest input information and the previously input information, and when it is determined that there is no semantic complexity problem, translation is performed through a multimodal large model to obtain a translation result; when this problem exists, a guided reply is given, and the user's supplementary input is obtained, and the above process is repeated until there is no semantic complexity and a reliable translation result cannot be directly obtained; through multiple rounds of analysis and supplementary input, the final translation result can be made more accurate and reliable, can better adapt to specific scenarios, and provide users with more practical and reliable language assistance.

[0081] The translation system provided by the present invention performs semantic analysis according to user input, and determines an output response according to the analysis result. When the output response is a guide to the user's input, the system obtains the user's supplementary input corresponding to the input guide; the system translates based on the user input and the supplementary input to obtain a translation result. The system uses an interactive method to translate. When the analysis determines that an accurate result cannot be obtained at present, the system provides guidance to enable the user to supplement information in a targeted manner to obtain more information about the translation target. The system translates based on this, and overcomes the problems of inaccurate, unreliable and impractical translation results in traditional solutions. The system greatly improves the accuracy and flexibility of translation, can better cope with complex scenarios, reduce ambiguity, obtain accurate and reliable translation results, and can provide effective help to users to optimize user experience.

[0082] Based on the above embodiment, the system is also deployed with a speech synthesis engine and a speech recognition engine, and is configured with a sound pickup device, an image acquisition device and a display device; the sound pickup device and the image acquisition device are used to collect the user's input speech and input image respectively; The speech recognition engine is used to obtain the input speech collected by the sound pickup device, and perform speech recognition on the input speech to obtain the input text; The speech synthesis engine is used to perform speech synthesis based on the output reply and the translation result, respectively, to obtain a reply synthesized speech and a result synthesized speech; The display device is used to display the output reply and the translation result, and / or the reply synthesized speech and the result synthesized speech.

[0083] Specifically, in addition to the above-mentioned multimodal large model, the translation system is also deployed with a speech recognition engine and a speech synthesis engine, as well as a sound pickup device, an image acquisition device, and a display device; the sound pickup device and the image acquisition device are used to collect the user's input voice and input image, respectively. The sound pickup device here can be a microphone, a recorder, etc., the image acquisition device can be a camera, etc., and the display device can be a display screen, an audio player, etc., which is not specifically limited in the embodiment of the present invention.

[0084] The sound pickup device can be connected to a speech recognition engine, and the collected input speech can be input into the speech recognition engine for speech recognition to convert the speech into text, thereby obtaining a transcribed text corresponding to the input speech, that is, the input text.

[0085] After the multimodal large model gives an output response for each round, the speech synthesis engine can perform speech synthesis based on the output response to obtain the response synthesized speech. When the output response is a translation result, speech synthesis can also be performed based on the translation result to obtain the result synthesized speech.

[0086] The display device can display the reply synthesized speech and the result synthesized speech obtained by speech synthesis, and can also display the output reply and translation results of the multimodal large model. It can also display the output reply and translation results, as well as the reply synthesized speech and the result synthesized speech at the same time, which is not specifically limited in the embodiment of the present invention.

[0087] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the translation method, which includes: determining user input, the user input includes input text and input image; performing semantic analysis based on the input text and the input image, and determining an output reply based on the analysis result obtained by the semantic analysis; in the case where the output reply is an input guide for the user, obtaining the supplementary input of the user corresponding to the input guide; performing translation based on the user input and the supplementary input to obtain the translation result corresponding to the user input.

[0088] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0089] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the translation method provided by the above-mentioned methods, and the method includes: determining user input, wherein the user input includes input text and input image; performing semantic analysis based on the input text and the input image, and determining an output response based on the analysis result obtained by the semantic analysis; when the output response is an input guidance for the user, obtaining the supplementary input of the user corresponding to the input guidance; performing translation based on the user input and the supplementary input to obtain a translation result corresponding to the user input.

[0090] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the translation method provided by the above-mentioned methods, the method comprising: determining user input, the user input comprising input text and input image; performing semantic analysis based on the input text and the input image, and determining an output reply based on the analysis result obtained by the semantic analysis; when the output reply is an input guidance for the user, obtaining a supplementary input of the user corresponding to the input guidance; performing translation based on the user input and the supplementary input to obtain a translation result corresponding to the user input.

[0091] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A translation method, characterized in that: include: Determining user input, wherein the user input includes input text and input image; Performing semantic analysis based on the input text and the input image, and determining an output response based on an analysis result obtained by the semantic analysis; In a case where the output reply is an input guidance to the user, obtaining a supplementary input of the user corresponding to the input guidance; Based on the user input and the supplementary input, a translation is performed to obtain a translation result corresponding to the user input.

2. The translation method according to claim 1, characterized in that: The performing semantic analysis based on the input text and the input image includes: Performing text encoding and visual encoding based on the input text and the input image, respectively, to obtain text features and image features; A semantic analysis is performed based on the text features and the image features, wherein the semantic analysis includes at least one of polysemous word analysis, cultural field analysis, and language difference analysis.

3. The translation method according to claim 2, characterized in that: The step of determining the output response based on the analysis result obtained by the semantic analysis includes: When the analysis result indicates that the translation target in the input image has at least one of a polysemy problem, a cross-domain problem, and a language difference problem, determining that the output reply is an input guidance for the user; When the analysis result indicates that the translation target in the input image does not have polysemy problems, cross-domain problems, and language difference problems, the output reply is determined to be the translation result corresponding to the user input.

4. The translation method according to claim 2, characterized in that: The step of performing text encoding and visual encoding based on the input text and the input image to obtain text features and image features includes: Performing object detection based on the input text and the input image; When the detection result obtained by the target detection indicates that the translation target exists in the input image, text encoding and visual encoding are performed based on the input text and the input image respectively to obtain text features and image features.

5. The translation method according to any one of claims 1 to 4, characterized in that: The translating based on the user input and the supplementary input to obtain a translation result corresponding to the user input includes: Use the supplementary input corresponding to the output reply of the previous round as the user input of the current round; Based on the user input of the current round, and the user input and output reply of the previous round of the current round, translation is performed to obtain the translation result.

6. The translation method according to any one of claims 1 to 4, characterized in that: The output response is determined based on the analysis result obtained by the semantic analysis, and then further includes: Performing speech synthesis based on the output reply to obtain a reply synthesized speech, and displaying the reply synthesized speech; The step of obtaining the translation result corresponding to the user input further includes: Speech synthesis is performed based on the translation result to obtain result synthesized speech, and the result synthesized speech is displayed.

7. A translation device, characterized in that: include: A determination unit, used to determine user input, wherein the user input includes input text and input image; An analysis unit, configured to perform semantic analysis based on the input text and the input image, and determine an output response based on an analysis result obtained by the semantic analysis; An acquiring unit, configured to acquire, when the output reply is an input guidance to the user, a supplementary input of the user corresponding to the input guidance; The translation unit is used to translate based on the user input and the supplementary input to obtain a translation result corresponding to the user input.

8. A translation system, characterized in that: A large multimodal model is deployed on the system; the large multimodal model is constructed on the basis of a large language model and a large visual model; The multimodal large model is used to perform semantic analysis based on input text and input images in user input, determine an output response based on the analysis result obtained by the semantic analysis, and when the output response is an input guidance for the user, obtain the user's supplementary input corresponding to the input guidance, and perform translation based on the user input and the supplementary input to obtain a translation result corresponding to the user input.

9. The translation system according to claim 8, characterized in that: The system is also equipped with a speech synthesis engine and a speech recognition engine, as well as a sound pickup device, an image acquisition device and a display device; The sound pickup device and the image acquisition device are used to collect the user's input voice and the input image respectively; The speech recognition engine is used to obtain the input speech collected by the sound pickup device, and perform speech recognition on the input speech to obtain the input text; The speech synthesis engine is used to perform speech synthesis based on the output reply and the translation result, respectively, to obtain a reply synthesized speech and a result synthesized speech; The display device is used to display the output reply and the translation result, and / or the reply synthesized speech and the result synthesized speech.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the translation method according to any one of claims 1 to 6 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the translation method according to any one of claims 1 to 6 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the translation method according to any one of claims 1 to 6 is implemented.

Images