Object information translation and derived information acquisition method and device

By identifying and translating the image features of objects, the problem of only translating text in the prior art is solved, the translation of objects is realized, and the translation quality is optimized under extreme conditions to improve user experience.

CN113407743BActive Publication Date: 2024-11-05BEIJING SAMSUNG TELECOM R&D CENT +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110693514.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2016-04-08
Publication Date
2024-11-05
Estimated Expiration
2036-04-08

AI Technical Summary

Technical Problem

Existing machine translation technology can only translate text and cannot meet users' translation needs for objects. In extreme conditions, such as network restriction and insufficient equipment power, the translation quality and user experience are poor.

Method used

By identifying the image features and text features of the object at the source, using the preset object recognition model and object alignment model, the object information at the target end is translated, and optimized under extreme conditions through information completion and traffic saving schemes.

Benefits of technology

It expands the scope of object translation by machine translation, can translate objects, meet users' object translation needs, and improves translation quality and user experience under extreme conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113407743B_ABST
    Figure CN113407743B_ABST
Patent Text Reader

Abstract

The present invention provides a method and device for translating object information and obtaining derived information, wherein the object information translation method comprises: based on the obtained source object information, translating to obtain target object information corresponding to the source object; and outputting the target object information. The language environment corresponding to the source object is different from the language environment corresponding to the target object. By applying the present invention, the scope of machine translation objects can be expanded, the applicability of translation can be enhanced, and the user's translation needs for objects can be met.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The present invention is a divisional application of the invention patent application with application number 201610218223.4 and invention name “Object information translation, and derivative information acquisition method and device”. Technical Field

[0002] The present invention relates to the field of translation technology, and in particular to a method and device for translating object information and acquiring derived information. Background Art

[0003] Machine translation technology is a technology that automatically translates one language into another. Compared with manual translation, machine translation has the advantages of high translation efficiency, low learning and translation costs, and easy expansion to different languages. Therefore, with the vigorous development of international economic trade, international tourism and other industries, machine translation technology has improved the convenience of users in international communication and its role in international communication is becoming increasingly important.

[0004] According to the different machine translation algorithm frameworks, current machine translation technology can be roughly divided into the following three methods:

[0005] 1) Rule-based machine translation

[0006] Linguists write translation rules, handle rule conflicts, and maintain the rule base. When a sentence is input, a translation is generated based on the translation rules.

[0007] 2) Statistical Machine Translation

[0008] Based on the parallel corpus, it automatically learns the word alignment between parallel sentence pairs, automatically extracts translation rules and estimates translation probability, builds a translation model and performs parameter tuning. When a sentence is input, N translation candidates are generated according to the translation rules, and the candidates are scored and sorted according to the translation model and language model, and the translation candidate ranked first is used as the final translation.

[0009] 3) Neural Machine Translation

[0010] Based on the training method of parallel corpus and recurrent neural network, the network parameters are automatically learned. When a sentence is input, the translation is automatically generated according to the network structure.

[0011] According to the different machine translation input objects, the current machine translation technology can be divided into:

[0012] 1) Text machine translation

[0013] The user enters the text and the text is directly machine translated.

[0014] 2) Voice machine translation

[0015] The voice input signal is detected, speech recognition is performed, the corresponding text is generated, and then the text is machine translated.

[0016] 3) Image machine translation

[0017] The image input is detected, optical character recognition is performed on the image, the text in the image is obtained, and then the text is machine translated.

[0018] The inventor of the present invention has found that the existing machine translation technology usually only focuses on the translation itself, and the content is translated according to the user's input. This passive translation method cannot truly meet the actual needs of users. Moreover, although the input object can be text, voice, or picture, the translation object is only text, including text from voice recognition or text extracted from pictures. If only text is translated, users sometimes still cannot truly understand the object being translated, and the translation needs of users cannot be truly met. Summary of the invention

[0019] In view of the defects of the above-mentioned prior art, the present invention provides a method and device for translating object information and obtaining derived information, so as to expand the scope of machine translation objects, enhance the applicability of translation, and meet users' translation needs for objects.

[0020] The present invention provides an object information translation method, comprising:

[0021] Based on the acquired source-end object information, the target-end object information corresponding to the source-end object is translated;

[0022] Output the target object information.

[0023] Preferably, the language environment corresponding to the source-end object is different from the language environment corresponding to the target-end object.

[0024] Preferably, the translating, based on the acquired source object information, to obtain the target object information corresponding to the source object includes:

[0025] Based on the acquired source object information, identifying the source object to be translated;

[0026] For the identified source object, the corresponding target object information is translated.

[0027] Preferably, the identifying the source object to be translated includes:

[0028] Extracting image features and / or text features based on the acquired source object information;

[0029] According to the extracted image features and / or text features, the source object to be translated is identified.

[0030] Preferably, the step of identifying the source object to be translated based on the extracted image features and / or text features includes:

[0031] According to the preset object recognition model and the extracted image features and / or text features, the source object to be translated is identified.

[0032] Preferably, the object recognition model is pre-constructed in the following manner:

[0033] For each object category, an object recognition model corresponding to the object category is trained based on image features and / or text features of each sampled object belonging to the object category.

[0034] Preferably, the identifying the source object to be translated based on a preset object recognition model and extracted image features and / or text features includes:

[0035] Identify the object category of the source object to be translated based on the extracted image features and / or text features;

[0036] According to the object recognition model corresponding to the object category and the extracted image features and / or text features, the source object to be translated is identified.

[0037] Preferably, for the identified source object, the corresponding target object information is translated, including:

[0038] Determine the language environments corresponding to the source object and the target object respectively;

[0039] Based on the language environments corresponding to the source-end object and the target-end object respectively, the target-end object information corresponding to the source-end object is translated.

[0040] Preferably, the language environment corresponding to the source object is determined according to at least one of the following:

[0041] The language environment corresponding to the location detected by the location detection module, the language environment identified from the acquired source object information, the language environment corresponding to the source object searched in the preset object knowledge graph database, and the language environment set by the user;

[0042] The language environment corresponding to the target object is determined according to at least one of the following:

[0043] The language environment corresponding to the location detected by the location detection module, the pre-specified language environment, the language environment set by the user, and the language environment determined based on the user's personal information.

[0044] Preferably, based on the language environments corresponding to the source object and the target object respectively, translating to obtain the target object information corresponding to the identified source object includes:

[0045] Selecting a corresponding object alignment model based on the language environments corresponding to the source object and the target object;

[0046] Based on the selected object alignment model, the target object information corresponding to the identified source object is translated.

[0047] Preferably, based on the selected object alignment model, the target end object information corresponding to the identified source end object is translated, including:

[0048] Based on the source object information, obtain text features and / or image features corresponding to the source object;

[0049] Based on the acquired text features and / or image features and the selected object alignment model, target-side object information corresponding to the identified source-side object is translated.

[0050] Preferably, the object alignment model is pre-constructed in the following manner:

[0051] Determine text features and / or image features between a sampled source end object and a corresponding sampled target end object;

[0052] An object alignment model is trained based on the determined text features and / or image features.

[0053] Preferably, the object information translation method further includes:

[0054] Optimizing the object alignment model based on at least one of the following information:

[0055] The user's feedback information on the target object information obtained by translation, the language environment corresponding to the source object, and the language environment corresponding to the target object.

[0056] Preferably, there are multiple source objects, and the multiple source objects belong to the same category of combined objects;

[0057] The translation obtains target-end object information corresponding to the source-end object, including at least one of the following:

[0058] For source-side composite objects corresponding to multiple source-side objects, corresponding target-side composite object information is obtained by translation;

[0059] For each source object, the corresponding target object information is translated respectively.

[0060] Preferably, when there are multiple target objects, outputting the target object information includes:

[0061] Arrange and sort the object information at each target end and then output it.

[0062] Preferably, the target end object information is sorted according to at least one of the following information:

[0063] Correlation with the source object, user behavior with respect to the target object information, attribute information of the target object, and user personal information.

[0064] Preferably, the source object information includes at least one of the following:

[0065] Multimedia information corresponding to the source object, text information identified from the multimedia information corresponding to the source object, location information corresponding to the source object, searched source object related information, and source object related information input by the user.

[0066] Preferably, the source object information includes multimedia information corresponding to the source object;

[0067] Obtain source object information in the following ways:

[0068] Multimedia information is collected in real time through multimedia collection equipment, and the real-time collected multimedia information is used as source object information.

[0069] Preferably, there are multiple source objects;

[0070] The object information translation method further includes:

[0071] Detecting a user's selection operation on the target object information obtained by translation;

[0072] The source end object corresponding to the target end object information selected by the user is located in the multimedia information collected in real time.

[0073] Preferably, outputting the target end object information includes:

[0074] Collect multimedia information in real time through multimedia acquisition equipment;

[0075] The target end object corresponding to the source end object is located in the multimedia information collected in real time.

[0076] Preferably, before translating the acquired source object information to obtain the target object information corresponding to the source object, the method further includes:

[0077] Predicting the user's translation intention and pre-storing a processing model for offline translation based on the user's translation intention;

[0078] Based on the acquired source object information, the target object information corresponding to the source object is translated, including:

[0079] Based on the acquired source-end object information, the target-end object information corresponding to the source-end object is translated and obtained by utilizing the stored processing model.

[0080] Preferably, the user's translation intention is predicted based on at least one of the following information:

[0081] User schedule, user personal information, device environment information, and device motion status.

[0082] Preferably, the outputting the target end object information includes:

[0083] Output the target object information in at least one of the following ways: text, image, audio, video.

[0084] Preferably, the outputting the target end object information includes:

[0085] Based on at least one of the following, the target end object information is adaptively adjusted, and the adjusted target end object information is output:

[0086] Device type, device storage status, current network status, device power status, and user personal information.

[0087] Preferably, the object information translation method further includes:

[0088] Acquire the derived information associated with the target object and / or the derived information associated with the source object;

[0089] The obtained derivative information is output.

[0090] Preferably, obtaining the derivative information associated with the target end object includes:

[0091] Searching for attribute information corresponding to a preset attribute of the target object from a preset object knowledge graph database;

[0092] Confirming the searched attribute information as derived information associated with the target end object;

[0093] Obtaining derivative information associated with the source object, including:

[0094] Searching for attribute information corresponding to the preset attribute of the source object from a preset object knowledge graph database;

[0095] The searched attribute information is confirmed as the derived information associated with the source object.

[0096] Preferably, the preset attribute is determined according to the category of the target end object and / or the source end object.

[0097] Preferably, obtaining the derivative information associated with the target end object includes:

[0098] Determining derivative information associated with the target end object based on the position information corresponding to the target end object;

[0099] Obtaining derivative information associated with the source object, including:

[0100] Based on the position information corresponding to the source object, the derived information associated with the source object is determined.

[0101] Preferably, if the derived information associated with the target end object and the derived information associated with the source end object are obtained, then outputting the obtained derived information includes:

[0102] Locating the derived information that needs to be highlighted according to the correlation between the derived information associated with the target-end object and the derived information associated with the source-end object;

[0103] Output the derived information obtained and highlight the located derived information.

[0104] Preferably, there are multiple source objects, and the multiple source objects belong to the same category of combined objects;

[0105] The derived information associated with the target object information obtained by translation is obtained, including at least one of the following:

[0106] For source-side composite objects corresponding to multiple source-side objects, obtaining derivative information associated with corresponding target-side composite object information;

[0107] For each source object, the derived information associated with the corresponding target object information is obtained.

[0108] Preferably, outputting the obtained derivative information includes:

[0109] Determining, based on the user personal information, a language environment corresponding to the obtained derivative information;

[0110] The derived information is displayed based on the determined locale.

[0111] Preferably, the object information translation method further includes:

[0112] Based on the user's personal information, locate the derivative information that needs to be highlighted;

[0113] Highlight the located derivative information.

[0114] Preferably, the object information translation method further includes:

[0115] Generate or change related reminder events based on the user personal information and / or the derived information obtained.

[0116] Preferably, based on the acquired source object information, the target object information corresponding to the source object is translated, including:

[0117] Obtain source object information to be sent to the recipient;

[0118] Based on the language environment of the receiving party and the acquired source object information, translating to obtain target object information corresponding to the source object;

[0119] Outputting the target end object information includes:

[0120] The target end object information obtained by translation is sent to the receiving party.

[0121] The present invention also provides a method for obtaining derivative information, comprising:

[0122] Determining derivative information associated with the object based on the acquired object information;

[0123] The determined derivative information is output.

[0124] Preferably, the determining of the derived information associated with the object based on the acquired object information includes:

[0125] Based on the acquired object information, identify the corresponding object;

[0126] Searching for attribute information corresponding to preset attributes of the identified object from a preset object knowledge graph database;

[0127] The searched attribute information is confirmed as the derived information associated with the object.

[0128] Preferably, the preset attribute is determined according to the object category of the object.

[0129] Preferably, the determining of the derived information associated with the object based on the acquired object information includes:

[0130] Based on the acquired object information, identify the corresponding object;

[0131] Derivative information associated with the object is determined based on the position information corresponding to the object.

[0132] Preferably, the object is a plurality of objects, and the plurality of objects belong to the same category of combined objects;

[0133] Determining derivative information associated with the object, including at least one of the following:

[0134] For a composite object corresponding to the plurality of objects, determining derivative information associated with the composite object;

[0135] For multiple objects, the derived information associated with each object is obtained respectively.

[0136] Preferably, the output determined derivative information includes:

[0137] Determining, based on the user personal information, a language environment corresponding to the obtained derivative information;

[0138] The derived information is displayed based on the determined locale.

[0139] Preferably, the derivative information acquisition method further includes:

[0140] Based on the user's personal information, locate the derivative information that needs to be highlighted;

[0141] Highlight the located derivative information.

[0142] Preferably, the derivative information acquisition method further includes:

[0143] Generate or change related reminder events based on the user's personal information and / or the derived information obtained.

[0144] Based on the above object information translation method, the present invention also provides an object information translation device, which includes:

[0145] An object information translation unit, used for translating the acquired source object information to obtain target object information corresponding to the source object;

[0146] The information output unit is used to output the target end object information translated by the object information translation unit.

[0147] Based on the above-mentioned derivative information acquisition method, the present invention also provides a derivative information acquisition device, which includes:

[0148] A derivative information acquisition unit, configured to determine derivative information associated with the object based on the acquired object information;

[0149] The information output unit is used to output the derivative information determined by the derivative information acquisition unit.

[0150] The technical solution of the present invention is to translate the object itself, not just the text, so it can solve the problem that the translation of text cannot fully cover the translation task. Foreign objects that are unfamiliar to the user can be translated into domestic corresponding objects that the user is familiar with; or the translation direction can be exchanged to translate domestic objects into corresponding objects in the target country. In this way, compared with the existing translation objects that are only text, the solution provided by the present invention can meet the user's translation needs for objects, expand the scope of machine translation objects, and enhance the applicability of translation.

[0151] The solution of the present invention also proposes the concept of derived information. The solution of the present invention can analyze the object, provide the derived information about the object, and enhance the user's understanding of the object information.

[0152] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0153] Figure 1 Schematic diagram of the process of the object information translation method according to the first embodiment of the present invention;

[0154] Figure 2 A schematic diagram of a single object input according to a second embodiment of the present invention;

[0155] Figure 3 A schematic diagram of multi-object input according to a second embodiment of the present invention;

[0156] Figure 4 A schematic diagram of interactive object translation of single object augmented reality according to the third embodiment of the present invention;

[0157] Figure 5 This is a schematic diagram of the derivative translation of a single object according to the fourth embodiment of the present invention;

[0158] Figure 6 Schematic diagram of automatic comparison of source and target objects according to the fourth embodiment of the present invention;

[0159] Figure 7 This is a schematic diagram of multi-object derivative translation according to the fourth embodiment of the present invention;

[0160] Figure 8 This is a schematic diagram of automatic comparison of two ends of multiple objects according to a fourth embodiment of the present invention;

[0161] Fig. 9 This is a schematic diagram of personalized derivative information according to Embodiment 5 of the present invention;

[0162] Fig.10 This is a schematic diagram of disambiguating translation results based on location according to Embodiment 5 of the present invention;

[0163] Fig.11 This is a schematic diagram of object translation in a social network according to a fifth embodiment of the present invention;

[0164] Fig.12 This is a schematic diagram of slogan derivation translation according to the fifth embodiment of the present invention;

[0165] Figure 13(a) , 13(b) 13(c) is a schematic diagram of the derivative translation-automatic reminder function of the fifth embodiment of the present invention;

[0166] Figure 14(a) , 14(b) Schematic diagram of multi-modal output of Embodiment 5 of the present invention;

[0167] Figure 15(a) , 15(b) This is a schematic diagram of object information translation on a smart watch according to Embodiment 6 of the present invention;

[0168] Fig.16 A schematic diagram of a derivative translation-automatic reminder function on a smart watch according to a sixth embodiment of the present invention;

[0169] Figure 17(a) , 17(b) This is a schematic diagram of object information translation on smart glasses according to Embodiment 6 of the present invention;

[0170] Figure 18(a) , 18(b) This is a schematic diagram of machine translation of a sign applied to smart glasses according to Embodiment 6 of the present invention;

[0171] Fig.19 This is a schematic diagram of collecting user behavior logs according to Embodiment 6 of the present invention;

[0172] Fig. 20 This is a schematic diagram of a crowdsourcing translation feedback mechanism according to Embodiment 6 of the present invention;

[0173] Fig.21 A schematic diagram of a method for a user to input an answer according to Embodiment 6 of the present invention;

[0174] Fig. 22 This is a schematic diagram of incomplete standard input according to Embodiment 7 of the present invention;

[0175] Figure 23(a) , 23(b) 23(c) is a schematic diagram of the translation of dishes according to a specific example of the seventh embodiment of the present invention;

[0176] Figure 24(a) , 24(b) 24(c) and 24(d) are schematic diagrams of translation of dishes according to another specific example in Embodiment 7 of the present invention;

[0177] Fig.25 A schematic diagram of a database classified according to user intention downloading in Embodiment 7 of the present invention;

[0178] Fig.26 Schematic diagram of the structure of an object information translation device according to Embodiment 8 of the present invention;

[0179] Fig. 27 Schematic diagram of the flow of the method for obtaining derivative information according to Embodiment 9 of the present invention;

[0180] Fig.28 It is a structural diagram of a derivative information acquisition device according to Embodiment 9 of the present invention. DETAILED DESCRIPTION

[0181] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0182] The terms "module", "device", etc. used in this application are intended to include computer-related entities, such as but not limited to hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a module can be, but is not limited to: a process running on a processor, a processor, an object, an executable program, a thread of execution, a program, and / or a computer. For example, an application running on a computing device and the computing device can both be modules. One or more modules can be located in a process and / or thread in execution, and a module can also be located on one computer and / or distributed between two or more computers.

[0183] The inventor of the present invention has found that in the existing translation scheme, although the input object can be text, voice, or picture, the object of translation is only text, including text recognized from voice, or text extracted from pictures. In actual applications, if only text is translated, users sometimes still cannot truly understand the translated object. For example, the name of a drug in a foreign pharmacy, or the name of a product with a strong localized naming feature in a foreign store, etc., may be difficult for users to understand even if the name is translated. Moreover, in extreme cases, the content that the user wants to translate may not even have text, such as the various driving signs encountered when traveling by car abroad. Therefore, the translation of text cannot fully cover the needs of the translation task, and only translating text cannot truly meet the translation needs of users.

[0184] Therefore, the inventors of the present invention consider that the target-end object information corresponding to the source-end object can be translated based on the acquired source-end object information. For example, the source-end object to be translated can be identified based on the acquired source-end object information; for the identified source-end object, the corresponding target-end object information is translated. Afterwards, the translated target-end object information is output. In this way, foreign source-end objects that are unfamiliar to the user, such as medicines, foods, cosmetics, road signs, etc., can be translated into corresponding domestic target-end objects that are familiar to the user; or the translation direction can be exchanged to translate the domestic source-end objects into the corresponding target-end objects in the target country, thereby meeting the user's translation needs for objects, expanding the scope of machine translation objects, and enhancing the applicability of translation.

[0185] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0186] Embodiment 1

[0187] Embodiment 1 of the present invention provides an object information translation method, such as Figure 1 As shown, the specific process may include the following steps:

[0188] S101: Identify source objects to be translated based on the acquired source object information.

[0189] Those skilled in the art will appreciate that, in the scheme of the present invention, an "object" refers to an individual whose shape can be captured by a sensor and described by language.

[0190] In the solution of the present invention, the object to be translated can be called a source object, and the translated object can be called a target object. There are one or more source objects to be translated; when there are multiple source objects, the multiple source objects can be a combination of objects belonging to the same category, or multiple objects belonging to different categories. The translation solutions for a single source object and multiple source objects will be described in detail in Example 2.

[0191] Source object information refers to information related to the source object, and may include at least one of the following modalities: text, picture, audio, video, etc.

[0192] In practical applications, the source object information can be directly captured by the terminal, such as by photographing with a camera. Alternatively, the source object-related information can be obtained from the network or other devices. The acquired source object information can include at least one of the following: multimedia information corresponding to the source object, text information identified from the multimedia information corresponding to the source object, location information corresponding to the source object, searched source object-related information, and source object-related information input by the user.

[0193] When the source object information includes multimedia information corresponding to the source object, the source object information may be obtained in the following manner: the multimedia information is collected in real time by a multimedia collection device, and the real-time collected multimedia information is used as the source object information.

[0194] The multimedia information corresponding to the source object may include but is not limited to at least one of the following: object image, object audio information, object video information, etc.

[0195] The searched source object related information contained in the source object information may be the source object related information searched from the network side. For example, when the recognition result is inaccurate, or when the source object information is incomplete, relevant information may be searched (from the network side or from locally stored resources), or the user may be prompted to input source object related information, so as to accurately identify the source object to be translated.

[0196] The position information corresponding to the source object contained in the source object information can be used to optimize a processing model (such as an object alignment model) for translation, thereby obtaining a more accurate translation result based on the position information.

[0197] Furthermore, the source object information may further include user personal information, so as to perform personalized translation based on the user personal information later.

[0198] The terminals include mobile devices such as smart phones and PADs (portable android devices), and may also include wearable devices such as smart watches and smart glasses.

[0199] After the source object information is obtained, image features and / or text features may be extracted based on the obtained source object information; and the source object to be translated may be identified based on the extracted image features and / or text features. For example, the source object to be translated may be identified based on the extracted image features and / or text features according to a preset object recognition model.

[0200] Among them, the object recognition model is pre-built, which can be built in the following way:

[0201] For a plurality of preset sampling objects, multimodal data of each sampling object is collected; image features and / or text features of the sampling object are extracted from the multimodal data; and an object recognition model is obtained by learning and training based on the image features and text features of the sampling object.

[0202] The multimodal data of the sampled object includes at least one of the following data:

[0203] Text, pictures, audio, video, etc. related to the sampled object.

[0204] Furthermore, the object recognition model may be pre-constructed in the following manner: for each object category, based on the image features and / or text features of each sampled object belonging to the object category, an object recognition model corresponding to the object category is trained.

[0205] In this way, after extracting image features and / or text features based on the acquired source object information, the object category of the source object to be translated can be identified based on the extracted image features and / or text features; the source object to be translated can be identified based on the object recognition model corresponding to the object category and the extracted image features and / or text features.

[0206] The inventors of the present invention have found that the quality of machine translation is highly dependent on the degree of normalization of the input text, the size of the database and the model. Therefore, existing commercial machine translation technologies usually require the support of networks and cloud services, as well as text normalization. However, current machine translation technologies do not take into account many extreme conditions, such as limited network, insufficient device power, incomplete input text / sound / image information, etc., resulting in poor user experience.

[0207] Therefore, the present invention can provide an optimization solution for the object information translation method under extreme conditions, including a solution for solving incomplete input object information, a solution for saving traffic and battery, and an object information translation output solution for the device under different conditions. For details, please refer to the subsequent Example 7.

[0208] Considering the scenario of incomplete information, it may be impossible to identify the source object to be translated based on the acquired source object information, or the recognition result may be inaccurate.

[0209] Therefore, in order to ensure the integrity of the input information of the object and the accuracy of the recognition result, in the first embodiment of the present invention, after extracting image features and / or text features based on the acquired source object information, if the source object to be translated is not recognized based on the extracted image features and / or text features, other information related to the source object to be translated (such as text and / or picture information) can be obtained based on the extracted image features and / or text features. Then, according to the preset object recognition model, the extracted image features and / or text features, and the acquired text and / or picture information, the source object to be translated is accurately identified.

[0210] The text and / or image information related to the source object to be translated can be obtained by at least one of the following methods:

[0211] Performing web search based on extracted image features and / or text features;

[0212] Based on the extracted image features and / or text features, matching and searching the locally stored pictures and / or texts;

[0213] Obtain source object related information input by the user, such as text and / or image information.

[0214] For example, the terminal device collects image information, text information, or audio information, and then converts the audio into text, and then performs object recognition; if the source object information collected by the terminal device is not sufficient to identify the object, the network search engine is started to search and match the images and text to obtain more images and texts related to the object, and then the obtained image and text information is filtered and normalized for recognition. If it still cannot be recognized, more images or text information are input, and this process is repeated until the source object is recognized, or the number of repetitions exceeds a threshold. In practical applications, in view of the characteristics of dish translation, such as dishes (cooked food) usually do not have product packaging, and in extreme cases, only the names of dishes are listed on the menu, the above translation scheme for incomplete input object information can be used to achieve matching and recognition of source dishes.

[0215] S102: For the identified source object, translate and obtain corresponding target object information.

[0216] Specifically, the language environments corresponding to the source object and the target object may be determined; then, based on the language environments corresponding to the source object and the target object, the target object information corresponding to the source object is translated.

[0217] Among them, the language environment corresponding to the source-end object is different from the language environment corresponding to the target-end object.

[0218] In practical applications, the language environment can be expressed as a language type. For example, when the source object and the target object contain text content, the language types of the text content are different. Or, when the source object and the target object do not contain text, such as only containing patterns, the language types of the environment in which the source object and the target object are located are different, such as traffic signs in different countries.

[0219] In practical applications, after the source object is identified, the translation direction can be determined; and the translation direction can be determined based on the source language of the source object and the target language after translation. The source language of the source object specifically refers to the language environment corresponding to the source object; and the target language specifically refers to the language environment corresponding to the target object corresponding to the source object.

[0220] In the first embodiment of the present invention, the language environment corresponding to the source object may be determined according to at least one of the following:

[0221] The language environment corresponding to the location detected by the location detection module, the language environment identified from the acquired source object information, the language environment corresponding to the source object searched in the preset object knowledge graph database, and the language environment set by the user. For example, the country and language of the source object are automatically detected through the GPS (Global Positioning System) module.

[0222] The language environment corresponding to the target object can be determined according to at least one of the following:

[0223] The language environment corresponding to the location detected by the location detection module, the pre-specified language environment, the language environment set by the user, and the language environment determined based on the user's personal information.

[0224] For example, the language environment of the system may be pre-specified as the language environment corresponding to the target object. Alternatively, if the user setting the target language on the terminal device is detected, the target language set by the user is used as the language environment corresponding to the target object; if the user setting the target language is not detected, the language environment of the default device system is used as the language environment corresponding to the target object.

[0225] In the first embodiment of the present invention, after the language environments corresponding to the source-end object and the target-end object are respectively determined, corresponding object alignment models may be selected based on the language environments corresponding to the source-end object and the target-end object.

[0226] Afterwards, based on the selected object alignment model, the target object information corresponding to the identified source object is translated. Specifically, based on the source object information, the text features and / or image features corresponding to the source object can be obtained; based on the obtained text features and / or image features and the selected object alignment model, the target object information corresponding to the identified source object is translated.

[0227] Among them, the object alignment model is pre-built in the following way:

[0228] Determine text features and / or image features between a sampling source end object and a corresponding sampling target end object; and train an object alignment model based on the determined text features and / or image features.

[0229] In practical applications, the multimodal data of multiple sampling source objects and the multimodal data of the sampling target objects corresponding to each sampling source object in different translation directions can be obtained in advance; image features and / or text features are extracted from the multimodal data; based on the extracted image features and / or text features, the text features and / or image features between the sampling source object and the sampling target object are determined; and according to the text features and / or image features between the sampling source object and the sampling target object, the object alignment model is learned and trained. Among them, the multimodal data includes at least one of the following data: text, pictures, audio, and video related to the object.

[0230] Preferably, in order to improve the quality of object translation, in the first embodiment of the present invention, the object alignment model is optimizable. Specifically, after obtaining the target end object information through this object information translation, the object alignment model can be optimized according to at least one of the following information:

[0231] The user's feedback information on the target object information obtained by translation, the language environment corresponding to the source object, and the language environment corresponding to the target object.

[0232] For example, user behavior data for target object information can be collected, and the collected user behavior data can be used as feedback information to update and optimize the object alignment model. Specifically, the location information of the source object can be detected by the terminal device, and it can be automatically determined whether it is a source of the source object based on the location characteristics. If it is, it is added to the location information in the knowledge graph of the source object. The alignment model parameters are updated by collecting a series of user log data such as the click rate of the user on the translation candidate list and the like rate. If a target object receives a high click rate or like rate, its alignment probability is increased, otherwise its alignment probability is reduced.

[0233] More preferably, in the first embodiment of the present invention, when there are multiple source objects to be translated, after the source objects to be translated are identified, the categories of each source object can also be detected. If the source objects to be translated belong to the same category of combined objects, all the source objects are translated as source combined objects. If the source objects to be translated do not belong to the same category, each source object can be translated separately to obtain the corresponding target object information.

[0234] Specifically, when there are multiple source objects and the multiple source objects belong to the same category of combined objects, translating and obtaining target object information corresponding to the source objects can be achieved by at least one of the following:

[0235] For source-side composite objects corresponding to multiple source-side objects, corresponding target-side composite object information is obtained by translation;

[0236] For each source object, the corresponding target object information is translated respectively.

[0237] In practical applications, there may be one or more target objects corresponding to a source object. Therefore, when there are multiple source objects, relevant information of one or more corresponding target objects may be translated for each source object.

[0238] More preferably, in the first embodiment of the present invention, based on the acquired source object information, before translating the target object information corresponding to the source object, the user's translation intention can be predicted, and a processing model for offline translation can be pre-stored according to the user's translation intention.

[0239] In this way, the target object information corresponding to the source object can be translated in the following manner: based on the acquired source object information, the target object information corresponding to the source object is translated using the stored processing model.

[0240] The user's translation intention can be predicted based on at least one of the following information:

[0241] User schedule, user personal information (such as interests and hobbies), device environment information, and device motion status.

[0242] The predicted translation intention may include: an action trajectory and / or a source object to be translated.

[0243] In practical applications, the processing models used for offline translation may include: object recognition models for offline translation, object knowledge graph databases, object alignment models, etc. Among them, the object knowledge graph database for offline use can be set according to the object categories and action trajectories of the source objects to be translated in the predicted translation intent, without downloading object knowledge data of other less relevant object categories, thereby reducing the database size.

[0244] For example, the pre-set models and databases in the system can be compressed and filtered first, and then the databases can be classified according to the categories of objects. The terminal predicts the user's translation intention in advance and downloads the relevant pre-filtered, compressed, and classified models and databases without consuming the user's mobile network data traffic.

[0245] In this way, when the terminal device detects that it is currently using mobile network data traffic, it can give priority to searching and translating based on the model and database downloaded to the local device, reducing the communication between the device and the cloud service. At the same time, since the database size has been greatly reduced, the search space in the translation process has been reduced, avoiding a large number of calculations, thereby saving power. The object information translation solution for saving traffic and battery will be further supplemented in Example 7.

[0246] S103: Output target end object information.

[0247] In the first embodiment of the present invention, the target end object information can be output in a multimodal manner. For example, the target end object information can be output in at least one of the following ways: text, image, audio, and video. Furthermore, the output mode of the target end object information can be adaptively adjusted based on at least one of the following: user operation, terminal environment, network conditions, terminal type, and power status.

[0248] The target object information includes but is not limited to at least one of the following: a picture, name, location information, price information, category information, and related audio / video information of the target object.

[0249] In actual applications, when the source object information is collected and obtained by the terminal device, the translated target object information can be displayed and output in the following ways:

[0250] Split screen to display source object information and target object information; or

[0251] By using augmented reality display technology, the target object information is displayed in the capture area of ​​the source object information.

[0252] More preferably, in the first embodiment of the present invention, the translated target object information can be adaptively adjusted based on at least one of the following items, and the adjusted target object information can be output:

[0253] Device type, device storage status, current network status, device power status, and user personal information.

[0254] Preferably, in the first embodiment of the present invention, the target object information can be obtained by: obtaining the source object information to be sent to the receiver; translating the target object information corresponding to the source object based on the language environment of the receiver and the obtained source object information. Then, the translated target object information can be sent to the receiver.

[0255] Therefore, the translation method provided by the present invention can be applied to social networks, and adaptive object information translation can be performed in social networks. During the translation process, personal information of social network users, such as their region, nationality, etc., can be collected to determine the language environment corresponding to the source object and the language environment corresponding to the target object, that is, to determine the translation direction of the object information. Then, a corresponding translation result is given for each translation direction. Among them, the translation direction is determined by the language environment corresponding to the source object and the language environment corresponding to the target object; in a social process, for the same source object, different language environments of the target objects can be determined according to the different regions and nationalities of the recipients who view the translation results.

[0256] For a solution for translating object information in a social network, please refer to Example 5 provided later.

[0257] In practical applications, when there are multiple target objects, the information of each target object can be sorted and output. Specifically, the information of each target object can be sorted according to at least one of the following information: relevance to the source object, user behavior on the target object information, attribute information of the target object, and user personal information.

[0258] Among them, the user's personal information includes information such as the user's preferences, and may also include information such as the user's current location (which can be obtained through the device location). The attribute information of the target object may include attributes such as price; based on the user's personal information and the attribute information of the target object, the user's purchase possibility can be comprehensively judged.

[0259] For example, when there are multiple target objects, the multiple target objects can be arranged based on the correlation with the source object, the user click frequency, the purchase difficulty, etc., so as to arrange the target objects in order of priority. The user click frequency can be obtained by collecting the user's operation log on the target object, extracting the number of clicks on the relevant object from the log data, and calculating the click frequency of each target object. The method for obtaining the purchase difficulty can comprehensively consider the distance between the current terminal location and the purchase location, or the purchase price, etc.

[0260] In order to present the translation results to the user more intuitively, and to provide feedback and quick response to the user's needs in real time, in the first embodiment of the present invention, if multimedia information is collected in real time by a multimedia collection device, and the real-time collected multimedia information is used as the source object information, then in the case of multiple source objects, the user's selection operation on the target object information obtained by translation can also be detected; and the source object corresponding to the target object information selected by the user can be located in the real-time collected multimedia information. Alternatively, the user's selection operation on the source object can also be detected. If the selection operation of a certain source object is detected, the target object information corresponding to the source object selected by the user can be located according to the preset output method.

[0261] More preferably, in order to improve the translation quality, in the first embodiment of the present invention, the translation result with the crowdsourcing translation feedback weight can be obtained based on the obtained source object information; and the target object information corresponding to the source object can be adjusted according to the translation result with the crowdsourcing translation feedback weight.

[0262] The translation results with crowdsourcing translation feedback weights can be obtained in the following ways:

[0263] A crowdsourcing translation request for source object information is sent to a plurality of preset answering users; translation results fed back by each answering user for the crowdsourcing translation request are collected; and a crowdsourcing translation feedback weight of each translation result is determined according to the category of the answering user and the frequency of occurrence of the translation result.

[0264] More preferably, in the first embodiment of the present invention, after obtaining the target-end object information, the output of the target-end object information can be adjusted based on the user operation on the source-end object information.

[0265] For example, when the user operation is specifically an activation operation of the browsing mode of the target object, the target object information can be output in the following manner: real-time acquisition of multimedia information by a multimedia acquisition device; locating the target object corresponding to the source object in the real-time acquired multimedia information. For example, real-time capture of text and / or images of multiple objects; if there is an object matching the target object in the captured text and / or image, the object is marked and displayed using the display method corresponding to the browsing mode.

[0266] When there are multiple source objects and the user operation is specifically an activation operation of the multi-object interaction mode of the target object, the selected target object can be detected; and the source object corresponding to the selected target object is marked and displayed.

[0267] Considering that traditional machine translation technology usually only focuses on translation itself, in real life, users often need not only translation, but also understanding of the translation content and acquisition of relevant knowledge. For example, when entering the name of a foreign drug, traditional machine translation will only translate accordingly based on the input, without further understanding whether the translated content corresponds to a named entity, and if so, whether it is necessary to provide relevant useful information such as efficacy, dosage, applicable population, and even purchase method. This passive translation method of translating what the user enters cannot truly meet the actual needs of users.

[0268] Preferably, in order to allow users to better understand the translation content, the object information translation method provided in the first embodiment of the present invention may also include a derivative translation scheme. Specifically, derivative information associated with the target end object and / or derivative information associated with the source end object may be obtained; and the obtained derivative information may be output.

[0269] In the first embodiment of the present invention, the derivative information associated with the target object can be obtained in the following manner: searching for attribute information corresponding to preset attributes of the target object from a preset object knowledge graph database; and confirming the searched attribute information as the derivative information associated with the target object.

[0270] Accordingly, the derived information associated with the source object can be obtained in the following manner: searching for attribute information corresponding to preset attributes of the source object from a preset object knowledge graph database; and confirming the searched attribute information as the derived information associated with the source object.

[0271] The preset attribute is determined according to the category of the target end object and / or the source end object.

[0272] For example, the derived information of the target object may include: key information related to the target object obtained according to the object category of the target object through a pre-built object knowledge graph database.

[0273] Furthermore, the derived information obtained may also include: comparison information between the source object and the corresponding target object, that is, the derived information of the source object and the derived information of the target object are displayed simultaneously. The comparison information is determined by the key information (attribute information of preset attributes) related to the source object and the corresponding target object. The key information related to the target object and the key information related to the source object can be obtained by pre-building an object knowledge graph database.

[0274] Therefore, more preferably, in the first embodiment of the present invention, the derived information associated with the target object can be determined based on the position information corresponding to the target object; and the derived information associated with the source object can be determined based on the position information corresponding to the source object.

[0275] In the first embodiment of the present invention, if the derivative information associated with the target-end object and the derivative information associated with the source-end object are obtained, the derivative information that needs to be highlighted can be located according to the correlation between the derivative information associated with the target-end object and the derivative information associated with the source-end object; the obtained derivative information is output, and the located derivative information is highlighted.

[0276] In the case where there are multiple source objects and the multiple source objects belong to the same category of combined objects, obtaining the derivative information associated with the target object information obtained by translation includes at least one of the following:

[0277] For source-side composite objects corresponding to multiple source-side objects, obtaining derivative information associated with corresponding target-side composite object information;

[0278] For each source object, the derived information associated with the corresponding target object information is obtained.

[0279] Furthermore, in order to make the output result more in line with the personalized needs of the user, in the first embodiment of the present invention, after the derivative information is obtained, the obtained derivative information can be output in the following manner:

[0280] Determine the language environment corresponding to the derived information obtained based on the user's personal information;

[0281] Displays derived information obtained based on the determined locale.

[0282] More preferably, in the first embodiment of the present invention, the derivative information that needs to be highlighted can be located according to the user's personal information; and the located derivative information can be highlighted.

[0283] For example, the derived information of the target object may be rated according to the correlation between the user's personal information and the derived information of the target object; and the derived information of the target object that needs to be highlighted may be located according to the rating.

[0284] Furthermore, it is also possible to generate or change related reminder events based on the user's personal information and / or the derived information obtained. Specifically, it is possible to detect whether there is a reminder event related to the target object in the terminal device; if not, a reminder event can be generated based on the user's personal information and / or the derived information obtained. If so, the reminder event related to the target object is changed based on the user's personal information and / or the derived information obtained.

[0285] Among them, user personal information may include at least one of the following: user's average work and rest time, user health status, user movement status, etc.; the derived information of the target object includes at least one of the following: time interval, start and end time, number of repetitions, etc. The user's average work and rest time can be determined by at least one of the following contents: power on and off time, user's daily activity time. The user's health status can be determined by at least one of the following contents: data recorded in health applications, blood pressure pulse sensors, etc. The user's movement status can be determined by at least one of the following contents: GPS records movement trajectory, gyroscope records movement speed, etc.

[0286] In practical applications, it is possible to detect whether there is a reminder event related to the target object in the terminal device by: judging whether there is a reminder event related to the target object according to the content relevance between the reminder event set in the terminal device and the target object. The derivative translation scheme for event reminder based on user personal information will be introduced in detail in the subsequent embodiment 5.

[0287] Because different users have different languages, cultures, and cognitions, the geographical location of the same user when using machine translation on a mobile device or wearable device is also changing all the time. In the object information translation method provided by the present invention, the location information where the translation occurs and the user's personal information are taken into account during translation, so that the translation results and derived translations are closer to the actual needs of the user.

[0288] The object-to-object translation method provided by the present invention can be applied to terminal devices. The "terminal" and "terminal device" used here can be portable, transportable, installed in a vehicle (aviation, shipping and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, running at any other location on the earth and / or space. The "terminal" and "terminal device" used here can also be a communication terminal, such as a PDA (Personal Digital Assistant, handheld computer), a MID (Mobile Internet Device, mobile Internet device) and / or a smart phone, and can also be a wearable intelligent terminal, such as a smart watch, smart glasses.

[0289] In the object information translation method provided in the first embodiment of the present invention, the object of translation is the object itself, rather than just the translation of text, which expands the scope of machine translation objects, enhances the applicability of translation tasks, and meets users' translation needs for objects that lack text descriptions.

[0290] Furthermore, the solution of the present invention proposes the concept of derivative translation, which not only provides the translation result, but also analyzes the translation content, provides derivative information, supplements the translation content, and enhances the user's understanding of the translation content.

[0291] Embodiment 2

[0292] In the second embodiment of the present invention, the object information translation method provided by the first embodiment of the present invention will be described from the perspective of a single object to be translated (or referred to as single object translation) and the perspective of multiple source objects to be translated (multi-object translation). For ease of description, "object information translation" may be referred to as "object translation" herein.

[0293] The following is an example of a Chinese user using the object information translation scheme proposed by the present invention on a terminal device to translate unfamiliar foreign goods. The terminal here can be a mobile terminal with a camera function and a touch screen, and the relevant information of the object translation is displayed on the mobile terminal. The display method here can be but is not limited to displaying the input information and the output result on one screen at the same time.

[0294] by Figure 2 As shown in the figure, the interface of the terminal device is divided into two parts, the upper part is the source object capture area, and the lower part is the target object display area. The source object capture area will provide an entry for object translation and display the text translation in the image, while the target object display area mainly provides system output for object translation and can also collect user behavior logs on translation results as feedback.

[0295] 1. Single Object Translation

[0296] The single object translation process provided by the second embodiment of the present invention can be divided into the following two steps:

[0297] S1: The terminal device obtains the source object information

[0298] In the second embodiment of the present invention, a terminal device is used to obtain the image information of the source object for specific description. When the terminal device detects that the user turns on the object translation function of the present invention, the device camera is automatically or based on the user's operation to obtain the image and other information captured by the current viewfinder; further, a thumbnail of the image can be displayed at the same time; further, based on the detected user's confirmation operation, the image and other information of the source object is determined.

[0299] Alternatively, the image, text and other information captured by the current viewfinder may be transmitted to the server; further, the information transmitted to the server may be information processed by the terminal device (e.g., screening and / or text recognition, etc.); further, based on the detected user confirmation operation, the image, text and other information of the source object may be determined.

[0300] For example, when the terminal device detects that the user has turned on object translation, the device camera will be automatically started, and a viewfinder will appear in the center of the source object capture area. If the device detects that the user has clicked the "OK" button, the image captured by the current viewfinder will be transmitted to the server, and a thumbnail of the image will be displayed at the same time. The display location can be the side or bottom edge of the image capture area of ​​the source end. If it is detected that the user has clicked the "Clear" button, all current thumbnails and their corresponding images uploaded to the server will be cleared. If it is detected that the user has clicked the "OK" button multiple times, the images will be uploaded to the server in sequence, and the thumbnails will be arranged in sequence.

[0301] Captured images can be viewed or deleted through thumbnails. When a user selects a thumbnail, the image corresponding to the thumbnail will be transmitted from the server to the terminal and displayed in the viewfinder. The camera will be temporarily turned off. If the user presses the image for a long time, the user will be given the option to delete it. If the user chooses to delete, the delete operation will be activated, the thumbnail will be deleted on the terminal, and the corresponding image will be deleted on the server.

[0302] When the camera is temporarily turned off, the terminal provides a way to turn on the camera, which can be by placing a return button in a suitable area of ​​the screen, by quickly clicking a certain area of ​​the screen multiple times, or by other interruption mechanisms.

[0303] S2: Display the translated target object information

[0304] The main task of this step is to display the translation results on the terminal. The translation results include but are not limited to at least one of the following: the image, name, location information, price information, category information, related audio / video, etc. of the target object, and the target object can be one or more; further, it can also include the text translation in the capture area of ​​the source object.

[0305] The display method includes: outputting the relevant content of the target end object listed above in the target end object display area, and may also include multiple target end object candidates, and the above multiple target end object candidates can be arranged based on the relevance to the source end object, the user click frequency, the difficulty of purchase, etc., so as to arrange the target end object candidates in order of priority. Here, the method for obtaining the user click frequency is to collect the user's operation log for the target end object, extract the number of clicks on the relevant object from the log data, and thus calculate the click frequency of each target end object. The method for obtaining the difficulty of purchase can comprehensively consider the distance between the current terminal location and the purchase location, or the purchase price.

[0306] In the object capture area of ​​the source end, if the current state is in the thumbnail viewing state, that is, the camera is turned off, the text translation obtained by the following steps based on analysis, object recognition and translation is displayed in the image in the viewfinder. If the current state is in the camera on state, the translation obtained by the following steps based on analysis, object recognition and translation is directly displayed in the viewfinder using AR (Augmented Reality).

[0307] The above translation results are achieved through the following steps based on analysis, object recognition and translation.

[0308] Analyze the objects in the image, including: extract image features, recognize text, etc. to identify the objects in the image; then translate the text in the image from the source language to the target language, and translate the source object in the image into the target object.

[0309] Through the above S1, the terminal device obtains the image of the object, or the terminal device transmits the image of the object to the translation system in the server, and first undergoes object recognition. Referring to the features required by the object recognition model, the image features are extracted from the image, and OCR (Optical Character Recognition) is performed to extract the text features, and then the object is recognized according to the object recognition model to obtain the source object to be translated and the text in the image; then the source object is translated into a candidate target object according to the score through the object alignment model, and the source text in the image is translated into the target language through the text translation model and the language model.

[0310] The method for determining the language of the source end includes at least: the GPS module of the terminal device automatically detects the country and language of the source end object, or it can be determined by automatically detecting the language of the text in the image, or it can be determined by searching the source of the source end object in the object knowledge graph database after identifying the source end object, or the source end language can be set by the user. The method for determining the language of the target end object includes at least: the GPS module of the terminal device automatically detects the country and language of the source end object, or it can be the target end language determined based on the user's personal information, for example, the language used by the user using the terminal device is determined according to the nationality of the user using the terminal device, thereby obtaining the target end language, or it can be the language set by the user. For example, if the behavior of the user setting the target end language on the terminal device is detected, the target end language set by the user is used as the language of the target end object. If the behavior of the user setting the target end language is not detected, the pre-specified language (such as the default device system language type) can be used as the language of the target end object. Figure 2As shown, the languages of the source end and the target end will be displayed in the upper right corner of the display screen, and it will be detected whether the user has set the languages. If the terminal does not detect the user's setting, the source end language will be displayed on the left and the target end language will be displayed on the right according to the above method. If the terminal detects that the user has set the target end language to Chinese, the language of the target end object should be configured as Chinese accordingly.

[0311] In the second embodiment of the present invention, the construction processes of the object recognition model, the object alignment model, and the object knowledge graph database are as follows: First, collect multi-modal data of a large number of source end objects and target end objects, including pictures and texts, extract text features and image features, and construct an object category discrimination model and an object recognition model in each category through a supervised learning method. Then, according to the existing translation model (object alignment model) between the source end language and the target end language, calculate the text description similarity scores of the source end and target end objects. Finally, extract text features and image features, and train to obtain the object alignment model through a supervised learning method. Extract the entity and relationship attributes of the object from the collected large-scale multi-modal data related to the object, and represent them with object triples <entity, entity, relationship>. Then, establish an object search index according to the preset object categories to form object knowledge graph data. Here, the "relationship" has corresponding specific meanings for different objects. For example, for the knowledge that "the color of object A is blue", a triple "<A, blue, color>" can be established. The present invention will introduce in detail the "relationship" attributes of different objects in derivative translation. In the derivative translation solution, the "relationship" is represented as the "key information" of the object, that is, the attribute information of the preset attribute. For example, the key information of drugs includes efficacy, side effects, dosage, contraindicated population, acquisition method, etc.; the key information of food includes ingredients, allergens, calories, applicable population, taste, etc. Among them, the key information of packaged food also includes origin, production date, shelf life, etc.; the key information of unpackaged dishes (meals) or recipes also includes taste, smell, temperature, color, eating method, cooking method, etc.; the key information of cosmetics is ingredients, origin, function, applicable population, shelf life, usage method, etc.

[0312] II. Multi-object Translation

[0313] In the multi-object translation scheme provided in the second embodiment of the present invention, the terminal device uploads an image of a source object, or the camera of the terminal device captures images of multiple source objects, and identifies multiple source objects through the object image and / or the text on the object image; then the recognition results are automatically classified. If they belong to the same category of combined objects in the predefined object categories, all source objects can be translated as combined objects first, and then each single object can be translated; if they do not belong to the same category in the predefined object categories, each object can be directly translated; finally, the terminal displays the target objects corresponding to each source object sorted by translation score. If the terminal device captures the user's behavior of selecting an object in the viewfinder, the terminal adjusts the translation result in real time to display the target object corresponding to the source object separately, and sorts it by translation score.

[0314] Specifically, the multi-object translation process provided by the second embodiment of the present invention can be divided into the following two steps:

[0315] S3: Get source object information

[0316] The terminal device captures images of multiple source objects through a camera, and the operation method at the terminal is the same as step S1 in the second embodiment. Figure 3 As shown, the terminal device captures images of multiple remote controllers.

[0317] S4: Display the translated target object information

[0318] If it is determined that the source object belongs to multiple identical objects, it means that multiple identical single objects are input, and only this single object is translated and output according to step S103 of embodiment 1 or displayed according to step S2 of embodiment 2 of the present invention.

[0319] If it is determined that the source objects belong to the same type of combined objects, that is, they belong to the same type of combined objects in the pre-defined object categories, all the source objects are first translated as combined objects, and then each individual object is translated. When displaying the translation results, the translation results of the combined objects are displayed first, and then the translation results of each object are displayed separately.

[0320] If it is determined that the source objects have no obvious combination relationship, each object is directly translated and then the translation result of each object is displayed.

[0321] If it is the latter two cases, embodiment three of the present invention provides an interactive method for multi-object input. When the terminal device captures the user's operation of selecting a source object in the viewfinder, the translation result is adjusted in real time at the terminal. The adjustment method includes but is not limited to: separately displaying the target object corresponding to the source object sorted by the translation score, or the target object sequence of the source object is arranged to the front, highlighted, framed, etc.

[0322] If it is the latter two cases, the third embodiment of the present invention further provides an interactive method of multi-object input with the help of augmented reality display technology, which can directly display the object that the user wants to find in the source object capture area.

[0323] In the second embodiment of the present invention, the method for determining whether multiple objects at the source end belong to the same category includes: searching for the name of each object in the object knowledge graph database, 1) if their names are exactly the same, they are processed as single object input; 2) if their names have the same brand or category prefix, they are processed as the same category, such as "Friso milk powder stage 1" and "Friso milk powder stage 2", 3) analyzing the image features and text features of the input objects, identifying them respectively and finding their corresponding positions from the knowledge graph, if they all belong to the sub-nodes under the same brand, such as cream and lotion under a certain brand name, they are processed as a group of objects in the same category. Except for these cases, they are processed according to different categories.

[0324] In the technical solution of the second embodiment of the present invention, unfamiliar foreign objects, including medicines, foods, cosmetics, road signs, etc., are translated into domestic corresponding objects familiar to users through a pre-constructed object alignment model, or the translation direction is exchanged to translate domestic objects into corresponding objects in the target country, thereby expanding the scope of machine translation objects, enhancing the applicability of translation tasks, and meeting users' translation needs for objects that lack text descriptions.

[0325] Embodiment 3

[0326] In order to present the translation results to the user more intuitively, and to provide instant feedback and quick response to the user's needs, the third embodiment of the present invention provides an interactive object information translation solution with the aid of augmented reality.

[0327] It is worth pointing out that the augmented reality interactive translation method introduced here is not only applicable to object-to-object machine translation, but can also constitute an improvement on other conventional translation methods, for example, a method of direct translation based on text recognized in the image information of the source object.

[0328] The augmented reality interactive object translation is divided into the following three steps:

[0329] Step 1: Get source object information.

[0330] Step 2: Display the translated target object information.

[0331] Step 3: The terminal responds accordingly based on the detected user operation.

[0332] In the third embodiment of the present invention, the implementation of step one and step two may refer to steps S1 and S2 of the second embodiment and will not be described in detail herein.

[0333] When the source object information includes multimedia information corresponding to the source object, the source object information can be obtained in the following manner: the multimedia information is collected in real time by a multimedia collection device, and the real-time collected multimedia information is used as the source object information.

[0334] The user operations detected in step 3 include but are not limited to the following: click, long press or frame selection of information in the relevant image in the source capture area (such as recognized text); click, long press or frame selection of a target object in the target display area; click the button to start the interactive mode, etc. The response corresponding to the user operation includes but is not limited to: adaptive adjustment based on the detected source object information, such as identifying the relevant source object, highlighting the text recognized in the relevant image, and adjusting the display mode of the source object based on the operation on the target object, or adjusting the display mode of the target object based on the operation on the source object, such as displaying with the help of augmented reality technology, etc.

[0335] Furthermore, the step three may be but is not limited to at least one of the following three methods: direct translation of the source object, positioning of the target object, and positioning of the source object.

[0336] Method 1: Direct translation of source objects

[0337] According to the detected operation on the source object capture area, the display of the source and target ends is adjusted. Specifically, the terminal first detects the operation on the source capture area. If it is detected that the user has clicked, zoomed in, long pressed, etc. on a certain area on the object, the text recognition result and text translation result of the area are highlighted in the source viewfinder. The display methods include: zooming in on the area, highlighting and / or selecting the range of the area, etc. Figure 2 As shown, when the terminal detects that the user has clicked, long pressed, zoomed in, etc. on the "Ingredients" area in the source end viewfinder, the "Ingredients" area presented in the source end viewfinder will be enlarged, and all text translations in the area will be displayed.

[0338] Method 2: Positioning of the target object

[0339] According to the detected user operation on the target object display area, the display of the source and / or target end is adjusted. Specifically, the terminal first detects the target object display area, and if it is detected that the user has activated the interactive display mode of a target object, the interactive display mode of the target augmented reality is activated. Figure 4 Take the example of the following illustration. In this illustration, a Chinese user wants to buy a certain milk powder sold in the Netherlands, but the user only has photos of the milk powder sold in China. When he arrives at a Dutch shopping mall, he can easily and quickly find the local milk powder product through the above-mentioned augmented reality interactive object information translation method.

[0340] Specifically, for example, the above steps 1 and 2 are first processed, that is: the terminal device detects that the user has uploaded an image of the source object, the milk powder, and then analyzes, identifies and translates the image on the server side to obtain the image and related text description of the corresponding object of the milk powder in the Netherlands (i.e., the target object).

[0341] Furthermore, in step 3, the present invention provides the user with a "browse mode" option in the display area of ​​the translation result. The "browse mode" can be activated in one of the following ways: providing a browse button next to the corresponding object image on the screen, or activating it by long pressing or double-clicking the object image, or by a preset voice command.

[0342] When the terminal device detects that the user has activated the "browsing mode", the terminal device's camera is turned on and the screen ratio of the viewfinder on the screen is adjusted to an appropriate ratio. The terminal uploads the image captured by the camera in real time, and the server performs multi-target recognition on the image and then matches it with the target object. If an object in the image can match the target object, the object is highlighted, framed and / or otherwise identified in the viewfinder using augmented reality.

[0343] In the third embodiment of the present invention, multiple objects (multiple different or identical objects, see the second embodiment for the specific judgment method) in the image uploaded by the terminal in real time can be called candidate objects, and the translated target object is called the reference object. The above-mentioned matching method specifically includes: extracting the image features and / or text features of the target object from the object knowledge graph database; extracting the image features of the candidate object from the object knowledge graph database, and performing text recognition on the image of the candidate object to extract the text features; then measuring the similarity between the image features and / or text features of the candidate object and the target object, and the similarity calculation method can be a cosine similarity method, a kernel method, or a calculation method based on distributed representation.

[0344] Method 3: Positioning of source objects

[0345] When multiple source objects are input, the user's selection operation on the translated target object information can be detected; and the source object corresponding to the target object information selected by the user is located in the multimedia information collected in real time.

[0346] Specifically, the display mode of the object capture area at the source end can be adjusted according to the operation of the user on the object display area at the target end detected by the terminal, such as directly displaying the object that the user wants to find in the object capture area at the source end. Figure 3 Take this as an example to illustrate.

[0347] The terminal device captures images of multiple source-side objects through a camera, identifies, classifies, and translates multiple objects according to the multi-object translation scheme provided in Implementation 2, and displays the translation results in the target-side object display area. The terminal detects that the user has activated the multi-object interaction mode, and the activation methods here include but are not limited to: gesture operations such as long pressing or double-clicking a target-side object image, voice input of a target-side object name, etc. The terminal activates the multi-object interaction mode and detects a selected target-side object, and highlights, frames, and / or otherwise identifies the source-side object corresponding to the selected target-side object in the viewfinder of the source-side object capture area.

[0348] Specifically, Figure 3 As shown, the terminal detects that the user has activated the multi-object interaction mode and detects a selected target object, and highlights or selects the source object corresponding to the selected target object in the viewfinder of the source object capture area.

[0349] After judging that the language of these source remote controllers is Korean, the categories include TV remote controller, air conditioner remote controller, microwave remote controller, etc., which do not belong to the same category, each of them is translated into an object in the target language. For the judgment method of whether they belong to the same category, see Example 2.

[0350] Embodiment 4

[0351] Considering that traditional machine translation technology usually only focuses on translation itself, in real life, users often need not only translation, but also understanding of the translation content and acquisition of relevant knowledge. For example, when entering the name of a foreign drug, traditional machine translation will only translate accordingly based on the input, without further understanding whether the translated content corresponds to a named entity, and if so, whether it is necessary to provide relevant useful information such as efficacy, dosage, applicable population, and even purchase method. This passive translation method of translating what the user enters cannot truly meet the actual needs of users.

[0352] Therefore, in order to allow users to better understand the translation content, the fourth embodiment of the present invention provides an object information translation solution that provides derivative translation. Specifically, the derived information associated with the target end object and / or the derived information associated with the source end object can be obtained; and the obtained derived information can be output.

[0353] In practical applications, the attribute information corresponding to the preset attributes of the target object can be searched from the preset object knowledge graph database; the searched attribute information is confirmed as the derived information associated with the target object. The attribute information corresponding to the preset attributes of the source object can be searched from the preset object knowledge graph database; the searched attribute information is confirmed as the derived information associated with the source object. The preset attributes are determined according to the categories of the target object and / or the source object.

[0354] In the fourth embodiment of the present invention, the preset attribute of the object may be a "relationship" attribute, and the attribute information corresponding to the preset attribute is represented as the "key information" of the object. For example, the key information of a drug includes efficacy, side effects, dosage, people who should not use it, how to obtain it, etc.; the key information of food includes ingredients, allergens, calories, applicable people, taste, etc., among which the key information of packaged food also includes origin, production date, shelf life, etc., and the key information of unpackaged dishes (meals) or recipes also includes taste, smell, temperature, color, eating method, cooking method, etc.; the key information of cosmetics includes ingredients, origin, function, applicable people, shelf life, usage, etc.

[0355] In the fourth embodiment of the present invention, the scheme of object information translation providing derivative translation is explained by taking key information related to the translation object and comparison information between the source object and the target object as derivative information.

[0356] The key information and comparative information of single objects and multiple objects are introduced below.

[0357] 1. Derivative Translation of Single Objects

[0358] The process of generating derived translations for a single input object can be divided into the following steps:

[0359] Step 1: Get source object information

[0360] The step of obtaining source object information in Embodiment 4 of the present invention may refer to the step of obtaining source object information in Embodiment 2. The terminal captures a source object image through a camera, or the terminal transmits an existing source object image (for example, obtained from a network or other device) to a translation system of the local machine or server.

[0361] For example, Figure 5As shown in the figure, the "Upload" button is provided in the lower right corner of the source object capture area. An American user who is traveling in China wants to buy cold medicine. The terminal detects that he has input a picture of the cold medicine he brought with him and sets the target language to Chinese. The terminal transmits the picture of the cold medicine on the source to the server and tells the server that the target language is Chinese, that is, translating the American cold medicine into the Chinese cold medicine.

[0362] Step 2: Display the translated target object information

[0363] In the fourth embodiment, after translating and obtaining the target-end object information corresponding to the source-end object, the derived information associated with the target-end object and / or the derived information associated with the source-end object may be obtained; and the obtained derived information may be output.

[0364] The derivative information of the target object can be displayed in the target object display area of ​​the screen, and the display method includes but is not limited to at least one of the following: displaying one by one in the form of a text list, or representing a certain derivative information in the form of a picture, or linking other applications of the device in the form of a hyperlink, such as map navigation, or broadcasting and displaying the derivative information in the form of audio / video.

[0365] Further, the steps of analyzing, identifying objects and generating derived information are as follows:

[0366] First, after analyzing and identifying the object according to the method mentioned in Example 2, it is translated into a target object, and relevant key information is obtained from the knowledge graph of the target object.

[0367] The knowledge graph construction method here is: based on the object knowledge graph database constructed in Example 2, the key information of each object is extracted and marked. The specific method is to label the key information of the object with derivative information according to the category of each object. The key information described here is related to the category of the object, such as: the key information of the drug includes efficacy, side effects, dosage, contraindications, acquisition methods, etc.; the key information of food includes ingredients, allergens, calories, applicable population, taste, etc., among which the key information of packaged food also includes origin, production date, shelf life, etc., and the key information of unpackaged dishes (meals) or recipes also includes taste, smell, temperature, color, eating method, cooking method, etc.; the key information of cosmetics is ingredients, origin, function, applicable population, shelf life, usage, etc.

[0368] In practical applications, derived information associated with the target object may also be determined based on the position information corresponding to the target object.

[0369] Secondly, generate derivative translations based on the target language. After obtaining the derivative information of the target object, the translation system starts the text translation module. If the language of the derivative information is the same as the terminal device system language, the derivative information is output to the terminal device. If the language of the derivative information is different from the terminal device system language (or the target output language set by the user), it is translated into the terminal device system language (or the target output language set by the user) and then output to the terminal device. Figure 5 As shown, the derivative information of Chinese cold medicine here is in Chinese, which is inconsistent with the device system language of American users and needs to be translated into the user's system language.

[0370] 2. Automatic comparison of source and target objects

[0371] In order to help users better understand the difference between the source object and the target object after translation, the fourth embodiment of the present invention further provides a method for comparing the objects at both ends, which is implemented by the following steps:

[0372] Step 1: Start comparison mode

[0373] After the translation system translates the source object into the target object and displays it on the terminal device according to steps S1 and S2 in Example 2, a mechanism for activating the comparison mode is provided for the terminal at the same time, including but not limited to one of the following: a comparison button can be set near the target object in the target object display area, or it can be activated through certain gesture operations, or the comparison mode can be activated through a pre-set voice command, or it can be automatically triggered by the system.

[0374] If the terminal detects that the user has activated the comparison mode between the source object and the target object, a comparison command between the current source object and the target object is sent to the server.

[0375] Step 2: Display comparison information on the terminal device

[0376] The display mode of the terminal device includes but is not limited to one of the following: single screen display on the terminal screen, which can be horizontal screen left and right display such as Figure 6 As shown, it can also be displayed vertically or horizontally; it can also be displayed in two split screens on the terminal, with one screen displaying the source end key information and the other screen displaying the key information of all candidate target end objects, and switching the candidate target objects by scrolling up and down, sliding left and right, etc.; it can also display the key information of the source end object and the target end object by voice broadcast. Furthermore, the difference information can be highlighted or framed, for example, Figure 6 The different dosages and usage of the medicines on the middle, left and right sides can be highlighted.

[0377] Furthermore, the method for obtaining the comparison information is as follows:

[0378] The key information of the target object is obtained through step 2 of the derivative translation scheme of a single object. The method for obtaining the key information of the source object is similar to that of the target object. Specifically, the source object is searched from the object knowledge graph database, and the corresponding information of the source object is extracted by comparing the key information of the target object. For example, if the target object is a drug, its key information is efficacy, side effects, dosage, contraindications, and acquisition methods. Then, the efficacy, side effects, dosage, contraindications, and acquisition methods of the source object are searched in the knowledge graph.

[0379] 3. Derivative Translation of Multiple Objects

[0380] When multiple objects are detected as input, the generation and processing of derivative information is as follows:

[0381] Step 1: The terminal device captures multiple source object images

[0382] This step is completely identical to step S3 of the second, multi-object translation step in the second embodiment, and will not be described again here.

[0383] Step 2: Terminal displays derivative translation

[0384] This step adds the display of the derived translation of the terminal device on the basis of the second embodiment, and the adding method is as described in step 2 of the automatic comparison scheme of the source and target objects.

[0385] Here, the method for analyzing and identifying multiple objects, generating multi-object derivative information, and translating the derivative information is as follows:

[0386] First, the server performs multi-target recognition on the image, and the recognition method is as described in steps S2 and S4 in Example 2. After the recognition is completed, the translation system will judge the recognized objects: if they are exactly the same, it means that multiple exactly the same single objects are input, and the translation system only generates derivative information for the target end object of this single object; if they belong to the same category in the pre-defined object category, all source end objects are first regarded as combined objects, and derivative information is generated for the translated target end objects, and then derivative information is generated for the target end objects of each single object; if they do not belong to the same category in the pre-defined object category, derivative information is directly generated for the target end object of each object.

[0387] Look up the name of each object in the object knowledge graph database. 1) If their names are exactly the same, they are processed as single object inputs. 2) If their names have the same brand or category prefix, they are processed as the same category (e.g., combined objects). Figure 73) Analyze the image features and text features of the input objects, identify them respectively and find their corresponding positions from the knowledge graph. If they all belong to the sub-nodes under the same brand, such as cream and lotion under a certain brand, they are combined and processed as the same category. Except for these cases, they are processed as different categories.

[0388] Secondly, a derivative translation is generated according to the target language. The specific method is the same as the translation method described in step 2 of the automatic comparison scheme of source and target objects, and will not be repeated here. Figure 7 As shown, the derivative information of cosmetics includes but is not limited to: manufacturer (or place of origin), efficacy (or function), steps (or method of use), recommended purchase location, etc.

[0389] 4. Automatic comparison of multiple source and target objects

[0390] If the input is multiple objects, the comparison process between the source objects and the translation result is carried out as follows:

[0391] Step 1: Start comparison mode

[0392] If the input multiple objects are combined objects, the system will provide a combined object comparison mode, which is activated in the same way as single object input. Specifically, a comparison button can be set near the combined target object in the target end object display area, or it can be activated through certain gestures, or through a pre-set voice command to activate the comparison mode, or it can be automatically triggered by the system.

[0393] If the multiple objects input do not belong to a combined object, a single object comparison mode is provided for each source object and its translated object. The activation method can refer to the steps in the above "II. Automatic comparison of source and target objects" solution, which will not be repeated here.

[0394] Step 2: Display comparison information on the terminal device

[0395] The display method at the terminal is similar to step 2 of the automatic comparison scheme of the source and target objects, and specifically includes but is not limited to one of the following: single-screen display on the terminal screen, which can be horizontal left and right display such as Figure 8 As shown, it can also be displayed vertically or horizontally; it can also be displayed in two split screens on the terminal, with one screen displaying the source end key information and the other screen displaying the key information of all candidate target end objects, and switching the candidate target objects by scrolling up and down, sliding left and right, etc.; it can also display the key information of the source end object and the target end object by voice broadcast. Furthermore, the difference information can be highlighted or framed, for example, Figure 8Different quantities, usages, etc. of cosmetics on the middle, left, and right sides can be highlighted.

[0396] If the comparison mode is for combined objects, the objects included in the source and the objects included in the target need to be displayed separately on the comparison interface, and then the derivative information of the combined objects is compared. If the comparison mode is for multiple single objects, the key derivative information of each source and target object needs to be compared separately on the comparison interface.

[0397] The method and content of the comparison information here are: if the terminal sends a comparison mode to start the combined object, the combined source object is searched from the knowledge graph, and the corresponding information of the combined source object is extracted by comparing the key information of the combined target object. List the objects contained in the combined source object and the combined target object. The specific operation method includes: first pair the combined objects of the source and target ends, and the pairing method can be to calculate the similarity of the names, functions, ingredients and other features between the two objects. If the combined objects of the source and target ends contain unpaired objects, they are highlighted, and the highlighted display methods include but are not limited to one of the following: bold, italic, highlight, etc. If the terminal sends a comparison mode to start multiple single objects, according to the results of the above pairing, if there is a successfully paired object, refer to step 2 in the automatic comparison scheme of the source and target objects to compare and extract key information.

[0398] Specifically: search for source objects in the knowledge graph database, and extract corresponding information of the source objects by comparing the key information of the target objects, such as Figure 8 As shown, the target object is a cosmetics set, and its key information is the origin, function, applicable population, usage, usage steps, and acquisition store. Then, the origin, function, applicable population, usage, usage steps, and acquisition store of the source object are searched in the knowledge graph. If the language of the source object in the knowledge graph is different from the terminal device system language (or the target output language set by the user), it is translated into the terminal device system language. Then, the key information of the source object and the target object will be transmitted to the terminal device in the terminal device system language for display. Figure 8 In the example, the similarity between the source and target objects is calculated and paired according to the method. If the combined objects of the source and target contain mismatched objects, such as Figure 8 In the example of "eye cream" and "makeup remover oil", when displaying the types of items contained in the two sets, the mismatched objects are displayed in a highlighted or italicized manner. Furthermore, the usage steps of the source and target sets can be displayed. If the terminal sends a command for the single object comparison mode, the similarity between the source and target object pairs is first calculated and paired according to the method, and then the paired object pairs are compared and displayed.

[0399] Since passive translation is far from meeting the needs of users, the fourth embodiment of the present invention proposes the concept of derivative translation, which not only provides the translation result, but also analyzes the translation content and provides derivative information about the translation content, which can supplement the translation content and enhance the user's understanding of the translation content.

[0400] Embodiment 5

[0401] 1. Personalized object information translation

[0402] In order to make the translation result more in line with the personalized needs of users, the fifth embodiment of the present invention provides an object information translation solution based on user personal information.

[0403] For different users, the object information translation method provided in the fifth embodiment of the present invention will supplement personalized derivative information, which specifically includes the following two steps.

[0404] Step 1: Get source object information

[0405] Furthermore, the acquired information may also include user personal information, and the acquisition methods include but are not limited to the following methods: the terminal acquires the user's schedule information by analyzing the information recorded in the user's calendar and / or memo software; the terminal acquires information related to the user's interests and hobbies by analyzing the information recorded in the user's mailbox, text messages, and / or call records; the terminal acquires information about the user's environment through a camera, microphone, sound sensor, light sensor, and / or humidity sensor; the terminal acquires information such as the user's motion status through a speed sensor, etc.; the terminal acquires information such as the user's health status through application software, etc.

[0406] Step 2: Display the translated target object information

[0407] Furthermore, before displaying the translated target object information, the derivative information is adjusted according to the user's personal information. After the terminal collects the user's personal information, it is combined with the current derivative translation to rate and annotate the content of the derivative translation. The specific rating and annotation methods are as follows:

[0408] The relevance analysis is performed on the derived translation content and the user information, the derived translation content is scored according to the relevance of the user information, and the derived translation content with high relevance is highlighted. The relevance analysis can be achieved by calculating information entropy, KL (Kullback Leibler divergence, relative entropy) divergence and other methods.

[0409] like Fig. 9As shown in the example, if the terminal detects that the user's schedule contains dieting and weight loss content according to the method of step 1, and records peanut allergy and other content in the health application, the system will highlight the allergens, calories and other information in the prominent position of the derivative information of the object translation for this user, so as to remind the user to pay attention. Fig. 9 As shown, in the derivative information area on the terminal screen, such sensitive content will be brought forward and highlighted. The display method includes but is not limited to: changing the display order of the content on the terminal screen, highlighting, bolding, and reminding the user through terminal vibration, sound, etc.

[0410] 2. Optimizing object information translation based on position

[0411] Considering that the existing machine translation technology does not fully consider the location information and personalized information of mobile device users, the translation is not adjusted according to the translation occasion or user needs. Users in different regions may have very different understandings of a certain concept. For example, "sunscreen" in English can refer to sunscreen or sunscreen. For the Chinese market, it usually refers to sunscreen, so it can usually be translated into sunscreen in China. However, for Chinese people traveling in Europe and the United States, if it is still translated into sunscreen without distinction, it is likely to cause misunderstanding among users, resulting in wrong purchasing behavior.

[0412] Therefore, in the fifth embodiment of the present invention, the user's location can be collected through the device sensor, and the object alignment model used for object translation can be optimized according to the different locations of the user to improve the accuracy of object information translation. The specific process includes:

[0413] Step 1: Get source object information

[0414] Furthermore, the acquired information also includes acquiring information such as the user's nationality and ethnicity. The terminal determines the user's nationality and ethnicity by analyzing the user's frequently located location information / the language used by the user.

[0415] Step 2: Display the translated target object information

[0416] When the system translates object information, it will optimize the object translation results by combining information such as user location, nationality, and ethnicity. First, we define the object ambiguous triple <source object, source language, target language>: If the target object candidate corresponding to the source object translation does not completely belong to the same object, it means that the source object is ambiguous in the target language. For example, if "sunscreen" is translated into "sunscreen" or "sun cream", it constitutes the object ambiguous triple <object represented by "sunscreen", English, Chinese>.

[0417] After the terminal determines the translation direction of the source and target ends through step S2 of the second embodiment, it determines whether the current source object is ambiguous according to the object ambiguity library. If so, the translation model (i.e., object alignment model) is optimized according to the ambiguous information about the source object in the knowledge graph database. The optimization method is to extract the text features of the source object and use the object ambiguity classifier to score the translation candidates. The text features here include: the derived information of the source object, such as Fig.10 The derived information contained in the object in the text is "Origin: Australia, Sun Protection Factor: 4, Efficacy: Helps to tan" and so on. The construction method of the object ambiguity analysis classifier here is specifically as follows: extract the above text features, use a supervised discriminant model training method to classify and train the translation candidates, and obtain the scores of the same ambiguous object corresponding to different translation candidates under different text features, and add them as features to the translation model, so as to help the translation system optimize the final translation candidates. Fig.10 As shown, according to the user's nationality, the system will automatically determine whether to use the ambiguous classifier. If the user is European, the translation result will be given directly. If the user is Asian, the translation system will add the ambiguous classifier score as a feature, disambiguate the translation result and output the translation result "sun cream". Or remind the user in the derivative information that the translation of the object may be ambiguous. The reminder method includes but is not limited to: highlighting the ambiguous content, adding a warning icon, popping up a warning window on the terminal screen, or using voice reminders.

[0418] In the above method, the construction of the object ambiguity library and the automatic judgment method of ambiguous objects are as follows: on the basis of the object alignment model construction method described in step S2 of embodiment 2, the target object candidates corresponding to each source object are subjected to category analysis, and the analysis method is as described in step S4 of embodiment 2. If the target candidates do not completely belong to the same category, the triplet <source object, source language, target language> is added to the object ambiguity library. When querying whether an object being translated is an ambiguous object, the judgment is made by querying whether its triplet is in the object ambiguity library. If it is in the object ambiguity library, the return value is positive, otherwise it is negative.

[0419] The method for constructing a knowledge graph with ambiguous information reminders described in step two is as follows: based on the knowledge graph construction method in Example 2, the ambiguous object corresponding to the target object is searched through the object ambiguity library and alignment model, and an ambiguous mark is added to the knowledge graph.

[0420] 3. Translation Adaptation in Social Networks

[0421] Embodiment 5 of the present invention further provides a solution for translating object information in a social network, including the following steps:

[0422] Step 1: Get source object information

[0423] The present invention also provides an entry for starting a group chat function in an object translation system and / or starting object translation in a social software. The starting method includes: starting object translation by long pressing, sliding, etc., and then uploading the source object image information to the server or the local object translation system according to step S1 of the second embodiment; or setting a translation button in the social software interface, and starting object translation according to user operation, and the social software can be a group chat software / application, or a software / application with a conversation function.

[0424] Step 2: Output the corresponding translation results for each person

[0425] In actual applications, two situations can be described depending on whether one-side terminal translation is used:

[0426] 1) Based on the receiver’s information, the sender requests the system to translate the object and then send the image.

[0427] The sender collects the location information of the receiver through the terminal GPS / wifi / 3G and social network, thereby determining the region and nationality of the target object. According to the location of each terminal, a request is sent to the translation system to translate the source object into the corresponding target object and generate derivative translations. The translation results are then sent to each receiver. The translation results here can include text, pictures, audio, and video.

[0428] 2) Send a picture of the object, and the recipient requests the system to translate the object.

[0429] The receiver collects the user's current location through the terminal GPS / wifi / 3G, thereby determining the region and nationality of the target object. According to the location of each terminal, a request is sent to the translation system to translate the object transmitted from the source end into the corresponding target end object and generate a derivative translation.

[0430] like Fig.11 As shown, when a user in the mainland translates "Friso" in a group chat software (such as Fig.11 As shown in the second UI picture in the chat, Wenfang in Hong Kong and Sophie in the Netherlands will receive different translation results adapted to their local conditions, even though they are in the same chat interface. Wenfang in Hong Kong will display the corresponding object image and derivative information of the item in Hong Kong, while Sophie in the Netherlands will display the corresponding image and derivative information of the item in the Netherlands.

[0431] 4. Adaptive translation of slogan derivatives

[0432] When the input object is a logo, the present invention proposes a method of automatically generating a derivative translation of the local laws and regulations for the current logo according to the current location or region. The processing process can be divided into the following steps:

[0433] Step 1: Get source object information

[0434] This step is the same as step S1 in the second embodiment and will not be described again here.

[0435] Step 2: Display the translated target object information

[0436] Further, the method for analyzing, identifying and generating derivative information of slogans is as follows:

[0437] According to step S2 of the second embodiment, the object is identified. If the object is a logo, the logo is translated and derivative information related to local laws and regulations is generated.

[0438] The method for determining whether an object is a logo is as follows: establishing a multilingual commonly used logo database, and after the input object is recognized for text, matching it with the sentences in the logo database, and if they match, determining that the current input object is a logo, otherwise translating it as a general object.

[0439] The method of generating derivative information related to local laws and regulations is: obtaining the current location of the terminal through the terminal GPS / WIFI / 3G, etc., and querying the regulatory entries containing the translated content from the regulatory database. The method of constructing the regulatory database is: establishing a regulatory policy database of different regions for common logos, which can be established by automatically extracting relevant clauses from the text on the legal / regulatory website, or by manual annotation.

[0440] Depending on the cultural background of the region where the slogan is located, slogan translation involves the following three situations:

[0441] 1) The slogan or sign is present in language environment A, but not in language environment B;

[0442] If the signboard at the source end does not have a corresponding sign at the target language end, only the text in the signboard at the source end is translated, and a derivative translation as described above is given. For example, the "right of way" signboard in Figure 18(a) is available in Europe but not in China. When it is translated into Chinese, the local meaning and derivative information of the signboard are output at the terminal target end.

[0443] 2) A and B have different patterns but the same meaning;

[0444] According to the method described in step 2, the image features and text features are extracted for translation, and the terminal will display the corresponding signboard of the target end (not necessarily exactly the same) and the derived translation.

[0445] 3) A and B have the same pattern but different meanings;

[0446] In this case, the method described in 2) is used to output the corresponding signboard and derived information obtained by the joint action of the image features and text features of A. If there is no corresponding signboard, the meaning and derived information of the signboard are output according to 1).

[0447] like Fig.12 As shown, the terminal detects the input object, and the system identifies it as a slogan "No Phone Calls Allowed". The current location is detected through the terminal. Based on the location information and the text information of the slogan, the local penalty regulations for violations of this behavior are searched from the regulations database, "Violators will be fined RMB 1,000", and then translated into the terminal system language (or the target language set by the user) and displayed on the terminal screen.

[0448] Furthermore, if the slogan involves the operation of a specified APP, it is associated with the APP, such as Fig.12 In the dialog box, if "Prohibit" and "Call" involve the operation of the terminal voice communication APP, the terminal will automatically associate with the APP and pop up a warning message "Turn off call App".

[0449] 5. Personalized event reminders based on derivative translation

[0450] The fifth embodiment of the present invention also provides a personalized event reminder solution, which can set reminders for users and can further adjust automatically.

[0451] For example, when translating objects such as medicines, if the translation system detects that the derived translation contains information such as time intervals and numbers, it will set a personalized reminder event for the user; further, if it is detected that the user has set a reminder, the reminder can be adaptively changed based on the comparison information in the derived information. The reminder can be applied to smart terminals such as mobile phones, wearable devices or other smart devices.

[0452] The specific steps include:

[0453] Step 1: Get source object information

[0454] This step is based on step S1 of the second embodiment. Further, the detection information includes whether there is a reminder event set on the terminal device, and it is determined whether it is related to the current translation content according to the content of the reminder event.

[0455] The above method for detecting whether there is a translation-related reminder in the reminder of the current device is: judging according to the relevance between the translation content and the device reminder content. Specifically, for example, the similarity of text keywords is calculated according to the translation content and the device reminder content. The calculation method can be cosine similarity, edit distance, etc. For example, in Figure 13 (a), the event keyword "dose" is extracted from the derived translation. If the current device detects that the topics of the existing reminder events are "appointment", "taking medicine", and "birthday", and the semantic similarity calculation finds the most relevant "dose" event, then the "taking medicine" reminder event is judged to be a reminder event related to the current translation content. In addition to calculating similarity based on keywords, it is also possible to determine whether the reminder content is related to the current existing reminder content through text semantic distributed representation or calculating text topic similarity.

[0456] Step 2: Display the translated target object information

[0457] In this step, based on step S2 of the second embodiment, the system automatically analyzes the translation content. If it is detected that the derived translation contains a time interval, a reminder event is automatically generated or an existing reminder event is adaptively modified according to the user information detected in the terminal. The reminder content here includes the content automatically extracted according to the derived translation, including but not limited to: name, dosage, time interval or other key information related to the object.

[0458] The specific method is as follows: through step 1, if the terminal detects that there is no relevant reminder in the reminder of the current device, the terminal prompts the user whether to create a new reminder, and automatically generates different reminder object options according to the text analysis results, and displays possible reminder options on the terminal, as shown in Figure 13 (b). Through text analysis of the derivative information, two reminder objects are automatically obtained, and reasonable time reminder suggestions are given according to the time interval of the text and the average work and rest time of the user. As shown in Figure 13 (c), according to the user's work and rest time detected by the terminal and the time interval analyzed by the text, a relatively reasonable reminder time is automatically arranged. If the terminal detects that there is already a relevant reminder in the reminder of the current device, the terminal prompts the user whether to change the reminder time, and automatically arranges a relatively reasonable reminder time according to the user's work and rest time detected by the terminal and the time interval analyzed by the text.

[0459] Here, the terminal detects the user's work and rest time in ways including but not limited to: by detecting the morning activation time and night sleep time of the terminal device and calculating the average value; it can also detect the user's daily activity time through device sensors, such as the temperature sensor and gyroscope on a watch, and calculate the average value, etc.

[0460] 6. Multimodal output of translation results

[0461] Embodiment 5 of the present invention further provides a solution for multimodal output of translation results, wherein the multimodal output includes text description, pictures, audio, video, etc.; further, the output mode can be adaptively adjusted based on user operation, terminal environment, current network conditions, terminal device type, etc.

[0462] For example, in the normal mode, the translation outputs the text description and picture of the target object, and further includes voice. If the derivative translation of the target object contains video, as shown in Figure 14 (a), the video can be output. If it is detected that the user sets the child mode, as shown in Figure 14 (b), a small amount of text plus pictures and videos are used in the translation result; if it is detected that the current environment is quiet, such as the library, hospital and other scenes shown in Figure 14 (b), the translation result uses the text description plus pictures, and further, audio or video can be provided, but a warning is issued before the audio and video are played; if it is detected that the current network / data signal is poor, or the battery is low, as shown in Figure 14 (b), the translation result uses text description and compressed pictures.

[0463] Because different users vary greatly in language, culture, and cognition, the geographical location of the same user when using machine translation on a mobile device or wearable device is constantly changing. Embodiment 5 of the present invention considers the location information where the translation occurs and the user's personal information when translating an object, so that the translation result and the derived translation are closer to the actual needs of the user.

[0464] Embodiment 6

[0465] Considering that the existing machine translation technology usually focuses on how to apply it on mobile devices such as mobile phones and pads, it does not fully explore how machine translation can adapt to multiple users and multiple devices, such as watches and glasses in wearable devices, to bring greater convenience to people. With the increasing ability of wearable devices to process data and the gradual popularization of devices, how to transplant machine translation to wearable devices, how to adapt to the characteristics of multiple users, and how to mine the data generated by multiple users to improve translation quality have become very important issues.

[0466] Furthermore, considering the increasing popularity and portability of wearable smart devices, embodiment 6 of the present invention provides a solution for object translation on a variety of smart devices, and provides a solution for translating for multiple users and improving translation quality through big user data.

[0467] 1. Object Information Translation for Smart Watches

[0468] As a smart device that users always carry with them, watches are more convenient to carry than terminals such as mobile phones. Therefore, Embodiment 6 of the present invention provides an object translation solution that can be applied to smart watches, and further provides a translation solution that is independent of terminals such as mobile phones.

[0469] Step 1: Get source object information

[0470] The detection, uploading and synchronization of the source object are completed through the watch, which may also include other terminal devices (such as mobile phones / PADs) and servers for communication. The interaction methods of the watch, other terminal devices and servers include but are not limited to: 1) The smart watch is only used as an input and / or output device, and the camera of the smart watch detects or other terminal devices detect the source object, transmits it to the terminal device or uploads it to the server and translates it, and the translation result is transmitted back to the terminal device and / or watch. Furthermore, the terminal device that obtains the translation result adaptively adjusts the translation result according to the size of the watch dial, and then transmits the translation result to the watch for display, such as Figure 15(a) , 15(b) As shown, the display method on the watch is introduced by step 2; 2) The smart watch is a smart terminal device. After the watch detects the source object, it is transmitted to the object translation system in the watch or the source object picture is uploaded to the server for translation. Furthermore, the server can adaptively adjust the translation result and return it to the watch.

[0471] Step 2: Display the translated target object information

[0472] It may further include displaying partial derivative translations.

[0473] When the translation results are returned to the watch, they will be adaptively adjusted, including but not limited to: adaptive object image size, automatic summary of descriptive text, minimizing text display, and no video provided for derivative translations; or adjustments will be made based on user operations, such as detecting whether the user has selected the full text or summary, detecting whether the user has activated the audio playback setting, detecting whether the user has selected the video playback setting, etc. Furthermore, if the derived translated text contains keywords such as time intervals, the text will be analyzed and a reminder event will be automatically generated. The reminder time will be arranged according to the user's schedule: the watch will automatically detect the user's average wake-up time and rest time, and then establish a reminder event based on the time interval in the text. Fig.16 As shown, the system detects the time interval in the derivative translation, and through text analysis finds that there are several possible time intervals, so it automatically generates different options to display on the dial. If an adult selected by the user is detected, the reminder time is automatically arranged according to the user's average schedule and the time interval of the adult. This method is the same as step 2 in the personalized event reminder solution based on derivative translation, and will not be repeated here.

[0474] 2. Object Information Translation for Smart Glasses

[0475] For the translation system, smart glasses, as wearable devices with the highest synchronization rate with human eye movements, have the great advantage of real-time capture of source objects and completely hands-free operation compared to other terminals. Therefore, embodiment 6 of the present invention provides an object translation solution applied to smart glasses.

[0476] The object information translation process on smart glasses is divided into two steps:

[0477] Step 1: Get source object information

[0478] As shown in FIG17(a), if the smart glasses are used as a display device, data can be transmitted through the communication method between the watch and the terminal device such as a mobile phone PAD, so as to receive the source object, the target object, and the derived translation. This method is the same as the communication method described in step one of the object information translation scheme applied to the smart watch, and will not be repeated here. If the smart glasses are used as a smart terminal device, then on the basis of the method steps S1, S2 or steps S3, S4 of the second implementation, an input method of eye tracking and line of sight focusing is further provided. By tracking the user's eyes, or the time the line of sight stays on the object, or the position where the line of sight is focused, the source object image and other information are captured and uploaded to the server for translation, and then the derived translation is generated according to the steps described in the derivative translation in the single object scheme.

[0479] Step 2: Display the translated target object information

[0480] As shown in FIG. 17( b), the target object is displayed on the glasses, and further, derivative translations may be included. Display methods include: displaying the target object and derivative translations on the glasses by means of augmented reality, such as displaying the translation results by means of semi-transparent overlay. In order to reduce interference with the user's line of sight, the display method of the translation results on smart glasses is simpler than that on terminals such as mobile phone screens, including but not limited to one of the following methods: only the most critical information may be displayed, such as only displaying the meaning of a translated sign without displaying additional derivative translations, or the translated text may be broadcasted by voice; such as Figure 18(a) , 18(b) As shown, the smart glasses detect the sign, translate it into the user's language, display it on the glasses, and broadcast it in real time.

[0481] 3. Feedback Mechanism of User Log Data

[0482] Embodiment 6 of the present invention further provides a solution for collecting large user usage log data as feedback to improve the quality of object translation, including the following three steps:

[0483] Step 1: Collect and analyze user logs, and further collect location information of source objects

[0484] The collected user behaviors include: the number of times the user clicks on the target object to view it, the number of times the user likes the target object, etc. Fig.19 As shown, after the terminal detects that the user clicks on a target object, the number of clicks on the target object is increased by 1. After the terminal detects that the user clicks "Like", the number of likes on the target object is increased by 1.

[0485] The method for collecting the location of the source object is: when the terminal starts object information translation, it records the source object and the current location of the terminal, and automatically determines whether it is a source of the source object based on the location characteristics. If so, it is added to the location information in the object's knowledge graph.

[0486] Step 2: Display updated translation results

[0487] Update model parameters through feedback mechanism, for example, assign higher error weights to target objects with low likes or clicks, and retrain alignment parameters in the alignment model. Update translation results based on the retrained alignment model.

[0488] 4. Crowdsourced Translation Feedback Mechanism

[0489] Embodiment 6 of the present invention further provides a solution for optimizing translation results through crowdsourcing translation, and the processing flow includes the following steps.

[0490] Step 1: Distribute translation tasks and collect translation results

[0491] When the terminal sends an object translation request, in addition to executing the translation process described in Example 2, it also pushes the translation request to other users and invites them to answer, such as Fig. 20 As shown, the user who responds to the request is called the answering user. The answering user can respond to the translation request by directly selecting a suitable answer from the translation results provided by the present invention. If all the provided candidates are not satisfactory, the answer can be submitted by inputting a picture of the object, a text description, an audio, etc. As shown in FIG21, after the answering user inputs multimodal object information, the system searches for matching objects in the knowledge graph after text recognition, text regularization, and speech recognition.

[0492] Step 2: Establish a user translation model and update the translation results based on the user translation model

[0493] The method for establishing the user translation model is specifically described as follows: after collecting the answers of the answering users, the answering users are classified according to their personal information. The answers of each category of users are analyzed, and the answer frequencies are counted and sorted. Each answer is given a weight corresponding to the sorting, and a user translation model from the source object to the target object with crowdsourcing translation feedback weights is obtained. Fig. 20 As shown in the example, according to the user's characteristics and answers, the answering users are divided into three categories, and the answers of each category of users are analyzed and ranked to calculate the translation results of similar users next time.

[0494] The specific method for analyzing the user's answer is: if it is detected that the translation result provided by the user matches an object in the existing knowledge base, the number of occurrences of the combination of <source object, target object> is increased by 1; if it is detected that the translation result provided by the user does not match all objects in the existing knowledge base, a new object object is created and the number of occurrences of the combination of <source object, target object> is initialized to 1.

[0495] When the translation system receives this object translation request again, it will first analyze and classify the requesting user. At this time, the translation result will consider both machine translation candidates and translation candidates with crowdsourced translation feedback weights, calculate the comprehensive score, and give the final translation result.

[0496] Embodiment 6 of the present invention proposes a solution for using object-to-object machine translation on a wearable device. Compared with mobile devices, the data processing and presentation methods are changed to adapt to the characteristics of wearable devices. In order to better adapt to the characteristics of multiple users and better utilize the user data generated by multiple users, the present invention proposes a multi-user-based object translation method and establishes a feedback mechanism based on a large amount of user data to improve the translation quality.

[0497] Embodiment 7

[0498] Considering that the quality of machine translation is highly dependent on the degree of normalization of the input text, the size of the database and the model, existing commercial machine translation technologies usually require the support of the network and cloud services, as well as text normalization. However, current machine translation technologies do not take into account many extreme conditions, such as limited network, insufficient device power, incomplete input text / sound / image information, etc., resulting in a poor user experience.

[0499] Therefore, embodiment seven of the present invention provides an optimization solution for the object information translation method under extreme conditions, including a solution for solving incomplete input object information, a solution for saving traffic and battery, and a solution for object information translation output of the device under different conditions.

[0500] 1. Standardize incomplete input

[0501] In real situations, the following scenarios may cause incomplete input information: the packaging of the item is incomplete, or the text or image on the item packaging is blurred, or the uploaded information, such as the source image, contains too little information. Fig. 22 To ensure the accuracy of the object translation results, the present invention proposes a solution that allows multimodal information input and information completion, including the following steps:

[0502] Step 1: Collection of multimodal input information

[0503] The terminal device will collect at least one of the following information: picture information, text information, audio information, etc. The above information can be directly obtained by the device or input by the user. Among them, the audio can be converted into text and then input into the object recognition system.

[0504] Step 2: Analyze and identify objects

[0505] This step is the same as the analysis and identification process described in step S2 of the second embodiment. However, if the input source object information is incomplete and the system cannot identify the source object, step three is used.

[0506] Step 3: Start a search to find information about the object

[0507] If the source object information collected by the terminal device is not sufficient to identify the object, the network search engine is started or the pictures and texts are searched and matched in the database pre-stored in the local machine to obtain more pictures and texts related to the object. The obtained pictures and text information are then filtered and normalized before executing step 2. If it is still not recognized, repeat steps 2 and 3 until the system recognizes it.

[0508] Specifically, for example, if the score of the existing source object information in the object recognition classifier is lower than the set threshold, we call it that the source object information collected by the terminal device is insufficient to identify the object, then start the network search engine to search and match the pictures and texts, get more pictures and texts about the object, and then filter and normalize the obtained pictures and text information before executing step 2. If it still cannot be recognized, repeat steps 2 and 3 until the system recognizes it, or the number of repetitions exceeds the iteration threshold. The set threshold and iteration threshold here are empirical values. Fig. 22The translation system extracts the corresponding identification features from the source image shown in the figure, classifies and identifies it in the object recognition system, and starts the search engine if the recognition score is lower than the set threshold. It searches for the features of the image on the Internet, and returns the set number of search results to the translation system. It re-extracts the features and repeats the above process until the recognition score is higher than the set threshold, or the number of repeated searches exceeds the iteration threshold. Finally, the object with the highest score in the last round of recognition is obtained as the recognition result.

[0509] 2. Menu Translation

[0510] When dining out, users may encounter foreign dishes and / or menus that need to be translated. In this case, simply providing the translation of the dish name cannot fully meet the user's needs. It is necessary to further provide the corresponding domestic dishes to intuitively provide information such as the ingredients, taste, and eating methods of the dishes;

[0511] Specifically, the dish (meal) translation discussed in the seventh embodiment involves unpackaged dishes and / or menus. Since it is impossible to directly obtain information such as ingredients, allergens, calories, and applicable population, dish translation generally belongs to the case of incomplete source information input. First, the source dishes are matched and identified in the knowledge graph according to the steps described in the standard incomplete input solution, and then the translated target dishes are displayed based on the similarity of ingredients, taste / smell / temperature and other characteristics of the dishes, and can be further sorted based on the degree of matching.

[0512] In addition, for dishes whose names, ingredients, allergens, calories, applicable populations, etc. can be directly obtained by the terminal, such as packaged dishes (packaged foods) and dishes with detailed descriptions, the translation method thereof refers to the schemes provided in Examples 2 to 6. Here, the translation of the famous Indonesian local food dirty duck meal is used as an example to illustrate the complete translation process.

[0513] For existing cooked food and other dishes, the source obtains the object image and uploads it to the server. Then, through the above steps 2 and 3, it is detected that the source object is a dirty duck from Indonesia. Then, through step S1 of embodiment 2, the image features and user personality information are obtained to translate it into dishes familiar to the user. As shown in Figure 23 (a), the system obtains the user's target language as Chinese, and extracts the image features and text features of "dirty duck" (such as ingredients: whole duck, baking methods: roasted, fried, etc.) through the translation system described in step S2 of embodiment 2, and obtains candidate translations of multiple Chinese dishes under the action of the translation model, such as roast duck, crispy fried duck (the image features and text features are similar to those of the source), etc.

[0514] Furthermore, the terminal can also display derivative translation information, the key information involved may be at least one of the following: taste, smell, temperature, color, ingredients, allergens, cooking methods, etc., wherein the taste includes but is not limited to at least one of the following: sweet, salty, spicy, sour, bitter, etc., the temperature includes but is not limited to at least one of the following: ice, cool, warm, hot, scalding, and the allergens include but are not limited to at least one of the following: peanuts, soybeans, seafood, etc.

[0515] Furthermore, according to different personalized needs, the system will give personalized translation results and derivative information through the steps described in the personalized object information translation method. For example, if the system detects that the user is allergic to a certain ingredient or does not like a certain taste, sensitive content such as allergens in the derivative translation will be highlighted according to the personalized object information translation scheme. For example, if it is detected that the user is allergic to peanuts, "peanuts" will be highlighted in the ingredients of the derivative information to remind the user to pay attention. If it is detected that the user does not eat pork, "pork" will be highlighted in the ingredients of the derivative information to remind the user to pay attention. For example, if it is detected that the user does not eat spicy food, the relevant content containing "spicy" will be highlighted in the "taste" column of the derivative information. The method for the system to obtain sensitive content such as whether the user is allergic or eats spicy food is as described in the personalized object information translation scheme.

[0516] Furthermore, based on user operations, key information of the dirty duck and the roast duck can be compared, such as taste, smell, temperature, color, ingredients, allergens, eating methods, cooking methods, etc. Figure 23(b) , 23(c) As shown in the figure. The figure compares the differences in derived information between "dirty duck" and "roast duck", such as "taste", "ingredients", "color", "eating temperature", "recommended eating method", etc. For example, in terms of "taste", "dirty duck" is slightly spicy, the meat is very dry and not greasy, while "roast duck" is fatter and slightly oily; in terms of ingredients, "dirty duck" uses a variety of Indonesian spices, while "roast duck" uses common Chinese ingredients; they are also different in color, eating temperature and eating method. Such a comparative display helps users obtain useful information at a glance and provides users with a convenient and quick way to compare.

[0517] Further, if the source end is a plain text input, such as a menu, as shown in FIG24 (a), the input is a plurality of incomplete object descriptions, firstly, the text recognition is performed on each dish name, and then it is matched to the determined object in the knowledge graph through the above steps 2 and 3, and its picture and derivative information are output in the source end capture area of ​​the terminal through the augmented reality method described in the first method of the third embodiment, as shown in FIG24 (a) and FIG24 (b), and at the same time, through the schemes for derivative translation in the first to sixth embodiments, the corresponding target end dish and derivative information are generated and output to the target end display area of ​​the terminal. Further, the comparison result of the derivative information of the source end and the target end dish can also be provided through the automatic comparison scheme of the source end and the target end object of the multi-object in the fourth embodiment, and the augmented reality interactive operation can also be provided for the translation result through the step 3 described in the third embodiment, including but not limited to: after selecting a dish name in the source end capture area, the target end display area will only display its corresponding translation result, or after selecting a dish in the target end display area, the corresponding dish name is automatically highlighted in the source end capture area, etc. As shown in FIG24(a), the source input is a plurality of dish names, and the source capture area uses AR to display the translation results of the source dish according to the first method in the third embodiment. At the same time, the target display area gives the translation results according to the step S4 of the multi-object translation in the second embodiment and the derivative translation of the multi-object in the fourth embodiment. If the device detects that the user has selected a translation result, as shown in FIG24(b), it is detected that the user has selected "dirty duck", and the operation includes but is not limited to long pressing and clicking a translation in the source dish name translation area, long pressing and clicking a translation result in the target display area, etc., then the detailed information of the corresponding source dish is automatically displayed in the source capture area in an AR manner, including pictures, derivative information, etc., see FIG24(c) for the detailed information of the source dish corresponding to "dirty duck", and further, the information of the corresponding dish of the target is output in the target display area according to the aforementioned method, see FIG24(d).

[0518] Furthermore, if there is no exact match to the source dish in the target language country, the system will give the closest translation result after comprehensive consideration of various features, and further give similarities and differences. For example, if "Sichuan hot pot" is translated into Japanese food, since there is no dish that is very similar to it in Japanese food, but after comprehensive consideration of features such as color, cooking method, and eating method, the closest translation results are Sukiyaki pot, beef hot pot, etc. Furthermore, the similarities and differences between the translation result and the source dish will be given in comparison mode, such as similarities include: the cooking method is to put the ingredients in a pot and add soup to cook, and differences include: taste, ingredients, etc.

[0519] 3. Ways to save data and battery

[0520] Furthermore, Embodiment 7 of the present invention provides a solution for saving data flow and battery, which is used to handle the following possible scenarios, including that there is no wifi or data flow is limited at the location of the terminal, and the user wants to save data flow and battery, such as the destination of overseas travel may not have full wifi coverage, and international data flow charges are expensive. The processing process includes the following steps. The following steps are performed before the basic steps of Embodiment 7 of the present invention.

[0521] Step 1: Predict user intent and pre-download

[0522] The terminal predicts the user's intention by detecting and analyzing at least one of the user's schedule, hobbies, environment, motion status, and the like.

[0523] Specifically, for example, the terminal obtains the user's schedule information by analyzing the information recorded in the user's calendar and / or memo software; the terminal obtains the user's hobbies and interests by analyzing the information recorded in the user's mailbox, text messages, and / or call logs; the terminal obtains the user's environment information through a camera, microphone, sound sensor, light sensor, and / or humidity sensor; the terminal obtains the user's motion status and other information through a speed sensor, etc. The user's possible action trajectory and possible translation target are predicted based on the user's schedule, hobbies, environment, motion status and other information. The prediction model can be learned through a supervised method.

[0524] If the terminal detects that it is currently under WiFi conditions, it prompts the user to download the translation offline model and database related to the destination and / or translation target.

[0525] Furthermore, this step also includes the steps of classification, compression and filtering of models and databases:

[0526] The database is classified into different categories according to the types of objects, and divided into food, medicine, cosmetics, sporting goods, signboards, etc. according to different translation scenarios.

[0527] The translation model is filtered through entropy methods / significance tests, the knowledge graph database is filtered through redundant information detection and pruning, the audio and video database is compressed through codecs such as AMR-EVS and H.265, and the image database is compressed through JPEG format.

[0528] Therefore, if the terminal detects that it is currently under wifi conditions, it prompts the user to download the translation offline model and database related to the destination and translation target that have been refined, filtered and compressed through the above steps.

[0529] like Fig.25As shown, all databases will be classified, filtered, compressed and indexed according to categories. When the terminal detects user information, the server classifies the user's intention and then prompts the user to download the corresponding database under the condition of WiFi.

[0530] Step 2: Use the database offline. Furthermore, the streamlined database can also be used offline.

[0531] When the terminal device detects that the network currently in use is data traffic, it gives priority to searching and translating based on the downloaded models and databases, reducing the communication between the terminal and the cloud service. At the same time, since the database size has been greatly reduced, the search space in the translation process is reduced, avoiding a large amount of calculations, thereby saving power.

[0532] 4. Output strategies in different states

[0533] When the device is in different network environments and different power conditions, Embodiment 7 of the present invention proposes an optimal adaptive strategy, including the system operation platform, the scale of the usage model, and the form of multimodal output, as shown in Table 1 below.

[0534] If the terminal device is connected to wifi and has sufficient power, the operating platform of this translation system is a remote server, the model scale used is the entire database, and the translation results can be presented in text, image, audio and / or video; if the terminal device is not connected to wifi and has sufficient power, the operating platform of this translation system is the terminal device, the model scale used is the filtered database, and the translation results can only be presented in text, image and / or audio; if the terminal device is connected to wifi but has insufficient power, the operating platform of this translation system is a remote server, the entire database is used, and the translation results only output text and / or images; if the terminal device is not connected to wifi and has low power, this translation system

[0535] Run on the terminal device, use the filtered model, and only output text and / or images.

[0536] Table 1 Optimal adaptive strategy

[0537]

[0538] Embodiment 7 of the present invention proposes optimization schemes for various situations, including filtering and compression schemes for models and databases, pre-judgment and pre-download schemes, approximate matching schemes for incomplete input information, and translation result presentation schemes when the battery is low, thereby improving the robustness of the translation system.

[0539] Embodiment 8

[0540] Based on the object information translation method provided in the first embodiment, the eighth embodiment of the present invention provides an object information translation device, such as Fig.26 As shown, the device includes: an object information translation unit 201 and an information output unit 202.

[0541] The object information translation unit 201 is used to translate the target object information corresponding to the source object based on the acquired source object information, and to identify one or more source objects to be translated based on the acquired source object information.

[0542] The information output unit 202 is used to output the target end object information translated by the object information translation unit 201.

[0543] In the solution of the present invention, the specific functions of each unit in the object information translation device provided in Example 8 can refer to the specific steps of the object information translation method provided in Example 1, which will not be described in detail here.

[0544] Embodiment 9

[0545] Embodiment 9 of the present invention provides a method for obtaining derivative information, such as Fig. 27 As shown, the specific process includes:

[0546] S301: Determine object-related derivative information based on the acquired object information.

[0547] Specifically, the corresponding object may be identified based on the acquired object information, wherein the acquired object information may include at least one of the following modalities: text, picture, audio, video, etc.

[0548] In practical applications, object information can be directly captured by the terminal, such as by photographing with a camera. Alternatively, object information can be transmitted from a network or other device. The acquired object information can include at least one of the following: multimedia information corresponding to the object, text information identified from the multimedia information corresponding to the object, location information corresponding to the object, searched object-related information, and object-related information input by the user. For example, the object information that can be acquired can be a picture collected in real time, and the scheme provided by the ninth embodiment of the present invention can generate derivative information corresponding to the picture in real time.

[0549] Regarding how to identify the corresponding object based on the acquired object information, reference may be made to Embodiment 1 to Embodiment 2 of the present invention, which will not be described in detail here.

[0550] After the object is identified, the attribute information corresponding to the preset attribute of the identified object can be searched from the preset object knowledge graph database, and the searched attribute information is confirmed as the derived information associated with the object. The preset attribute is determined according to the object category of the object.

[0551] Alternatively, derived information associated with the object may also be determined based on position information corresponding to the object.

[0552] In practical applications, the number of objects identified is one or more. When the number of objects identified is multiple and the multiple objects belong to the same category of combined objects, the derived information associated with the objects can be determined by at least one of the following:

[0553] For a composite object corresponding to the plurality of objects, determining derivative information associated with the composite object;

[0554] For multiple objects, the derived information associated with each object is obtained respectively.

[0555] S302: Output the determined derivative information.

[0556] Preferably, in the ninth embodiment of the present invention, the language environment corresponding to the acquired derivative information can be determined according to the user's personal information; and the derivative information can be displayed based on the determined language environment.

[0557] Preferably, the derivative information that needs to be highlighted can be located according to the user's personal information; and the located derivative information can be highlighted.

[0558] Preferably, relevant reminder events may also be generated or changed based on the user's personal information and / or the acquired derivative information.

[0559] Regarding the scheme for determining the derived information of the object in the ninth embodiment, reference may be made to the schemes for determining the derived information in the first, fourth and fifth embodiments, which will not be described in detail here.

[0560] Based on the above information acquisition method, the ninth embodiment of the present invention further provides a derivative information acquisition device, such as Fig.28 As shown, it includes: a derivative information acquisition unit 401 and an information output unit 402.

[0561] The derived information acquisition unit 401 is used to determine the derived information associated with the object based on the acquired object information.

[0562] The information output unit 402 is used to output the derived information determined by the derived information acquisition unit 401 .

[0563] In the scheme of the present invention, the specific functional implementation of each unit in the derivative information acquisition device provided in Example 9 can refer to the object information translation method provided in Example 1 and the specific steps of the information acquisition method provided in Example 9, which will not be described in detail here.

[0564] The technical solution provided by the present invention can translate the object itself, not just the text, so as to effectively prevent the translation of text from failing to fully cover the needs of the translation task. Through the pre-built object alignment model, foreign objects that are unfamiliar to the user can be translated into domestic corresponding objects that the user is familiar with; or the translation direction can be exchanged to translate domestic objects into corresponding objects in the target country. In this way, compared with the existing translation objects that are only text, the solution provided by the present invention can meet the user's translation needs for objects, expand the scope of machine translation objects, and enhance the applicability of translation.

[0565] Furthermore, the solution of the present invention proposes the concept of derivative translation based on object-to-object translation. Compared with the existing passive translation, the solution of the present invention can not only provide the direct translation information of the source object, but also analyze the target object corresponding to the source object and provide derivative information about the target object, so as to supplement the translation content and enhance the user's understanding of the translation content.

[0566] Furthermore, the solution of the present invention can also adjust the translation content according to the location information, user information, type of terminal device and other information where the translation occurs, so as to meet different actual translation needs of users.

[0567] Furthermore, the present invention proposes a method for using object-to-object machine translation on a wearable device. Compared with mobile devices, the data processing and presentation methods are changed to adapt to the characteristics of wearable devices.

[0568] Furthermore, in order to better adapt to the multi-user characteristics and better utilize the user data generated by multiple users, the present invention proposes a multi-user-based object translation method and establishes a feedback mechanism based on a large amount of user data to improve the translation quality.

[0569] Furthermore, the present invention also proposes optimization schemes for various situations, including filtering and compression schemes for models and databases, pre-judgment and pre-download schemes, approximate matching schemes for incomplete input information, and translation result presentation schemes when the battery is low, so as to improve the robustness of the translation system.

[0570] It will be appreciated by those skilled in the art that the present invention includes devices for performing one or more of the operations described in the present application. These devices may be specially designed and manufactured for the desired purpose, or may include known devices in general-purpose computers. These devices have computer programs stored therein, which are selectively activated or reconstructed. Such computer programs may be stored in a device (e.g., computer) readable medium or in any type of medium suitable for storing electronic instructions and coupled to a bus, respectively, the computer readable medium including but not limited to any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the readable medium includes any medium that stores or transmits information in a readable form by a device (e.g., computer).

[0571] Those skilled in the art will appreciate that each block in these structure diagrams and / or block diagrams and / or flow charts and combinations of blocks in these structure diagrams and / or block diagrams and / or flow charts can be implemented using computer program instructions. Those skilled in the art will appreciate that these computer program instructions can be provided to a general-purpose computer, a professional computer, or a processor of other programmable data processing methods to implement, thereby executing the scheme specified in the block or multiple blocks of the structure diagram and / or block diagram and / or flow chart disclosed in the present invention through the processor of the computer or other programmable data processing method.

[0572] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes discussed in the present invention may be alternated, modified, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and schemes discussed in the present invention may also be alternated, modified, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in the present invention may also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0573] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method performed by a user equipment, It is characterized in that include: Get source object information; Based on the source object information, identifying the source object; Acquire target information of a target end object corresponding to the source end object; Displaying the target information on a display device according to the user personal information associated with the user's needs, wherein the displayed target information includes a portion of information that is highlighted, and the highlighted information is determined based on the user's needs; Wherein, the acquiring target information of the target end object corresponding to the source end object includes: Determine the language environments corresponding to the source object and the target object respectively; Based on the language environments respectively corresponding to the source-end object and the target-end object, target information of the target-end object corresponding to the source-end object is obtained.

2. The method according to claim 1, It is characterized in that The acquiring target information of the target end object corresponding to the source end object includes: Based on the location information of the user device, target information of a target-end object corresponding to the source-end object is acquired; wherein the target-end object is obtained at a location corresponding to the location information of the user device.

3. The method according to claim 1, It is characterized in that The source object information includes source multimedia information of the source object, and the source multimedia information includes at least one of an image feature, an identification feature, and a text feature associated with the source object.

4. The method according to claim 1, It is characterized in that The target information includes at least one of image features, identification features, text features and description information associated with the target end object.

5. The method according to claim 3, It is characterized in that The identifying of the source object comprises: Based on the source multimedia information of the source object, extract at least one of an image feature, a logo feature, and a text feature associated with the source object; The source object is identified according to at least one of the extracted image features, identification features, and text features associated with the source object.

6. The method according to claim 4, It is characterized in that The acquiring target information of the target end object corresponding to the source end object includes: Searching for attribute information corresponding to a preset attribute of the target end object; The searched attribute information is determined as information associated with the description information.

7. The method according to claim 1, It is characterized in that The acquiring target information of the target end object corresponding to the source end object includes: The target information of the target object is determined according to the location information of the user equipment and the location information of the target object.

8. The method according to claim 1, It is characterized in that Display the target information, including: Displaying at least one piece of information of at least one target object candidate; receiving an input signal for selecting at least one target-end object from among the at least one target-end object candidate; Determine the selected target end object as the target end object corresponding to the source end object, and display at least one piece of information of the target end object; The at least one piece of information of the at least one target-end object candidate is displayed in the order of priority of the at least one target-end object candidate.

9. The method according to claim 8, It is characterized in that Also includes: The information related to the selected target end object and the source end object information are sent to the server.

10. The method according to claim 1, It is characterized in that The identifying of the source object comprises: Displaying at least one piece of information of at least one source object candidate; receiving an input signal for selecting at least one of the at least one piece of information of the at least one source object candidate; The source object is identified according to the selected at least one piece of information.

11. The method according to claim 1, It is characterized in that The user personal information includes information related to the user's language, and the user personal information associated with the user's needs includes at least one of the following information: The user's schedule information, the user's interests, the user's hobbies, the user's environment information, the user's exercise status, and the user's health status.

12. The method according to claim 1, It is characterized in that The identifying of the source object comprises: Extracting image features and / or text features based on the acquired source object information; The source object is identified based on the extracted image features and / or text features.

13. The method according to claim 12, It is characterized in that The step of identifying the source object based on the extracted image features and / or text features includes: The source object is identified based on the preset object recognition model and the extracted image features and / or text features.

14. The method according to claim 13, It is characterized in that The object recognition model is pre-built in the following way: For each object category, an object recognition model corresponding to the object category is trained based on image features and / or text features of each sampled object belonging to the object category.

15. The method according to claim 14, It is characterized in that The identifying of the source object according to the preset object recognition model and the extracted image features and / or text features includes: Identify the object category of the source object based on the extracted image features and / or text features; The source object is identified according to the object recognition model corresponding to the object category and the extracted image features and / or text features.

16. The method according to any one of claims 1 to 15, It is characterized in that The language environment corresponding to the source object is determined according to at least one of the following: The language environment corresponding to the location detected by the location detection module, the language environment identified from the acquired source object information, the language environment corresponding to the source object searched in the preset object knowledge graph database, and the language environment set by the user; The language environment corresponding to the target object is determined according to at least one of the following: The language environment corresponding to the location detected by the location detection module, the pre-specified language environment, the language environment set by the user, and the language environment determined based on the user's personal information.

17. The method according to any one of claims 1 to 15, It is characterized in that Based on the language environments corresponding to the source object and the target object, target information of the target object corresponding to the source object is obtained, including: Selecting a corresponding object alignment model based on the language environments corresponding to the source object and the target object; Based on the selected object alignment model, target information of a target-end object corresponding to the identified source-end object is obtained.

18. The method according to claim 17, It is characterized in that Based on the selected object alignment model, target information of a target object corresponding to the identified source object is obtained, including: Based on the source object information, obtain text features and / or image features corresponding to the source object; Based on the acquired text features and / or image features and the selected object alignment model, target information of a target-end object corresponding to the identified source-end object is obtained.

19. The method according to claim 18, It is characterized in that The object alignment model is pre-built in the following way: Determine text features and / or image features between a sampled source end object and a corresponding sampled target end object; An object alignment model is trained based on the determined text features and / or image features.

20. The method according to claim 17, It is characterized in that Also includes: Optimizing the object alignment model based on at least one of the following information: The user's feedback information on the target object information obtained by translation, the language environment corresponding to the source object, and the language environment corresponding to the target object.

21. The method according to any one of claims 1 to 15, It is characterized in that There are multiple source objects, and the multiple source objects belong to the same category of combined objects; The step of acquiring target information of a target end object corresponding to a source end object includes at least one of the following: For source-side combined objects corresponding to multiple source-side objects, corresponding target-side combined object information is obtained; For each source-end object, the target information of the corresponding target-end object is obtained respectively.

22. The method according to any one of claims 1 to 15, It is characterized in that When there are multiple target objects, displaying the target information includes: The target information of each target end object is arranged and sorted and then output.

23. The method according to claim 22, It is characterized in that The target information of each target end object is sorted according to at least one of the following information: The correlation with the source object, the user behavior with respect to the target information of the target object, the attribute information of the target object, and the user personal information.

24. The method according to claim 3, It is characterized in that The source multimedia information further includes at least one of the following: Multimedia information corresponding to the source object, text information identified from the multimedia information corresponding to the source object, location information corresponding to the source object, searched source object related information, and source object related information input by the user.

25. The method according to claim 24, It is characterized in that Multimedia information is collected in real time through multimedia collection equipment, and the real-time collected multimedia information is used as multimedia information corresponding to the source object.

26. The method according to claim 25, It is characterized in that There are multiple source objects; The method further comprises: Detecting a user's selection operation on target information of a target object obtained by translation; A source-end object corresponding to target information of a target-end object selected by a user is located in the multimedia information collected in real time.

27. The method according to any one of claims 1 to 15, It is characterized in that Display the target information, including: Collect multimedia information in real time through multimedia acquisition equipment; The target end object corresponding to the source end object is located in the multimedia information collected in real time.

28. The method according to any one of claims 1 to 15, It is characterized in that Before acquiring the target information of the target end object corresponding to the source end object, the method further includes: Predicting the user's translation intention and pre-storing a processing model for offline translation based on the user's translation intention; The acquiring, based on the location information of the user equipment, target information of the target end object corresponding to the source end object comprises: Based on the location information of the user equipment, the target information of the target end object corresponding to the source end object is acquired by utilizing the stored processing model.

29. The method according to claim 28, It is characterized in that The user's translation intention is predicted based on at least one of the following information: User schedule, user personal information, device environment information, and device motion status.

30. The method according to any one of claims 1 to 15, It is characterized in that The displaying of the target information includes: Display target information in at least one of the following ways: text, image, audio, video.

31. The method according to any one of claims 1 to 15, It is characterized in that The displaying of the target information includes: Based on at least one of the following, the target information is adaptively adjusted, and the adjusted target information is output: Device type, device storage status, current network status, device power status, and user personal information.

32. The method according to any one of claims 1 to 15, It is characterized in that Also includes: Acquire the derived information associated with the target object and / or the derived information associated with the source object; The obtained derivative information is output.

33. The method according to claim 32, It is characterized in that Obtaining derivative information associated with the target object, including: Searching for attribute information corresponding to a preset attribute of the target object from a preset object knowledge graph database; Confirming the searched attribute information as derived information associated with the target end object; Obtaining derivative information associated with the source object, including: Searching for attribute information corresponding to the preset attribute of the source object from a preset object knowledge graph database; The searched attribute information is confirmed as the derived information associated with the source object.

34. The method according to claim 33, It is characterized in that The preset attribute is determined according to the category of the target end object and / or the source end object.

35. The method according to claim 32, It is characterized in that Obtaining derivative information associated with the target object, including: Determining derivative information associated with the target end object based on the position information corresponding to the target end object; Obtaining derivative information associated with the source object, including: Based on the position information corresponding to the source object, the derived information associated with the source object is determined.

36. The method according to claim 32, It is characterized in that If the derived information associated with the target end object and the derived information associated with the source end object are obtained, then the derived information obtained is output, including: Locating the derived information that needs to be highlighted according to the correlation between the derived information associated with the target-end object and the derived information associated with the source-end object; Output the derived information obtained and highlight the located derived information.

37. The method according to claim 32, It is characterized in that There are multiple source objects, and the multiple source objects belong to the same category of combined objects; Obtaining derivative information associated with the target object, including at least one of the following: For source-side composite objects corresponding to multiple source-side objects, obtaining derivative information associated with corresponding target-side composite object information; For each source object, the derived information associated with the corresponding target object is obtained.

38. The method according to claim 32, It is characterized in that Output the derived information obtained, including: Determining, based on the user personal information, a language environment corresponding to the obtained derivative information; The derived information is displayed based on the determined locale.

39. The method according to claim 32, It is characterized in that Also includes: Based on the user's personal information, locate the derivative information that needs to be highlighted; Highlight the located derivative information.

40. The method according to claim 32, It is characterized in that Also includes: Generate or change related reminder events based on the user's personal information and / or the derived information obtained.

41. The method according to any one of claims 1 to 15, It is characterized in that Obtain the target information of the target object corresponding to the source object, including: Obtain source object information to be sent to the recipient; Based on the language environment of the receiver and the acquired source object information, acquiring target information of a target object corresponding to the source object; The displaying of the target information includes: The obtained target information is sent to the receiving party.

42. An electronic device, It is characterized in that include: An object information translation unit, configured to obtain source-end object information, identify the source-end object based on the source-end object information, and obtain target information of a target-end object corresponding to the source-end object; An information output unit, configured to display the target information obtained by the object information translation unit on a display device according to the user personal information associated with the user's needs, wherein the output target information includes highlighted information, and the highlighted information is determined based on the user's needs; Wherein, when acquiring target information of a target end object corresponding to a source end object, the object information translation unit is used to: Determine the language environments corresponding to the source object and the target object respectively; Based on the language environments respectively corresponding to the source-end object and the target-end object, target information of the target-end object corresponding to the source-end object is obtained.

43. The apparatus according to claim 42, It is characterized in that The object information translation unit is used to obtain target information of a target end object corresponding to a source end object: Based on the location information of the user device, target information of a target-end object corresponding to the source-end object is acquired; wherein the target-end object is obtained at a location corresponding to the location information of the user device.

44. The apparatus according to claim 42, It is characterized in that The source object information includes source multimedia information of the source object, and the source multimedia information includes at least one of an image feature, an identification feature, and a text feature associated with the source object.

45. The apparatus according to claim 42, It is characterized in that The target information includes at least one of image features, identification features, text features and description information associated with the target end object.

46. ​​The apparatus according to claim 44, It is characterized in that The object information translation unit is used for: Based on the source multimedia information of the source object, extract at least one of an image feature, a logo feature, and a text feature associated with the source object; The source object is identified according to at least one of the extracted image features, identification features, and text features associated with the source object.

47. The apparatus according to claim 45, It is characterized in that The object information translation unit is used for: Searching for attribute information corresponding to a preset attribute of the target end object; The searched attribute information is determined as information associated with the description information.

48. The apparatus according to claim 42, It is characterized in that The object information translation unit is used for: The target information of the target object is determined according to the location information of the user equipment and the location information of the target object.

49. The apparatus according to claim 42, It is characterized in that The information output unit is used to display the target information: Displaying at least one piece of information of at least one target object candidate; receiving an input signal for selecting at least one target-end object from among the at least one target-end object candidate; Determine the selected target end object as the target end object corresponding to the source end object, and display at least one piece of information of the target end object; The at least one piece of information of the at least one target-end object candidate is displayed in the order of priority of the at least one target-end object candidate.

50. The apparatus according to claim 49, It is characterized in that The information output unit is also used for: The information related to the selected target end object and the source end object information are sent to the server.

51. The apparatus according to claim 42, It is characterized in that The object information translation unit is used for: Displaying at least one piece of information of at least one source object candidate; receiving an input signal for selecting at least one of the at least one piece of information of the at least one source object candidate; The source object is identified according to the selected at least one piece of information.

52. The apparatus according to claim 42, It is characterized in that The user personal information includes information associated with the user's language, and the user personal information associated with the user's needs includes at least one of the following information: The user's schedule information, the user's interests, the user's hobbies, the user's environment information, the user's exercise status, and the user's health status.

53. The apparatus according to claim 42, It is characterized in that When identifying the source object, the object information translation unit is used to: Extracting image features and / or text features based on the acquired source object information; The source object is identified based on the extracted image features and / or text features.

54. The apparatus according to claim 53, It is characterized in that The object information translation unit is used for: The source object is identified based on the preset object recognition model and the extracted image features and / or text features.

55. The apparatus according to claim 54, It is characterized in that The object recognition model is pre-built in the following way: For each object category, an object recognition model corresponding to the object category is trained based on image features and / or text features of each sampled object belonging to the object category.

56. The apparatus according to claim 42, It is characterized in that The object information translation unit is used for: Identify the object category of the source object based on the extracted image features and / or text features; The source object is identified according to the object recognition model corresponding to the object category and the extracted image features and / or text features.

57. Apparatus according to any one of claims 42 to 56, It is characterized in that The language environment corresponding to the source object is determined according to at least one of the following: The language environment corresponding to the location detected by the location detection module, the language environment identified from the acquired source object information, the language environment corresponding to the source object searched in the preset object knowledge graph database, and the language environment set by the user; The language environment corresponding to the target object is determined according to at least one of the following: The language environment corresponding to the location detected by the location detection module, the pre-specified language environment, the language environment set by the user, and the language environment determined based on the user's personal information.

58. Apparatus according to any one of claims 42 to 56, It is characterized in that The object information translation unit is used for: Selecting a corresponding object alignment model based on the language environments corresponding to the source object and the target object; Based on the selected object alignment model, target information of a target-end object corresponding to the identified source-end object is obtained.

59. The apparatus according to claim 58, It is characterized in that When the object information translation unit acquires the target information of the target end object corresponding to the identified source end object based on the selected object alignment model, it is used to: Based on the source object information, obtain text features and / or image features corresponding to the source object; Based on the acquired text features and / or image features and the selected object alignment model, target information of a target-end object corresponding to the identified source-end object is obtained.

60. The apparatus according to claim 59, It is characterized in that The object alignment model is pre-built in the following way: Determine text features and / or image features between a sampled source end object and a corresponding sampled target end object; An object alignment model is trained based on the determined text features and / or image features.

61. The apparatus according to claim 58, It is characterized in that The object information translation unit is also used for: Optimizing the object alignment model based on at least one of the following information: The user's feedback information on the target object information obtained by translation, the language environment corresponding to the source object, and the language environment corresponding to the target object.

62. The apparatus according to any one of claims 42 to 56, It is characterized in that There are multiple source objects, and the multiple source objects belong to the same category of combined objects; When acquiring target information of a target-end object corresponding to a source-end object, the object information translation unit is configured to perform at least one of the following: For source-side combined objects corresponding to multiple source-side objects, corresponding target-side combined object information is obtained; For each source-end object, the target information of the corresponding target-end object is obtained respectively.

63. The apparatus according to any one of claims 42 to 56, It is characterized in that When there are multiple target objects, the information output unit is used to: The target information of each target end object is arranged and sorted and then output.

64. The apparatus according to claim 63, It is characterized in that The target information of each target end object is sorted according to at least one of the following information: Correlation with the source object, user behavior with respect to the target object information, attribute information of the target object, and user personal information.

65. The apparatus according to claim 44, It is characterized in that The source multimedia information includes at least one of the following: Multimedia information corresponding to the source object, text information identified from the multimedia information corresponding to the source object, location information corresponding to the source object, searched source object related information, and source object related information input by the user.

66. The apparatus according to claim 65, It is characterized in that Multimedia information is collected in real time through multimedia collection equipment, and the real-time collected multimedia information is used as multimedia information corresponding to the source object.

67. The apparatus according to claim 66, It is characterized in that There are multiple source objects; The object information translation unit is also used for: Detecting a user's selection operation on the target object information obtained by translation; A source-end object corresponding to target information of a target-end object selected by a user is located in the multimedia information collected in real time.

68. The apparatus according to any one of claims 42 to 56, It is characterized in that When displaying the target information, the information output unit is used to: Collect multimedia information in real time through multimedia acquisition equipment; The target end object corresponding to the source end object is located in the multimedia information collected in real time.

69. The apparatus according to any one of claims 42 to 56, It is characterized in that The object information translation unit is also used for: Predicting the user's translation intention and pre-storing a processing model for offline translation based on the user's translation intention; When acquiring target information of a target-end object corresponding to a source-end object, the object information translation unit is used to: Based on the acquired source-end object information, the target information of the target-end object corresponding to the source-end object is acquired by utilizing the stored processing model.

70. The apparatus according to claim 69, It is characterized in that The user's translation intention is predicted based on at least one of the following information: User schedule, user personal information, device environment information, and device motion status.

71. The apparatus according to any one of claims 42 to 56, It is characterized in that The displaying of the target information includes: Output the target information in at least one of the following ways: text, image, audio, video.

72. The apparatus according to any one of claims 42 to 56, It is characterized in that The information output unit is used for: Based on at least one of the following, adaptively adjust the target information of the target end object and output the adjusted target information: Device type, device storage status, current network status, device power status, and user personal information.

73. The apparatus according to any one of claims 42 to 56, It is characterized in that The information output unit is also used for: Acquire the derived information associated with the target object and / or the derived information associated with the source object; The obtained derivative information is output.

74. The apparatus according to claim 73, It is characterized in that When acquiring the derivative information associated with the target end object, the information output unit is used to: Searching for attribute information corresponding to a preset attribute of the target object from a preset object knowledge graph database; Confirming the searched attribute information as derived information associated with the target end object; Obtaining derivative information associated with the source object, including: Searching for attribute information corresponding to the preset attribute of the source object from a preset object knowledge graph database; The searched attribute information is confirmed as the derived information associated with the source object.

75. The apparatus according to claim 74, It is characterized in that The preset attribute is determined according to the category of the target end object and / or the source end object.

76. The apparatus according to claim 73, It is characterized in that When acquiring the derivative information associated with the target end object, the information output unit is used to: Determining derivative information associated with the target end object based on the position information corresponding to the target end object; Obtaining derivative information associated with the source object, including: Based on the position information corresponding to the source object, the derived information associated with the source object is determined.

77. The apparatus according to claim 73, It is characterized in that If the derived information associated with the target end object and the derived information associated with the source end object are obtained, the information output unit, when outputting the obtained derived information, is used to: Locating the derived information that needs to be highlighted according to the correlation between the derived information associated with the target-end object and the derived information associated with the source-end object; Output the derived information obtained and highlight the located derived information.

78. The apparatus according to claim 73, It is characterized in that There are multiple source objects, and the multiple source objects belong to the same category of combined objects; When the information output unit obtains the derivative information associated with the target end object information, it is used to perform at least one of the following: For source-side composite objects corresponding to multiple source-side objects, obtaining derivative information associated with corresponding target-side composite object information; For each source object, the derived information associated with the corresponding target object is obtained.

79. The apparatus according to claim 73, It is characterized in that When outputting the derived information obtained, the information output unit is used to: Determining, based on the user personal information, a language environment corresponding to the obtained derivative information; The derived information is displayed based on the determined locale.

80. The apparatus of claim 73, It is characterized in that The information output unit is also used for: Based on the user's personal information, locate the derivative information that needs to be highlighted; Highlight the located derivative information.

81. The apparatus according to claim 73, It is characterized in that The information output unit is also used for: Generate or change related reminder events based on the user's personal information and / or the derived information obtained.

82. The apparatus according to any one of claims 42 to 56, It is characterized in that The object information translation unit is used for: Obtain source object information to be sent to the recipient; Based on the language environment of the receiver and the acquired source object information, obtaining target information of a target object corresponding to the source object; The information output unit is used to display the target information: The obtained target information is sent to the receiving party.

83. An electronic device, It is characterized in that The electronic device comprises a memory and a processor; The memory is used to store computer programs; When the processor runs the computer program, the processor performs the method according to any one of claims 1 to 42.

84. A computer readable storage medium, It is characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 42 is executed.