Image translation method and device, electronic equipment and storage medium
By converting the initial image into a character sequence and determining the target image, the error propagation problem of cascade method in image translation is solved, and the translation quality is improved.
Patent Information
- Application Number
- CN202311387897.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-24
- Publication Date
- 2025-04-25
AI Technical Summary
The prior art is based on the cascade method when translating images, and there are error propagation problems, which affects the translation quality.
By converting the initial image into a character sequence and determining the target image based on the character sequence, the recognition of the initial image text is avoided and error propagation is reduced.
Improve the quality of image translation and avoid error propagation problems caused by text recognition errors.
Smart Images
Figure CN119887956A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, and in particular to an image translation method, device, electronic device and storage medium. Background Art
[0002] In-Image Machine Translation (IIMT) can convert an image containing text in one language into an image containing text in another language.
[0003] When performing image translation, the related art generally translates the image to be translated based on the cascade method. Since the cascade method has the problem of error propagation, it will affect the translation quality of the image. Summary of the invention
[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image translation method, device, electronic device and storage medium.
[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image translation method, the method comprising:
[0006] Get an initial image with source language text;
[0007] Inputting the initial image into an image translation model to obtain a target image with target language text;
[0008] The image translation model is used to obtain the target image through the following operations:
[0009] Converting the initial image into a character sequence;
[0010] The target image is determined according to the character sequence.
[0011] Optionally, converting the initial image into a character sequence comprises:
[0012] Converting the initial image into a character image;
[0013] The characters in the character image are sorted according to a preset character arrangement order to obtain the character sequence.
[0014] Optionally, converting the initial image into a character image comprises:
[0015] Converting the initial image into a grayscale image;
[0016] For each pixel value in the grayscale image, determining a corresponding target character;
[0017] The character image is determined according to the target characters corresponding to all pixel values in the grayscale image.
[0018] Optionally, determining the corresponding target character for each pixel value in the grayscale image includes:
[0019] For each pixel value in the grayscale image, perform the following steps:
[0020] Discretizing the pixel value to obtain a target discrete interval corresponding to the pixel value;
[0021] According to the target discrete interval and a preset corresponding relationship, a corresponding target character is determined, wherein the corresponding relationship is used to characterize the corresponding relationship between the discrete interval and the character.
[0022] Optionally, determining the target image according to the character sequence includes:
[0023] According to a preset character merging rule, characters in the character sequence are merged to obtain a character merging sequence;
[0024] Encoding the character combination sequence to obtain a coding information matrix;
[0025] Decoding the encoding information matrix to obtain a target character merge sequence, and splitting the merged characters in the target character merge sequence into single characters to obtain a target character sequence;
[0026] The target image is determined according to the target character sequence.
[0027] Optionally, merging characters in the character sequence according to a preset character merging rule to obtain a character merging sequence includes:
[0028] Taking the character sequence as a first character sequence to be processed, traversing from the first sub-rule in the character merging rule, and determining a first target character and a second target character in the first character sequence to be processed according to the characters included in the traversed sub-rule;
[0029] The first target character and the second target character adjacent to each other in the character sequence are merged to obtain a new character sequence, and the new character sequence is used as the first character sequence to be processed, until all sub-rules in the character merging rule are traversed to obtain a character merging sequence.
[0030] Optionally, the characters included in each sub-rule in the character merging rule are determined in the following manner:
[0031] Obtain a starting sample character sequence, take the starting sample character sequence as a second character sequence to be processed, and loop through the following process:
[0032] Determine the frequency of occurrence of the character bigram included in the second character sequence to be processed, and determine the characters in the character bigram with the highest occurrence frequency as the characters included in the first sub-rule, wherein the character bigram is used to represent two adjacent characters in the second character sequence to be processed;
[0033] The adjacent first target characters and second target characters in the starting sample character sequence are merged to obtain a new starting sample character sequence, and the new starting character sequence is used as a second character sequence to be processed until a preset number of merging times is reached.
[0034] Optionally, determining the target image according to the target character sequence includes:
[0035] For each character in the target character sequence, convert the character into a grayscale value;
[0036] According to all the converted grayscale values, determine the target grayscale image;
[0037] The target grayscale image is converted into a color image, and the color image is determined as the target image.
[0038] Optionally, for each character in the target character sequence, converting the character into a grayscale value includes:
[0039] For each character in the target character sequence, the character is converted into an integer, and the product of the integer and a preset conversion coefficient is determined, and the product is determined as a grayscale value.
[0040] Optionally, the training process of the image translation model includes:
[0041] Acquire a sample image annotated with a label, wherein the label is used to indicate a standard image after translating the sample image, the sample image is an image obtained by embedding a source language text into a blank image, and the standard image is an image obtained by embedding a target language text into a blank image;
[0042] Inputting the sample image into the image translation model to obtain a predicted image, and determining a loss function value according to the predicted image and the label;
[0043] The parameters of the image translation model are updated according to the loss function value.
[0044] According to a second aspect of an embodiment of the present disclosure, there is provided an image translation device, the device comprising:
[0045] An acquisition module, configured to acquire an initial image with source language text;
[0046] A processing module, configured to input the initial image into an image translation model to obtain a target image with target language text;
[0047] Wherein, the image translation model includes:
[0048] A conversion module, configured to convert the initial image into a character sequence;
[0049] The determination module is configured to determine the target image according to the character sequence.
[0050] Optionally, the conversion module includes:
[0051] A conversion submodule, configured to convert the initial image into a character image;
[0052] The sorting submodule is configured to sort the characters in the character image according to a preset character arrangement order to obtain the character sequence.
[0053] Optionally, the conversion submodule includes:
[0054] A first conversion unit, configured to convert the initial image into a grayscale image;
[0055] A first determining unit is configured to determine a corresponding target character for each pixel value in the grayscale image;
[0056] The second determining unit is configured to determine the character image according to the target characters corresponding to all pixel values in the grayscale image.
[0057] Optionally, the first determining unit includes:
[0058] A processing subunit is configured to discretize the pixel value to obtain a target discrete interval corresponding to the pixel value;
[0059] The first determination subunit is configured to determine a corresponding target character according to the target discrete interval and a preset corresponding relationship, wherein the corresponding relationship is used to characterize a corresponding relationship between the discrete interval and the character.
[0060] Optionally, the determining module includes:
[0061] a merging submodule, configured to merge characters in the character sequence according to a preset character merging rule to obtain a character merging sequence;
[0062] An encoding submodule is configured to encode the character combination sequence to obtain an encoding information matrix;
[0063] A splitting submodule is configured to decode the encoding information matrix to obtain a target character merge sequence, and split the merged characters in the target character merge sequence into single characters to obtain a target character sequence;
[0064] The determination submodule is configured to determine the target image according to the target character sequence.
[0065] Optionally, the merging submodule includes:
[0066] A third determining unit is configured to use the character sequence as a first character sequence to be processed, traverse from the first sub-rule in the character merging rule, and determine a first target character and a second target character in the first character sequence to be processed according to the characters included in the traversed sub-rule;
[0067] The merging unit is configured to merge the adjacent first target characters and the second target characters in the character sequence to obtain a new character sequence, and use the new character sequence as the first character sequence to be processed until all sub-rules in the character merging rule are traversed to obtain a character merging sequence.
[0068] Optionally, the device further comprises:
[0069] The character determination module is configured to obtain a starting sample character sequence, use the starting sample character sequence as a second character sequence to be processed, and cyclically execute the following process: determine the frequency of occurrence of a character bigram included in the second character sequence to be processed, and determine the characters in the character bigram with the highest occurrence frequency as the characters included in the first sub-rule, wherein the character bigram is used to represent two adjacent characters in the second character sequence to be processed; merge the first target character and the second target character that are adjacent in the starting sample character sequence to obtain a new starting sample character sequence, and use the new starting character sequence as the second character sequence to be processed until a preset number of merges is reached.
[0070] Optionally, the determining submodule includes:
[0071] A second conversion unit is configured to convert each character in the target character sequence into a grayscale value;
[0072] a fourth determining unit, configured to determine a target grayscale image according to all converted grayscale values;
[0073] The fifth determining unit is configured to convert the target grayscale image into a color image and determine the color image as the target image.
[0074] Optionally, the second conversion unit includes:
[0075] The second determining subunit is configured to convert each character in the target character sequence into an integer, determine the product of the integer and a preset conversion coefficient, and determine the product as a grayscale value.
[0076] Optionally, the device further comprises:
[0077] A training module is configured to obtain a sample image annotated with a label, wherein the label is used to indicate a standard image after the sample image is translated, the sample image is an image obtained by embedding a source language text into a blank image, and the standard image is an image obtained by embedding a target language text into a blank image; input the sample image into the image translation model to obtain a predicted image, and determine a loss function value based on the predicted image and the label; and update the parameters of the image translation model based on the loss function value.
[0078] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:
[0079] processor;
[0080] a memory for storing processor-executable instructions;
[0081] Wherein, the processor is configured to:
[0082] Execute the steps of the image translation method provided in any one of the first aspects of the present disclosure.
[0083] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the image translation method provided in any one of the first aspects of the present disclosure are implemented.
[0084] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:
[0085] Through the above technical solution, the initial image with the source language text can be input into the image translation model to obtain the target image with the target language text. In this way, the image-to-image (i.e., end-to-end) image translation task can be achieved. On the other hand, when the image translation model determines the target image based on the initial image, it first converts the initial image into a character sequence, and then determines the target image based on the character sequence. Compared with the related art that determines the target image based on the cascade method, the present disclosure does not need to recognize the text in the initial image, thereby avoiding the error propagation problem caused by text recognition errors, thereby improving the translation quality of the initial image.
[0086] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0087] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0088] Figure 1 is a schematic diagram of an image translation process in a related technology according to an exemplary embodiment;
[0089] Figure 2 is a flow chart of an image translation method according to an exemplary embodiment;
[0090] Figure 3 is a schematic diagram of an application scenario of an image translation method according to an exemplary embodiment;
[0091] Figure 4 is a schematic diagram of character merging according to an exemplary embodiment;
[0092] Figure 5 is a schematic diagram of a translation process of an image translation model according to an exemplary embodiment;
[0093] Figure 6 is a schematic diagram showing a method of constructing a training sample image according to an exemplary embodiment;
[0094] Figure 7 is a schematic diagram of a test picture with incomplete text according to an exemplary embodiment;
[0095] Figure 8 is a block diagram of a text translation device according to an exemplary embodiment;
[0096] Fig. 9 is a block diagram of a device for text translation according to an exemplary embodiment;
[0097] Fig.10 The invention is a block diagram of a device for text translation according to an exemplary embodiment. DETAILED DESCRIPTION
[0098] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0099] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.
[0100] As mentioned in the background, image translation is the process of converting an image containing text in one language into text in another language. This enables visual text translation across different languages to help users understand street signs and scanned documents, etc., without having to manually enter text.
[0101] When performing image translation, the related technology generally translates the original image based on the cascade method. Specifically, the optical character recognition (OCR) model is first used to recognize the text in the original image, and then the text recognized by the OCR model is translated into the target text through the neural machine translation (NMT) model. Finally, the text in the original image is eliminated and the translated target text is re-embedded into the original image after the text is eliminated to obtain the translated target image.
[0102] Since there is a problem of text recognition error when using the OCR model to recognize the text in the original image, it will affect the translation quality of the image.
[0103] For example, Figure 1 As shown in (a), when the cascade method is used to translate the original image to be translated, the "STADT" in the original image is mistakenly recognized as "STAOT" by the OCR model, which causes the translation result of the NMT model to be wrong, that is, the incorrect translation text "ACCESS TO STAOT" is obtained; after the translated text is embedded into the original image, an incorrect target image will be output.
[0104] In addition, since some text in the original image may be incomplete due to occlusion by obstacles, the OCR model recognizes incomplete text, which leads to the degradation of the translation results and output image quality of NMT.
[0105] To overcome the above problems, an end-to-end image translation model is proposed in the related art, which translates the input image by learning the relationship between the input image and the output image. Figure 1 (b) as shown.
[0106] However, the end-to-end image translation models in related technologies generally include the Convolutional Baseline Model (CBM) and the Attention Convolutional Model (ACM). Among them, the Convolutional Baseline Model is based on the UNet architecture and can only generate images containing one or two words. The Attention Convolutional Model contains a convolutional encoder, a convolutional decoder, and a self-attention encoder, which is greatly affected by the incomplete text in the image, so there is a problem of poor output image quality or poor translation quality.
[0107] In view of this, the embodiments of the present disclosure provide an image translation method, device, electronic device and storage medium to solve the above technical problems.
[0108] The embodiments of the present disclosure are further explained below with reference to the accompanying drawings.
[0109] Figure 2 is a flowchart of an image translation method according to an exemplary embodiment. Figure 2 As shown, the image translation method may include steps S201-S202.
[0110] In step S201, an initial image with source language text is obtained.
[0111] The initial image with the source language text can be determined according to the actual application scenario, and the embodiments of the present disclosure do not impose any restrictions on this. For example, in the scenario of road sign recognition, the initial image with the source language text can be obtained by taking a photo of the road sign. For another example, in the scenario of document content recognition, the initial image with the source language text can be obtained by taking a photo or screenshot of the document content.
[0112] In step S202, the initial image is input into an image translation model to obtain a target image with target language text.
[0113] The image translation model is used to obtain the target image through the following operations:
[0114] Converting the initial image into a character sequence;
[0115] The target image is determined according to the character sequence.
[0116] For example, a user may take a photo of a road sign and input the road sign image obtained by taking the photo into the image translation model to obtain a translated road sign image, such as Figure 3 shown.
[0117] Through the above technical solution, the initial image with the source language text can be input into the image translation model to obtain the target image with the target language text. In this way, the image-to-image (i.e., end-to-end) image translation task can be achieved. On the other hand, when the image translation model determines the target image based on the initial image, it first converts the initial image into a character sequence, and then determines the target image based on the character sequence. Therefore, compared with the related art that determines the target image based on the cascade method, the present disclosure does not need to recognize the text in the initial image, thereby avoiding the error propagation problem caused by text recognition errors, thereby improving the translation quality of the initial image.
[0118] In order to more clearly understand the image translation method in the present disclosure, the processing process of the image translation model on the initial image is described in detail below.
[0119] In a possible implementation, converting the initial image into a character sequence may include:
[0120] The initial image is converted into a character image; and the characters in the character image are sorted according to a preset character arrangement order to obtain the character sequence.
[0121] It should be understood that the character arrangement order can be set according to actual conditions, and the embodiments of the present disclosure do not impose any restrictions on this. In a possible implementation, the character arrangement order can be the arrangement order of character columns in a character image. Accordingly, the characters in the character image are sorted according to a preset character arrangement order to obtain a character sequence, which can be: arranging all character columns in the character image into a row, and making the last character in the i-th character column adjacent to the first character in the i+1-th character column, thereby converting the character image into a character sequence.
[0122] In a possible implementation manner, converting the initial image into a character image may include:
[0123] The initial image is converted into a grayscale image; for each pixel value in the grayscale image, a corresponding target character is determined; and according to the target characters corresponding to all pixel values in the grayscale image, the character image is determined.
[0124] It should be understood that, for each pixel value in the grayscale image, the corresponding target character can be determined by converting according to a preset conversion formula between the pixel value and the character, so as to obtain the target character corresponding to each pixel value. Alternatively, the target character corresponding to each pixel value can be obtained by converting according to a preset correspondence between the pixel value and the character, and the embodiments of the present disclosure do not impose any limitation on this.
[0125] In a possible implementation manner, determining the corresponding target character for each pixel value in the grayscale image may include:
[0126] For each pixel value in the grayscale image, perform the following steps:
[0127] The pixel value is discretized to obtain a target discrete interval corresponding to the pixel value; and the corresponding target character is determined according to the target discrete interval and a preset corresponding relationship, wherein the corresponding relationship is used to characterize the corresponding relationship between the discrete interval and the character.
[0128] It should be understood that each pixel value in a grayscale image is a floating point number between 0 and 1. If a character is assigned to each pixel value, a large number of characters will be obtained, thereby increasing the difficulty of establishing and processing the vocabulary. Therefore, this embodiment discretizes all pixel values in the grayscale image to divide all pixel values into several intervals, and assigns a specific character to each interval, thereby reducing the number of characters, making the size of the vocabulary controllable, and facilitating modeling and processing.
[0129] It should also be understood that the correspondence between discrete intervals and characters can be set according to actual conditions, and the embodiments of the present disclosure do not impose any restrictions on this. In order to make the character sequence after discretization have a certain diversity, in a possible implementation, 0 and 1 can be divided into 10 discrete intervals, and a character is configured for each discrete interval, thereby obtaining the correspondence between discrete intervals and characters.
[0130] For example, the characters can be configured for each discrete interval as follows:
[0131] For each pixel value in the sample grayscale image, first calculate the ratio of the pixel value to the conversion coefficient k1, then round down the ratio to get an integer value; finally, convert the integer to a character using the integer-character conversion function. That is, the character of each interval can be determined by the following formula:
[0132]
[0133] Wherein, c represents a character, char() represents an integer-character conversion function, and v represents a pixel value. Wherein, k1 is set according to actual conditions, and the disclosed embodiment does not impose any restrictions on this. Since each pixel value is a floating point number between 0 and 1, in order to avoid rounding down the comparison value to obtain the same integer value, in a possible implementation, k1 can be set to 0.1, thereby increasing the diversity of characters.
[0134] In a possible implementation manner, determining the target image according to the character sequence may include:
[0135] The character sequence is encoded to obtain a coding information matrix; the coding information matrix is decoded to obtain a target character sequence; and the target image is determined according to the target character sequence.
[0136] This embodiment can convert the character sequence into a target character sequence of a target language by encoding and decoding the character sequence, so that the target image can be determined according to the target character sequence.
[0137] It should be understood that when the image translation model processes the input character sequence, it needs to process each character in the character sequence step by step. Therefore, a longer character sequence will cause the image translation model to perform more calculation steps, which will increase the amount of calculation and memory consumption of the image translation model, thereby affecting the translation efficiency. Therefore, in a possible implementation, the characters in the character sequence can be merged to reduce the length of the character sequence, thereby improving the calculation efficiency and translation efficiency.
[0138] That is, in a possible implementation manner, determining the target image according to the character sequence may include:
[0139] According to a preset character merging rule, the characters in the character sequence are merged to obtain a character merging sequence; the character merging sequence is encoded to obtain a coding information matrix; the coding information matrix is decoded to obtain a target character merging sequence, and the merged characters in the target character merging sequence are split into single characters to obtain a target character sequence; according to the target character sequence, the target image is determined.
[0140] The characters in the character sequence may be merged by merging two or more characters into one character. For example, the character sequence AB CD may be merged into AX D. It may also be written together with two or more separate characters. For example, the character sequence AB CD may be merged into ABC D or ABCD. The embodiments of the present disclosure do not impose any restrictions on this.
[0141] It should be understood that the merged characters are represented as one character, so the length of the character sequence can be reduced by merging characters. Figure 4 As shown in the figure, by merging characters at different times, the length of the character sequence can be effectively shortened, thereby reducing the amount of calculation of the image translation model on the character sequence, thereby improving the translation efficiency of the image translation.
[0142] In a possible implementation, merging characters in the character sequence according to a preset character merging rule to obtain a character merging sequence may include:
[0143] The character sequence is taken as the first character sequence to be processed, and the first sub-rule in the character merging rule is traversed. According to the characters included in the traversed sub-rule, the first target character and the second target character are determined in the first character sequence to be processed; the adjacent first target character and the second target character in the character sequence are merged to obtain a new character sequence, and the new character sequence is taken as the first character sequence to be processed, until all the sub-rules in the character merging rule are traversed to obtain a character merging sequence.
[0144] For example, the character sequence is: AB CDEFD, and the character merging rule includes 3 sub-rules, wherein the first sub-rule includes characters D and E, the second sub-rule includes characters A and B, and the third sub-rule includes characters C and DE. Then, according to the character merging rule, the characters in the character sequence are merged, and the obtained character merging sequence may include:
[0145] Start traversing according to the first sub-rule in the character merging rule, and merge the adjacent D and E in AB CDEFD according to the characters in the first sub-rule traversed, that is, D and E, to obtain the sequence after the first merge: AB CDE FD. Then, according to the characters in the second sub-rule traversed, that is, A and B, merge the adjacent A and B in AB C DE FD, to obtain the sequence after the second merge: AB C DE FD. Finally, according to the characters in the third sub-rule traversed, that is, C and DE, merge the adjacent C and DE in AB C DE FD, to obtain the sequence after the third merge: AB CDEF D.
[0146] It can be seen that by merging characters three times, the sequence length of the character sequence can be reduced from 7 to 4, thereby reducing the amount of calculation of the image translation model on the character sequence and improving the translation efficiency of image translation.
[0147] In a possible implementation manner, the characters included in each sub-rule in the character merging rule may be determined in the following manner:
[0148] Obtain a starting sample character sequence, take the starting sample character sequence as a second character sequence to be processed, and loop through the following process:
[0149] Determine the frequency of occurrence of the character bigram included in the second character sequence to be processed, and determine the characters in the character bigram with the highest occurrence frequency as the characters included in the first sub-rule, wherein the character bigram is used to represent two adjacent characters in the second character sequence to be processed; merge the adjacent first target character and second target character in the starting sample character sequence to obtain a new starting sample character sequence, and use the new starting character sequence as the second character sequence to be processed until a preset number of merges is reached.
[0150] The starting sample character sequence is set to one or more, and the embodiment of the present disclosure does not impose any limitation on this.
[0151] For example, the starting sample character sequence may be set to four, and the four starting sample character sequences are respectively expressed as:
[0152] AB CDEF;
[0153] BCDEFA;
[0154] CDEF AB;
[0155] DEGFAB;
[0156] First, by counting the frequency of each character bigram in the four starting sample character sequences, we can get: character bigram AB appears 3 times, character bigram BC appears 2 times, character bigram CD appears 3 times, character bigram DE appears 4 times, character bigram EF appears 3 times, character bigram FA appears 3 times, character bigram EG appears 1 time, and character bigram GF appears 1 time. Since character bigram DE has the highest frequency of occurrence, character bigram DE can be included as the character in the first sub-rule.
[0157] Then, the character tuples DE in each starting sample character sequence are merged to obtain the following four new character sequences:
[0158] AB C DE F;
[0159] BC DE FA;
[0160] C DE F AB;
[0161] DE GF AB;
[0162] After obtaining a new character sequence, the steps of counting the frequency of occurrence of each character bigram and merging the character bigrams with the highest occurrence frequency are repeated to obtain the characters included in each sub-rule.
[0163] It is worth noting that when there are multiple character bigrams with the highest frequency, any of the character bigrams with the highest frequency can be merged, or the character bigrams with the highest frequency and arranged in the front can be merged according to the arrangement order of the character bigrams in the character sequence. For example, a starting sample character sequence is: a X aa aX a X aa;
[0164] Among them, the character bigram aX appears 3 times, the character bigram Xa appears 3 times, and the character bigram aa appears 3 times. Since the character bigram aX is arranged in the front, the character bigram aX can be merged to obtain the merged starting sample character sequence: aX aa aX aX aa.
[0165] In a possible implementation manner, determining the target image according to the target character sequence may include:
[0166] For each character in the target character sequence, the character is converted into a grayscale value, and a target grayscale image is determined according to all the converted grayscale values; the target grayscale image is converted into a color image, and the color image is determined as the target image.
[0167] As mentioned above, each grayscale value in the grayscale image is a floating point number between 0 and 1, so each character in the target character sequence can be converted into a floating point number to obtain a target grayscale image. After that, each pixel value in the target grayscale image is converted into a color value, that is, an RGB value, to obtain a target image.
[0168] In a possible implementation, converting each character in the target character sequence into a grayscale value may include:
[0169] For each character in the target character sequence, the character is converted into an integer, and the product of the integer and a preset conversion coefficient is determined, and the product is determined as a grayscale value.
[0170] For example, the character can be converted into an integer by using a character-to-integer conversion function, and then the grayscale value can be obtained by calculating the product of the integer and a preset conversion coefficient. That is, the grayscale value of each character can be determined by the following formula:
[0171] v = int(c) × k2
[0172] Among them, v represents the grayscale value, int() represents the character-integer conversion function, c represents the character, and k2 represents the conversion coefficient.
[0173] The value of the conversion coefficient can be set according to actual conditions, and the embodiment of the present disclosure does not impose any restrictions on this. In a possible implementation, the value of the conversion coefficient k2 can be the same as the value of the conversion coefficient k1, for example, set to 0.1.
[0174] In summary, through the above processing, the image translation model can convert the initial image into the target image.
[0175] To further understand the disclosed image translation model, the image translation model of the present disclosure is described below.
[0176] According to the above discussion, when the image translation model determines the target image based on the initial image, it needs to first convert the initial image into a character sequence, then convert the character sequence into a target character sequence, and finally determine the target image based on the target character sequence. It can be seen that the image translation model of the present disclosure may include an image-sequence conversion structure, a sequence-sequence conversion structure, and a sequence-image conversion structure, such as Figure 5 shown.
[0177] Among them, the image-sequence conversion structure is used to convert the initial image into a character sequence. The process can be formally described as:
[0178] F ITS (x):
[0179] Among them, F ITS (x) means converting the initial image into a character sequence, x means the initial image, R means a real number, H1 means the height of the initial image, W1 means the width of the initial image, C1 means the number of channels of the initial image, x tok represents a character sequence, and T1 represents the sequence length of the character sequence.
[0180] The sequence-sequence conversion structure is used to convert a character sequence into a target character sequence. The process can be formally described as:
[0181]
[0182] Among them, y tok Represents the target character sequence, x tok represents the character sequence, θ represents the parameters of the image translation model, represents the i-th target character in the target character sequence, Represents a sequence consisting of the first i-1 target characters in the target character sequence.
[0183] The sequence-sequence conversion structure is used to convert the target character sequence into the target image. The process can be formally described as:
[0184] F STI (y tok ):
[0185] Among them, F STI (y tok ) represents the conversion of the target character sequence into the target image, y tok Represents the target character sequence, T2 represents the sequence length of the target character sequence, y represents the target image, H2 represents the height of the target image, W2 represents the width of the target image, C2 represents the number of channels of the target image, and R represents a real number.
[0186] Among them, the sequence-sequence conversion structure can be implemented based on the sequence-to-sequence model in the relevant technology, or based on the improved sequence-to-sequence model, and the embodiments of the present disclosure do not impose any restrictions on this. In possible implementations, the sequence-sequence conversion structure can be implemented based on a Transformer model, a recurrent neural network (RNN) or a convolutional neural network (CNN), and the embodiments of the present disclosure do not impose any restrictions on this.
[0187] After determining the structure of the image translation model, in order to use the image translation model to obtain the translated target image based on the input initial image, it is also necessary to iteratively train the image translation model to obtain the optimal parameters of the image translation model, so that the image translation model can obtain an accurate translation image, that is, the target image, based on any initial image input.
[0188] That is, in a possible implementation, the training process of the image translation model includes:
[0189] A sample image annotated with a label is obtained, wherein the label is used to indicate a standard image after the sample image is translated, the sample image is an image obtained by embedding a source language text into a blank image, and the standard image is an image obtained by embedding a target language text into a blank image; the sample image is input into the image translation model to obtain a predicted image, and a loss function value is determined based on the predicted image and the label; and the parameters of the image translation model are updated based on the loss function value.
[0190] It should be understood that for the image translation task, there is no publicly available training dataset in the related art. Therefore, in order to train and evaluate the image translation model provided by the present disclosure, in this embodiment, sample images for training are constructed by embedding the text in the parallel corpus into blank images, so as to train the image translation model, as Figure 6 shown.
[0191] It should be further understood that when determining the loss function value, the Smooth L1 Loss function can be used to determine the loss function value according to the predicted image and the label, or the Cross-Entropy Loss function can be used to determine the loss function value according to the predicted image and the label. The embodiments of the present disclosure do not make any restrictions on this.
[0192] Exemplarily, during the training process, the sample images are input into the image translation model to obtain an output sequence Y, Y=(y1,..., y I ), and then the output sequence Y and the character sequence X of the standard image, X=(x1,..., x J ) are brought into the cross-entropy loss function to obtain the loss function value, and the parameters of the image translation model are updated according to the loss function value to obtain a trained image translation model.
[0193] Among them, the loss function value of the character sequence X and the output sequence Y can be determined by the following formula:[[]]END]]
[0194]
[0195] Among them, L represents the loss function value, I represents the number of characters in the output sequence Y, y i represents the i-th character in the output sequence, y<i represents the sequence composed of the first i-1 characters, and θ represents the parameters of the image translation model.
[0196] Thus, through the above iterative training, a trained image translation model can be obtained, so that the target image after translation can be obtained by inputting the initial image into the image translation model.
[0197] In order to verify the translation effect of the image translation model disclosed in the present invention, this embodiment conducts a comparative experiment on the cascade method in the related art and the image translation model disclosed in the present invention based on the constructed training data set, and uses the Bilingual Evaluation Understudy (BLEU) and the Cross-lingual Optimized Metric for Evaluation of Translation (COMET) to evaluate the translation quality. In the experiment, not only the news test set (the same field as the training data set) but also the spoken language and medical test sets (different fields from the training data set) are used to evaluate the translation quality and domain adaptability. The results are shown in Table 1.
[0198] Table 1 Experimental results of the cascade method and the disclosed image translation model on different datasets
[0199]
[0200] According to the experimental results in Table 1, the translation quality of the image translation model proposed in the present disclosure is higher than that of the cascade method.
[0201] In addition, on the newstest-2013 dataset, the translation quality of the end-to-end image translation model in the related art and the image translation model of the present disclosure are compared, as shown in Table 2.
[0202] Table 2 Experimental results on the newstest-2013 dataset
[0203] system BLEU Convolutional Baseline Model 0.5 Attention Convolution Model 7.7 The Disclosure 28.1
[0204] According to the experimental results in Table 2, the translation quality of the image translation model proposed in the present disclosure is much higher than the translation quality of the end-to-end image translation model in the related art.
[0205] In addition, the end-to-end image translation model in the related art is greatly affected by the incomplete text in the image, while the image translation model of the present disclosure does not perform the text recognition process, so it has better translation quality for incomplete text in the image. In order to verify the performance of the image translation model of the present disclosure on this type of data, a test image with incomplete text is constructed, such as Figure 7 As shown in the figure. The upper picture is the original picture. By blocking the part of the dotted box, the picture with incomplete text in the figure below is obtained. Experiments are conducted on the dataset with incomplete text pictures, and the results are shown in Table 3.
[0206] Table 3 Experimental results of the cascade method and this technical solution on a dataset with incomplete text
[0207]
[0208] According to the experimental results in Table 3, the image translation model proposed in the present disclosure has higher translation quality on this type of data.
[0209] Based on the same technical concept, the present disclosure also provides an image translation device, such as Figure 8 As shown, the image translation device 800 may include:
[0210] An acquisition module 801 is configured to acquire an initial image with source language text;
[0211] The processing module 802 is configured to input the initial image into an image translation model to obtain a target image with target language text;
[0212] Wherein, the image translation model may include:
[0213] A conversion module, configured to convert the initial image into a character sequence;
[0214] The determination module is configured to determine the target image according to the character sequence.
[0215] Through the above-mentioned image translation device, the initial image with the source language text can be input into the image translation model to obtain the target image with the target language text. In this way, the image-to-image (i.e., end-to-end) image translation task can be achieved. On the other hand, when the image translation model determines the target image based on the initial image, it first converts the initial image into a character sequence, and then determines the target image based on the character sequence. Compared with the related art that determines the target image based on the cascade method, the present disclosure does not need to recognize the text in the initial image, thereby avoiding the error propagation problem caused by text recognition errors, thereby improving the translation quality of the initial image.
[0216] In a possible implementation, the conversion module may include:
[0217] A conversion submodule, configured to convert the initial image into a character image;
[0218] The sorting submodule is configured to sort the characters in the character image according to a preset character arrangement order to obtain the character sequence.
[0219] In a possible implementation, the conversion submodule may include:
[0220] A first conversion unit, configured to convert the initial image into a grayscale image;
[0221] A first determining unit is configured to determine a corresponding target character for each pixel value in the grayscale image;
[0222] The second determining unit is configured to determine the character image according to the target characters corresponding to all pixel values in the grayscale image.
[0223] In a possible implementation manner, the first determining unit may include:
[0224] A processing subunit is configured to discretize the pixel value to obtain a target discrete interval corresponding to the pixel value;
[0225] The first determination subunit is configured to determine a corresponding target character according to the target discrete interval and a preset corresponding relationship, wherein the corresponding relationship is used to characterize a corresponding relationship between the discrete interval and the character.
[0226] In a possible implementation, the determining module may include:
[0227] a merging submodule, configured to merge characters in the character sequence according to a preset character merging rule to obtain a character merging sequence;
[0228] An encoding submodule is configured to encode the character combination sequence to obtain an encoding information matrix;
[0229] A splitting submodule is configured to decode the encoding information matrix to obtain a target character merge sequence, and split the merged characters in the target character merge sequence into single characters to obtain a target character sequence;
[0230] The determination submodule is configured to determine the target image according to the target character sequence.
[0231] In a possible implementation, the merging submodule may include:
[0232] A third determining unit is configured to use the character sequence as a first character sequence to be processed, traverse from the first sub-rule in the character merging rule, and determine a first target character and a second target character in the first character sequence to be processed according to the characters included in the traversed sub-rule;
[0233] The merging unit is configured to merge the adjacent first target characters and the second target characters in the character sequence to obtain a new character sequence, and use the new character sequence as the first character sequence to be processed until all sub-rules in the character merging rule are traversed to obtain a character merging sequence.
[0234] In a possible implementation, the image translation device 800 may further include:
[0235] The character determination module is configured to obtain a starting sample character sequence, use the starting sample character sequence as a second character sequence to be processed, and cyclically execute the following process: determine the frequency of occurrence of a character bigram included in the second character sequence to be processed, and determine the characters in the character bigram with the highest occurrence frequency as the characters included in the first sub-rule, wherein the character bigram is used to represent two adjacent characters in the second character sequence to be processed; merge the first target character and the second target character that are adjacent in the starting sample character sequence to obtain a new starting sample character sequence, and use the new starting character sequence as the second character sequence to be processed until a preset number of merges is reached.
[0236] In a possible implementation, the determining submodule may include:
[0237] A second conversion unit is configured to convert each character in the target character sequence into a grayscale value;
[0238] a fourth determining unit, configured to determine a target grayscale image according to all converted grayscale values;
[0239] The fifth determining unit is configured to convert the target grayscale image into a color image and determine the color image as the target image.
[0240] In a possible implementation manner, the second conversion unit may include:
[0241] The second determining subunit is configured to convert each character in the target character sequence into an integer, determine the product of the integer and a preset conversion coefficient, and determine the product as a grayscale value.
[0242] In a possible implementation, the image translation device 801 may further include:
[0243] A training module is configured to obtain a sample image annotated with a label, wherein the label is used to indicate a standard image after the sample image is translated, the sample image is an image obtained by embedding a source language text into a blank image, and the standard image is an image obtained by embedding a target language text into a blank image; input the sample image into the image translation model to obtain a predicted image, and determine a loss function value based on the predicted image and the label; and update the parameters of the image translation model based on the loss function value.
[0244] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0245] Based on the same technical concept, the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the image translation method provided in the first aspect of the present disclosure are implemented.
[0246] Based on the same technical concept, the present disclosure also provides an electronic device, including:
[0247] processor;
[0248] a memory for storing processor-executable instructions;
[0249] Wherein, the processor is configured to:
[0250] Execute the steps of the image translation method provided in the first aspect of the present disclosure.
[0251] Reference Fig. 9 , the device 900 may include one or more of the following components: a first processing component 902 , a first memory 904 , a first power component 906 , a multimedia component 908 , an audio component 910 , a first input / output interface 912 , a sensor component 914 , and a communication component 916 .
[0252] The first processing component 902 generally controls the overall operation of the device 900, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The first processing component 902 may include one or more first processors 920 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the first processing component 902 may include one or more modules to facilitate the interaction between the first processing component 902 and other components. For example, the first processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the first processing component 902.
[0253] The first memory 904 is configured to store various types of data to support operations on the device 900. Examples of such data include instructions for any application or method operating on the device 900, contact data, phone book data, messages, pictures, videos, etc. The first memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0254] The first power source component 906 provides power to various components of the device 900. The first power source component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 900.
[0255] The multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0256] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), and when the device 900 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the first memory 904 or sent via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.
[0257] The first input / output interface 912 provides an interface between the first processing component 902 and a peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0258] The sensor assembly 914 includes one or more sensors for providing various aspects of status assessment for the device 900. For example, the sensor assembly 914 can detect the open / closed state of the device 900, the relative positioning of components, such as the display and keypad of the device 900, and the sensor assembly 914 can also detect the position change of the device 900 or a component of the device 900, the presence or absence of user contact with the device 900, the orientation or acceleration / deceleration of the device 900, and the temperature change of the device 900. The sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0259] The communication component 916 is configured to facilitate wired or wireless communication between the device 900 and other devices. The device 900 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0260] In an exemplary embodiment, the apparatus 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.
[0261] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a first memory 904 including instructions, and the instructions can be executed by a first processor 920 of the apparatus 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0262] In addition to being an independent electronic device, the above-mentioned device can also be a part of an independent electronic device. For example, in one embodiment, the device can be a chip, and the chip can include but is not limited to the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), SOC (System on Chip, SoC, system on chip or system-level chip), etc. The above-mentioned chip can be used to execute executable instructions (or codes) to implement the above-mentioned image translation method. The executable instructions can be stored in the chip or obtained from other devices or equipment. For example, the chip includes a second processor, a second memory, and an interface for communicating with other devices. The executable instructions can be stored in the second memory, and when the executable instructions are executed by the second processor, the above-mentioned image translation method can be implemented; or the chip can receive the executable instructions through the interface and transmit them to the second processor for execution to implement the above-mentioned image translation method.
[0263] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above-mentioned image translation method when executed by the programmable device.
[0264] Fig.10 1 is a block diagram of a device 1000 for image translation according to an exemplary embodiment. For example, the device 1000 may be provided as a server. Fig.10 , the apparatus 1000 includes a second processing component 1022, which further includes one or more processors, and a memory resource represented by a third memory 1032 for storing instructions, such as an application, that can be executed by the second processing component 1022. The application stored in the third memory 1032 may include one or more modules, each corresponding to a set of instructions. In addition, the second processing component 1022 is configured to execute the instructions to perform the above method.
[0265] The device 1000 may also include a second power supply component 1026 configured to perform power management of the device 1000, a wired or wireless network interface 1050 configured to connect the device 1000 to a network, and a second input / output interface 1058. The device 1000 may operate based on an operating system stored in the memory 1032, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.
[0266] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0267] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. An image translation method, characterized in that: The method comprises: Get an initial image with source language text; Inputting the initial image into an image translation model to obtain a target image with target language text; The image translation model is used to obtain the target image through the following operations: Converting the initial image into a character sequence; The target image is determined according to the character sequence.
2. The method according to claim 1, characterized in that The converting the initial image into a character sequence comprises: Converting the initial image into a character image; The characters in the character image are sorted according to a preset character arrangement order to obtain the character sequence.
3. The method according to claim 2, characterized in that The converting the initial image into a character image comprises: Converting the initial image into a grayscale image; For each pixel value in the grayscale image, determining a corresponding target character; The character image is determined according to the target characters corresponding to all pixel values in the grayscale image.
4. The method according to claim 3, characterized in that The step of determining a corresponding target character for each pixel value in the grayscale image includes: For each pixel value in the grayscale image, perform the following steps: Discretizing the pixel value to obtain a target discrete interval corresponding to the pixel value; According to the target discrete interval and a preset corresponding relationship, a corresponding target character is determined, wherein the corresponding relationship is used to characterize the corresponding relationship between the discrete interval and the character.
5. The method according to any one of claims 1 to 4, characterized in that Determining the target image according to the character sequence includes: According to a preset character merging rule, characters in the character sequence are merged to obtain a character merging sequence; Encoding the character combination sequence to obtain a coding information matrix; Decoding the encoding information matrix to obtain a target character merge sequence, and splitting the merged characters in the target character merge sequence into single characters to obtain a target character sequence; The target image is determined according to the target character sequence.
6. The method according to claim 5, characterized in that The step of merging the characters in the character sequence according to a preset character merging rule to obtain a character merging sequence includes: Taking the character sequence as a first character sequence to be processed, traversing from the first sub-rule in the character merging rule, and determining a first target character and a second target character in the first character sequence to be processed according to the characters included in the traversed sub-rule; The first target character and the second target character adjacent to each other in the character sequence are merged to obtain a new character sequence, and the new character sequence is used as the first character sequence to be processed, until all sub-rules in the character merging rule are traversed to obtain a character merging sequence.
7. The method according to claim 6, characterized in that The characters included in each sub-rule in the character merging rule are determined in the following manner: Obtain a starting sample character sequence, take the starting sample character sequence as a second character sequence to be processed, and loop through the following process: Determine the frequency of occurrence of the character bigram included in the second character sequence to be processed, and determine the characters in the character bigram with the highest occurrence frequency as the characters included in the first sub-rule, wherein the character bigram is used to represent two adjacent characters in the second character sequence to be processed; The adjacent first target characters and second target characters in the starting sample character sequence are merged to obtain a new starting sample character sequence, and the new starting character sequence is used as a second character sequence to be processed until a preset number of merging times is reached.
8. The method according to claim 5, characterized in that Determining the target image according to the target character sequence includes: For each character in the target character sequence, convert the character into a grayscale value; According to all the converted grayscale values, determine the target grayscale image; The target grayscale image is converted into a color image, and the color image is determined as the target image.
9. The method according to claim 8, characterized in that The step of converting each character in the target character sequence into a grayscale value includes: For each character in the target character sequence, the character is converted into an integer, and the product of the integer and a preset conversion coefficient is determined, and the product is determined as a grayscale value.
10. The method according to any one of claims 1 to 4, characterized in that The training process of the image translation model includes: Acquire a sample image annotated with a label, wherein the label is used to indicate a standard image after translating the sample image, the sample image is an image obtained by embedding a source language text into a blank image, and the standard image is an image obtained by embedding a target language text into a blank image; Inputting the sample image into the image translation model to obtain a predicted image, and determining a loss function value according to the predicted image and the label; The parameters of the image translation model are updated according to the loss function value.
11. An image translation device, characterized in that: The device comprises: An acquisition module, configured to acquire an initial image with source language text; A processing module, configured to input the initial image into an image translation model to obtain a target image with target language text; Wherein, the image translation model includes: A conversion module, configured to convert the initial image into a character sequence; The determination module is configured to determine the target image according to the character sequence.
12. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Execute the steps of the method according to any one of claims 1 to 10.
13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 10 are implemented.