Text fragmentation method, text translation method, text fragmentation device, text translation device and electronic equipment

Abstract

The embodiment of the invention provides a text fragmentation method and device, a text translation method and device and electronic equipment, and the method comprises the steps: determining a source text, generating a target fragmentation strategy for the source text through a genetic algorithm according to the source text and a preset corpus model, the corpus model is used for evaluating the fitness of the fragment position information according to the semantic scene information and the length limitation information; the semantic scene information comprises subject information to which the source text belongs and use area information of the source text; the length limitation information represents the maximum character length which can be received by the translation model; and fragmenting the source text based on the fragmentation position information contained in the target fragmentation strategy. Therefore, the technical problem that the translation result is affected due to unreasonable fragmentation in traditional machine translation is solved, the to-be-translated source text is reasonably fragmented through the capability of obtaining the globally optimal solution through the genetic algorithm, and thus the translation accuracy and quality are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text segmentation method, a text translation method, a device and an electronic device. Background Art

[0002] With the rapid development of economic globalization, the exchanges between peoples of all countries are becoming more and more frequent, and the demand for language translation between countries is also increasing. According to statistics, there are more than 7,000 languages ​​in the world, and humans need to invest a lot of time to learn each language. Moreover, a large number of documents, materials, books and other contents rely on human translation, which is not only inefficient, but also consumes a lot of human energy. With the advent of the third information technology industrial revolution, computer network technology has developed rapidly. By combining powerful algorithms with a large amount of training data, machine translation was born. Machine translation can quickly process a large amount of text data and eliminate some human errors in the translation process. However, due to differences in human culture, context, customary background and subject terminology, the accuracy of machine translation is greatly limited, and the quality of machine translation in many aspects cannot meet expectations.

[0003] At present, the field of AI (Artificial Intelligence) has entered a new stage of rapid development. At the same time, applying AI to the field of translation is another major technological breakthrough compared to machine translation. It is not only faster and involves more languages, but the AI ​​translation system can continuously improve the quality and accuracy of translation through large amounts of translation data training and continuous learning.

[0004] However, the current AI translation model has a limitation on the length of text data. Therefore, before inputting the source text into the translation model, the source text needs to be segmented to meet the limitation on the length of the text data.

[0005] The current AI translation solution mainly obtains the source text data to be translated, segments the source text using a fixed length according to the length of data that the model can receive, passes the segmented data to the AI ​​model in sequence, processes the translation content, and returns the target language for translation.

[0006] Using fixed-length segments, or segmenting without taking into account language scenarios, contexts, customs and culture, will reduce translation quality due to unreasonable segmentation. Summary of the invention

[0007] The purpose of the embodiments of the present application is to provide a text segmentation method, a text translation method, a device and an electronic device, which can reasonably segment the source text to be translated by using the ability of a genetic algorithm to obtain a global optimal solution, thereby improving the accuracy and quality of the translation. The specific technical solution is as follows:

[0008] The present application provides a text segmentation method, the method comprising:

[0009] Determine a source text, and generate a target segmentation strategy for the source text using a genetic algorithm based on the source text and a preset corpus model, wherein the target segmentation strategy for the source text includes segmentation position information; the corpus model is used to evaluate the fitness of the segmentation position information based on semantic scene information and length limit information; the semantic scene information includes: subject information to which the source text belongs and area of ​​use information of the source text; the length limit information represents the maximum character length that can be accepted by the translation model;

[0010] The source text is segmented based on the segmentation location information included in the target segmentation strategy.

[0011] Optionally, the step of generating a target segmentation strategy for the source text using a genetic algorithm according to the source text and a preset corpus model includes:

[0012] Obtain an initial population, the population including multiple individuals, each individual representing a sharding strategy, the sharding strategy including sharding location information, semantic scene information and length limit information, and the sharding strategy represented by each individual is randomly selected from a preset sharding strategy set;

[0013] The method comprises the steps of calculating the fitness of each individual according to the sharding strategy represented by each individual in the population, the source text, and the corpus model; screening the individuals in the population according to the fitness of each individual to obtain target individuals; performing cross-breeding and mutation using the target individuals to obtain the next round of population; in response to detecting that the termination condition of the genetic algorithm is not currently met, returning to the step of calculating the fitness of each individual according to the sharding strategy represented by each individual in the population, the source text, and the corpus model for the next round of population; and in response to detecting that the termination condition of the genetic algorithm is currently met, determining the sharding strategy obtained when the termination condition of the genetic algorithm is met as the target sharding strategy.

[0014] Optionally, the calculating the fitness of each individual according to the sharding strategy represented by each individual in the population, the source text, and the corpus model includes:

[0015] Inputting the segmentation strategy represented by each of the individuals and the segmentation source text obtained under the segmentation strategy into the corpus model, so that the corpus model generates a predicted text based on the segmentation source text under the semantic scene information contained in the segmentation strategy, and outputs a cross entropy calculated based on the segmentation source text and the predicted text, and a first length difference between the segmentation source text and the predicted text;

[0016] Calculate basic fitness according to the cross entropy, the first length difference, and the first length of the segment source text; the basic fitness is negatively correlated with the cross entropy and the first length difference; the basic fitness is positively correlated with the first length;

[0017] Based on the preset first weighted weight, second weighted weight and third weighted weight, the basic fitness is weightedly calculated to obtain the fitness of the individual; the first weighted weight is used to characterize the influence of the subject information on the fitness, the second weighted weight is used to characterize the influence of the usage area information on the fitness, and the third weighted weight is used to characterize the influence of the length limit information on the fitness.

[0018] Optionally, after calculating the basic fitness according to the cross entropy, the length difference, and the first length of the segment source text, the method further includes:

[0019] A length influencing factor is superimposed on the basic fitness, wherein the length influencing factor is negatively correlated with a second length difference; the second length difference is a difference between a maximum character length acceptable to the translation model and the first length;

[0020] The step of performing a weighted operation on the basic fitness based on the preset first weighted weight, the second weighted weight and the third weighted weight to obtain the fitness of the individual comprises:

[0021] The basic fitness is weighted based on the first weighted weight and the second weighted weight, and the basic fitness after the length influencing factor is superimposed is weighted based on the third weighted weight to obtain the fitness of the individual.

[0022] The present application also provides a text translation method, which includes: inputting a segmented source text into a translation model to obtain a translation result, wherein the source text is segmented according to any of the above-mentioned text segmentation methods.

[0023] The present application also provides a text segmentation device, the device comprising:

[0024] A generation module is used to determine a source text, and generate a target segmentation strategy for the source text using a genetic algorithm based on the source text and a preset corpus model, wherein the target segmentation strategy for the source text includes segmentation position information; the corpus model is used to evaluate the fitness of the segmentation position information based on semantic scene information and length limit information; the semantic scene information includes: subject information to which the source text belongs and area of ​​use information of the source text; the length limit information represents the maximum character length that can be accepted by the translation model;

[0025] The fragmentation module is used to fragment the source text based on the fragmentation position information included in the target fragmentation strategy.

[0026] Optionally, the generating module includes:

[0027] An acquisition submodule is used to acquire an initial population, wherein the population includes multiple individuals, each individual represents a sharding strategy, and the sharding strategy includes sharding location information, semantic scene information, and length limit information. The sharding strategy represented by each individual is randomly selected from a preset sharding strategy set;

[0028] A calculation submodule, used for calculating the fitness of each individual according to the sharding strategy represented by each individual in the population, the source text, and the corpus model;

[0029] A screening submodule, used for screening the individuals in the population according to the fitness of each individual to obtain a target individual;

[0030] A mutation submodule, used to use the target individuals to perform cross-breeding and mutation to obtain the next round of population;

[0031] A return submodule, for returning to the step of calculating the fitness of each individual according to the sharding strategy represented by each individual, the source text, and the corpus model for the next round of population in response to detecting that the termination condition of the genetic algorithm is not currently met;

[0032] The determination submodule is used to determine, in response to the termination condition of the genetic algorithm being met, the sharding strategy obtained when the termination condition of the genetic algorithm is met as the target sharding strategy.

[0033] Optionally, the computing submodule is specifically used for:

[0034] Inputting the segmentation strategy represented by each of the individuals and the segmentation source text obtained under the segmentation strategy into the corpus model, so that the corpus model generates a predicted text based on the segmentation source text under the semantic scene information contained in the segmentation strategy, and outputs a cross entropy calculated based on the segmentation source text and the predicted text, and a first length difference between the segmentation source text and the predicted text;

[0035] Calculate basic fitness according to the cross entropy, the first length difference, and the first length of the segment source text; the basic fitness is negatively correlated with the cross entropy and the first length difference; the basic fitness is positively correlated with the first length;

[0036] Based on the preset first weighted weight, second weighted weight and third weighted weight, the basic fitness is weightedly calculated to obtain the fitness of the individual; the first weighted weight is used to characterize the influence of the subject information on the fitness, the second weighted weight is used to characterize the influence of the usage area information on the fitness, and the third weighted weight is used to characterize the influence of the length limit information on the fitness.

[0037] The present application also provides a text translation device, which includes: a translation module, which is used to input the segmented source text into a translation model to obtain a translation result, wherein the source text is segmented according to any of the above-mentioned text segmentation methods.

[0038] The present application also provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0039] Memory, used to store computer programs;

[0040] The processor is used to implement any of the above-mentioned text segmentation methods or text translation methods when executing the program stored in the memory.

[0041] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned text segmentation methods or text translation methods is implemented.

[0042] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned text segmentation methods or text translation methods.

[0043] Beneficial effects of the embodiments of the present application:

[0044] The text segmentation method, text translation method, device and electronic device provided in the embodiments of the present application are applied to determine the source text, and generate a target segmentation strategy for the source text using a genetic algorithm based on the source text and a preset corpus model, wherein the target segmentation strategy for the source text includes segmentation position information; the corpus model is used to evaluate the fitness of the segmentation position information based on semantic scene information and length limit information; the semantic scene information includes: subject information to which the source text belongs and area of ​​use information of the source text; the length limit information represents the maximum character length that the translation model can accept; and the source text is segmented based on the segmentation position information contained in the target segmentation strategy.

[0045] It can be seen that in the embodiment of the present application, in the process of slicing the source text, the slicing strategy is set including the slicing position information, and the slicing position is evaluated under specific semantic scene information and length limit information in combination with the evaluation capability of the corpus model. The slicing position is optimized by using a genetic algorithm. Since the specific discipline, the use area or country of the source text and other information are defined, the influence of the discipline scene, local customs and culture on the source text slicing is fully considered, the rationality of the source text slicing is improved, and the technical problem of using fixed-length slicing or slicing without considering the language scene, context and customs and culture factors, which reduces the translation quality due to unreasonable slicing, is solved.

[0046] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0048] Figure 1 A flowchart of a text segmentation method provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of a process for optimizing a sharding strategy using a genetic algorithm provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of a flow chart for calculating fitness provided in an embodiment of the present application;

[0051] Figure 4 A structural schematic diagram of a text segmentation device provided in an embodiment of the present application;

[0052] Figure 5A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0054] The current AI translation solution mainly obtains the source text data to be translated, segments the source text using a fixed length according to the length of data that the model can receive, passes the segmented data to the AI ​​model in sequence, processes the translation content, and returns the target language for translation.

[0055] Using fixed-length segments, or segmenting without taking into account language scenarios, contexts, customs and culture, will reduce translation quality due to unreasonable segmentation.

[0056] In order to solve the above technical problems, the embodiments of the present application provide a text segmentation method, device, electronic device and storage medium.

[0057] For details, see Figure 1 , is a flow chart of a text segmentation method provided in an embodiment of the present application, comprising the following steps:

[0058] S101: Determine a source text, and generate a target segmentation strategy for the source text using a genetic algorithm based on the source text and a preset corpus model; wherein the target segmentation strategy for the source text includes segmentation position information; the corpus model is used to evaluate the fitness of the segmentation position information based on semantic scene information and length limit information; the semantic scene information includes: subject information to which the source text belongs and area of ​​use information of the source text; the length limit information represents the maximum character length that the translation model can accept.

[0059] The source text is the text to be translated. Since the current AI translation model has a limit on the number of characters that can be input into the model, the source text needs to be segmented before it is input into the AI ​​translation model. Segmentation can be understood as dividing the source text into shorter texts based on punctuation marks to meet the character limit of the translation model.

[0060] In an embodiment of the present application, the fragmentation position information represents the fragmentation position obtained by fragmenting the source text. As an example, the maximum character length that the AI ​​translation model can receive is 700, and the character length of the source text is 1000. By locating punctuation marks, the source text is divided into a text of 499 characters and a text of 501 characters, and each part of the text meets the character restrictions of the AI ​​translation model. For another example, by locating punctuation marks, the source text is divided into a text of 600 characters and a text of 400 characters.

[0061] In the embodiment of the present application, the corpus model is a model constructed by collecting a large amount of text data, and has the function of predicting a complete text based on a partial text. In addition, the training of the corpus model refers to information such as customs and culture in different contexts, different regions or countries. Therefore, the same text is processed in different contexts, and the results obtained may also be different. The embodiment of the present application does not limit the selection of the corpus model. For example, existing corpus models include: the National Language Committee Modern Chinese General Balanced Corpus, the BOE (the Bank of English) corpus, the BNC (British National Corpus) corpus, etc.

[0062] In the embodiment of the present application, the subject information of the source text and the usage area information of the source text are defined in advance and encoded respectively. For example, the basic information of the subject scene to which the source text belongs is D = {d1, d2…d l}, which may include "geography", "chemistry", "biology", "history" and other subjects, and l is the total number of defined subject scenarios. The usage area information of the source text is W = {w1,w2…w m The usage area can be a specific country or region, and m is the total number of defined regions or countries. The usage area information represents the country or region where the source text is used. Specifically, for the same language, there may be multiple countries or regions using the language. Although different countries or regions use the same language, the usage habits, customs and cultures of different places may be different. Therefore, in the embodiment of the present application, the usage area information of the source text is used as a consideration for text segmentation. The maximum character length that the translation model can accept is P = {p1, p2…p b}, for example, the maximum length characters are defined as 700, 500, 800, etc., and l is the total number of the maximum length characters defined. In the embodiment of the present application, the above-defined disciplines, usage area information and maximum length characters are encoded in advance and represented by a specific number of binary bits.

[0063] In an embodiment of the present application, a genetic algorithm is used to calculate a target segmentation strategy of a source text, wherein each segmentation strategy includes segmentation position information, semantic scene information, and length limit information.

[0064] It should be noted that the genetic algorithm is a computational model of biological evolution that simulates the natural selection and genetics mechanism of Darwin's theory of biological evolution. It is a method of searching for the optimal solution by simulating the natural evolution process. The algorithm uses mathematical methods and computer simulation operations to convert the problem-solving process into a process similar to the crossover and mutation of chromosome genes in biological evolution. It draws on the mechanisms of genetic evolution in biology, such as mutation, crossover, natural selection, and survival of the fittest, to optimize a set of iterative candidate solutions to the target.

[0065] In the process of iterative operations based on genetic algorithms, the fitness of the fragmentation strategy can be calculated according to the output of the corpus model, and the fragmentation strategies with higher fitness can be continuously screened to finally determine the target fragmentation strategy. Among them, the corpus model has the ability to predict the complete text, that is, to perform predictive completion operations based on the input fragmentation source text, and then evaluate whether the fragmentation is reasonable based on the completed text, so as to output the result representing the rationality of the fragmentation position. In addition, it should be noted that since the semantic scene information and length limit information are defined, the corpus model will also combine specific disciplines, the use area or country of the source text, etc. when evaluating, so as to fully consider the impact of the discipline scene and local customs and culture.

[0066] S102: Segment the source text based on the segmentation location information included in the target segmentation strategy.

[0067] Since the target sharding strategy contains sharding location information, sharding can be performed based on the sharding location.

[0068] It can be seen that in the embodiment of the present application, in the process of slicing the source text, the slicing strategy is set including the slicing position information, and the slicing position is evaluated under specific semantic scene information and length limit information in combination with the evaluation capability of the corpus model. The slicing position is optimized by using a genetic algorithm. Since the specific discipline, the use area or country of the source text and other information are defined, the influence of the discipline scene, local customs and culture on the source text slicing is fully considered, the rationality of the source text slicing is improved, and the technical problem of using fixed-length slicing or slicing without considering the language scene, context and customs and culture factors, which reduces the translation quality due to unreasonable slicing, is solved.

[0069] In one embodiment of the present application, the steps of using a genetic algorithm to generate a target segmentation strategy for the source text according to the source text and a preset corpus model can be found in Figure 2 , Figure 2 A schematic diagram of a flow chart of optimizing the sharding strategy using a genetic algorithm provided in an embodiment of the present application specifically includes the following steps:

[0070] S201: Obtain an initial population, the population includes multiple individuals, each individual represents a sharding strategy, the sharding strategy includes sharding location information, semantic scene information and length limit information, and the sharding strategy represented by each individual is randomly selected from a preset sharding strategy set.

[0071] In the embodiment of the present disclosure, each individual represents a segmentation strategy, and each segmentation strategy includes specific segmentation location information, subject information, source text usage area information, and a maximum character length acceptable to the translation model.

[0072] In order to ensure that the sharding strategy represented by each individual in the initial population does not deviate from the actual application scenario, the sharding strategy represented by each individual can be determined within a preset range. The preset range can refer to a predefined range, that is, the set D defined above = {d1, d2…d l}、W={w1,w2…w m}、P={p1,p2…p b}.

[0073] At the same time, each individual in the population has its own set of chromosomes, and chromosomes are composed of genes. Specifically in this scheme, the phenotype corresponding to the chromosomes of each individual in the population is the fragmentation strategy it represents.

[0074] S202: Calculate the fitness of each individual according to the sharding strategy, source text, and corpus model represented by each individual in the population.

[0075] In the embodiment of the present application, when calculating the fitness of an individual, it is necessary to input the fragmentation strategy and source text represented by the individual into the corpus model, and calculate the fitness through the evaluation results output by the corpus model. Figure 3 The steps shown are not described in detail here.

[0076] S203: Screen the individuals in the population according to the fitness of each individual to obtain the target individual.

[0077] Specifically, according to the fitness of each individual, it is judged whether the fitness of each individual meets the preset conditions, so that the individuals meeting the preset conditions are taken as target individuals.

[0078] As an example, if the preset individual screening condition is the two individuals with the highest fitness, the individuals in the population can be sorted from high to low according to fitness, and the first two individuals in the sorting result are taken as target individuals.

[0079] As another example, in order to improve the diversity of individuals in the population, the probability of selecting the corresponding individual can also be set according to the fitness of each individual. The individual with higher fitness has a greater probability of being selected as the target individual.

[0080] S204: Use the target individuals for cross-breeding and mutation to obtain the next round of population.

[0081] Crossbreeding refers to the method of cross-transforming the chromosomes of the target individual to produce a new individual. Usually, cross-breeding can include either single-point crossover or multi-point crossover.

[0082] Specifically in this solution, cross breeding is to cross the subject information, the usage area information of the source text or the length limit information of the translation model in each of the two individuals to obtain a new individual.

[0083] In biology, during the growth of offspring, the genes in the body will change, making them different from the parents. This process is called mutation. Therefore, we can randomly select an individual from the individuals obtained by cross breeding, and randomly change the genes of the chromosomes of the selected individual with a preset probability to obtain a new individual.

[0084] Specifically in this solution, the variation refers to: making random changes to the sharding location information, subject information, source text usage area information or translation model length limit information contained in the sharding strategy within their respective preset limits.

[0085] S205: Determine whether the termination condition of the genetic algorithm is met.

[0086] In the embodiment of the present application, the termination condition of the genetic algorithm is usually that the population generation reaches a preset number of rounds. If the termination condition is met, step S206 is executed, otherwise, for the next round of population, step S202 is returned to execute. It should be noted that the termination condition is not specifically limited in the embodiment of the present application. For example, the termination condition can also be: the difference in fitness of the best individuals in multiple consecutive generations of populations is within a preset range.

[0087] S206: Determine the sharding strategy obtained when the termination condition of the genetic algorithm is met as the target sharding strategy.

[0088] Specifically, if the termination condition of the genetic algorithm is met, the sharding strategy represented by the individual with higher fitness in the population at this time can be determined as the target sharding strategy.

[0089] As an example, individuals in the population obtained when the termination condition of the genetic algorithm is met are sorted from high to low according to fitness, and a preset number of individuals are taken out in sequence, and the sharding strategy represented by the taken individuals is used as the target sharding strategy.

[0090] It can be seen that in the embodiment of the present application, in the process of determining the target fragmentation strategy by using a genetic algorithm, the fitness of each individual is calculated according to the fragmentation strategy, source text, and corpus model represented by each individual; thus, the individuals in the population can be screened based on the fitness of each individual to obtain the next round of populations, that is, the purpose of optimizing the fragmentation strategy can be achieved based on the way of calculating the fitness of each individual, and finally a reasonable fragmentation position can be determined. This solves the technical problem of unreasonable fragmentation in machine translation that affects the translation results. The ability of the genetic algorithm to obtain the global optimal solution and reasonably fragment the source text to be translated can improve the accuracy and quality of the translation.

[0091] See also Figure 3 , is a schematic diagram of a flow chart of calculating fitness provided in an embodiment of the present application, which specifically includes the following steps:

[0092] S301: The segmentation strategy represented by each individual and the segmentation source text obtained under the segmentation strategy are input into the corpus model, so that the corpus model generates a predicted text based on the segmentation source text under the semantic scene information contained in the segmentation strategy, and outputs the cross entropy obtained based on the operation of the segmentation source text and the predicted text, and the first length difference between the segmentation source text and the predicted text.

[0093] Specifically, the corpus model in this application is a tool that can predict the complete text based on partial text. In addition, the training of the corpus model refers to information such as customs and culture of different contexts, regions or countries. Therefore, the results obtained from processing the same text in different contexts may be different.

[0094] For example, for the same source text, the results obtained by the corpus model when processing the source text in the geography subject scenario and in the chemistry subject scenario may be different.

[0095] In an embodiment of the present application, for a specific segmentation strategy, the segmentation source text obtained based on the segmentation strategy, the subject scenario to which the text belongs, the usage area information of the text, and the length limit information of the translation model are input into the corpus model. The corpus model will perform a predictive completion operation on the segmentation source text, and the obtained text is recorded as the predicted text.

[0096] The corpus model can calculate the cross entropy similarity, referred to as cross entropy, based on the input segmented source text and the predicted text generated by the model. Cross entropy similarity has a wide range of applications in tasks such as information retrieval and text clustering. Cross entropy similarity can be used to measure the similarity between texts, and can be specifically calculated based on the vector of the text. The embodiment of this application does not elaborate on the calculation of cross entropy.

[0097] In addition, the length difference between the segmented source text and the predicted text generated by the model is calculated and recorded as the first length difference.

[0098] S302: Calculate basic fitness according to the cross entropy, the first length difference, and the first length of the segmented source text; the basic fitness is negatively correlated with the cross entropy and the first length difference; the basic fitness is positively correlated with the first length.

[0099] In the embodiment of the present application, the larger the cross entropy and the larger the first length difference, the worse the evaluation result of the corpus model on the fragment source text, that is, the worse the rationality of the fragment position. In addition, if the length of the text obtained by fragmentation is too small, it is obviously not conducive to subsequent translation. Therefore, the first length of the fragment source text is also used to characterize the rationality of the fragmentation.

[0100] Based on the above considerations, the basic fitness is calculated according to the cross entropy, the first length difference, and the first length of the segment source text. The basic fitness is negatively correlated with the cross entropy and the first length difference; the basic fitness is positively correlated with the first length.

[0101] As an example, the basic fitness is calculated using the following formula:

[0102]

[0103] Among them, f i0 represents the basic fitness of the i-th individual, L i Indicates the first length of the fragment source text, H i represents the cross entropy, Δl i Indicates the first length difference.

[0104] S303: Based on the preset first weighted weight, second weighted weight and third weighted weight, a weighted operation is performed on the basic fitness to obtain the individual fitness; the first weighted weight is used to characterize the influence of subject information on fitness, the second weighted weight is used to characterize the influence of use area information on fitness, and the third weighted weight is used to characterize the influence of length limit information on fitness.

[0105] In the embodiment of the present application, weighted weights are set in advance for different dimensions, and weighted operations are performed on the basis of the basic fitness to obtain the weighted fitness. The weighted weights of each dimension can be set according to needs.

[0106] It can be seen that when calculating the fitness for a single individual, the length difference and cross entropy between the segmented source text and the predicted text are used as the basis for fitness calculation by using the corpus module's ability to predict text and evaluate text cross entropy. The basic fitness is weighted to fully reflect the influence of factors such as subject information, usage area information, and length limit information.

[0107] In one embodiment of the present application, after calculating the basic fitness according to the cross entropy, the length difference, and the first length of the segmented source text, it also includes:

[0108] The length influencing factor is superimposed on the basic fitness, and the length influencing factor is negatively correlated with the second length difference; the second length difference is the difference between the maximum character length acceptable to the translation model and the first length;

[0109] Accordingly, based on the preset first weighted weight, the second weighted weight and the third weighted weight, the basic fitness is weighted to obtain the fitness of the individual, including:

[0110] The basic fitness is weighted based on the first weighted weight and the second weighted weight, and the basic fitness after the length influence factor is superimposed is weighted based on the third weighted weight to obtain the fitness of the individual.

[0111] Specifically, if only cross entropy and the first length difference are used to evaluate fitness, it is easy to fall into the local optimum and it is not easy to calculate the global optimum. Therefore, in order to further reflect the role of the segment position in calculating fitness and improve the convergence speed, the length influence factor is superimposed on the basic fitness. The length influence factor is negatively correlated with the second length difference. The second length is the difference between the maximum character length acceptable to the translation model and the first length. The smaller the difference, the closer the source text segment length is to the acceptable length of the model, and the better the position is, which is conducive to subsequent translation. Therefore, the length influence factor is negatively correlated with the second difference.

[0112] As an example, the fitness is calculated using the following formula:

[0113] f i =k1×f i0 +k2×f i0 +k3×(f i0 +Δm i )

[0114] Among them, f i is the fitness of the ith individual, f i0 is the basic fitness of the ith individual, k1, k2, k3 represent the first weighted weight, the second weighted weight and the third weighted weight respectively. i is the length influence factor, and its value can be P r The maximum length of characters that the translation model can accept.

[0115] By superimposing the length influencing factor, the importance of the shard position in calculating the fitness can be increased, which is conducive to ensuring that the shard position is the factor with the greatest influence on the fitness, and will not have a significant impact on the fitness due to changes in the subject scenario or the area of ​​use information. It is also conducive to reducing the number of iterations, improving the calculation speed, and obtaining the target individual faster.

[0116] In the embodiments of the present application, multimodal information that affects the translation quality, such as the subject scenario to which the source text belongs, the area of ​​use of the source text, and the maximum character length that the translation model can accept, is effectively integrated together, so that the corpus model can evaluate the rationality of the segmentation position based on the above multimodal information, and fully consider the impact of the subject scenario and local customs and culture, so that the evaluation of the segmentation position is more reasonable. And the genetic algorithm is continuously iterated to finally locate the best position for segmenting the source text, which makes up for the shortcomings of the AI ​​translation model. It solves the technical problem of using fixed-length segments, or segmenting without considering the language scenario, context, customs and culture, and reducing the translation quality due to unreasonable segmentation.

[0117] The embodiment of the present application also provides a text translation method, which includes: inputting the segmented source text into a translation model to obtain a translation result. The segmentation of the source text is implemented based on the text segmentation method provided in the embodiment of the present application.

[0118] Since the above multimodal information is fully considered in the fragmentation process and reasonable fragmentation positions are obtained, the accuracy and quality of the translation are improved.

[0119] Based on the same inventive concept, the present application embodiment also provides a text segmentation device, see Figure 4 ,include:

[0120] The generation module 401 is used to determine a source text, and generate a target segmentation strategy for the source text using a genetic algorithm based on the source text and a preset corpus model, wherein the target segmentation strategy for the source text includes segmentation position information; the corpus model is used to evaluate the fitness of the segmentation position information based on semantic scene information and length limit information; the semantic scene information includes: subject information to which the source text belongs and area of ​​use information of the source text; the length limit information represents the maximum character length that the translation model can accept;

[0121] The fragmentation module 402 is used to fragment the source text based on the fragmentation position information included in the target fragmentation strategy.

[0122] It can be seen that in the embodiment of the present application, in the process of slicing the source text, the slicing strategy is set including the slicing position information, and the slicing position is evaluated under specific semantic scene information and length limit information in combination with the evaluation capability of the corpus model. The slicing position is optimized by using a genetic algorithm. Since the specific discipline, the use area or country of the source text and other information are defined, the influence of the discipline scene, local customs and culture on the source text slicing is fully considered, the rationality of the source text slicing is improved, and the technical problem of using fixed-length slicing or slicing without considering the language scene, context and customs and culture factors, which reduces the translation quality due to unreasonable slicing, is solved.

[0123] In one embodiment of the present application, the generation module includes:

[0124] An acquisition submodule is used to acquire an initial population, wherein the population includes multiple individuals, each individual represents a sharding strategy, and the sharding strategy includes sharding location information, semantic scene information, and length limit information. The sharding strategy represented by each individual is randomly selected from a preset sharding strategy set;

[0125] The calculation submodule is used to calculate the fitness of each individual according to the sharding strategy, source text, and corpus model represented by each individual;

[0126] The screening submodule is used to screen the individuals in the population according to the fitness of each individual to obtain the target individual;

[0127] The mutation submodule is used to use the target individuals for cross-breeding and mutation to obtain the next round of population;

[0128] A return submodule, for returning to execute the step of calculating the fitness of each individual according to the sharding strategy, source text, and corpus model represented by each individual in response to the termination condition of the genetic algorithm not being met;

[0129] The determination submodule is used to determine, in response to the termination condition of the genetic algorithm being met, the sharding strategy obtained when the termination condition of the genetic algorithm is met as the target sharding strategy.

[0130] It can be seen that in the embodiment of the present application, in the process of determining the target fragmentation strategy by using a genetic algorithm, the fitness of each individual is calculated according to the fragmentation strategy, source text, and corpus model represented by each individual; thus, the individuals in the population can be screened based on the fitness of each individual to obtain the next round of populations, that is, the purpose of optimizing the fragmentation strategy can be achieved based on the way of calculating the fitness of each individual, and finally a reasonable fragmentation position can be determined. This solves the technical problem of unreasonable fragmentation in machine translation that affects the translation results. The ability of the genetic algorithm to obtain the global optimal solution and reasonably fragment the source text to be translated can improve the accuracy and quality of the translation.

[0131] In one embodiment of the present application, the calculation submodule is specifically used for:

[0132] Input the fragmentation strategy represented by each individual and the fragmentation source text obtained under the fragmentation strategy into the corpus model, so that the corpus model generates a predicted text based on the fragmentation source text under the semantic scene information contained in the fragmentation strategy, and outputs a cross entropy obtained by calculating the fragmentation source text and the predicted text, and a first length difference between the fragmentation source text and the predicted text;

[0133] The basic fitness is calculated according to the cross entropy, the first length difference, and the first length of the segment source text; the basic fitness is negatively correlated with the cross entropy and the first length difference; the basic fitness is positively correlated with the first length;

[0134] Based on the preset first weighted weight, second weighted weight and third weighted weight, the basic fitness is weightedly calculated to obtain the individual fitness; the first weighted weight is used to characterize the influence of subject information on fitness, the second weighted weight is used to characterize the influence of use area information on fitness, and the third weighted weight is used to characterize the influence of length limit information on fitness.

[0135] It can be seen that when calculating the fitness for a single individual, the length difference and cross entropy between the segmented source text and the predicted text are used as the basis for fitness calculation by using the corpus module's ability to predict text and evaluate text cross entropy. The basic fitness is weighted to fully reflect the influence of factors such as subject information, usage area information, and length limit information.

[0136] In one embodiment of the present application, it also includes:

[0137] A superposition submodule, used for superimposing a length influence factor on the basis of the basic fitness, wherein the length influence factor is negatively correlated with a second length difference; the second length difference is a difference between a maximum character length acceptable to the translation model and the first length;

[0138] The computing submodule is specifically used for:

[0139] The basic fitness is weighted based on the first weighted weight and the second weighted weight, and the basic fitness after the length influence factor is superimposed is weighted based on the third weighted weight to obtain the fitness of the individual.

[0140] By superimposing the length influencing factor, the importance of the shard position in calculating the fitness can be increased, which is conducive to ensuring that the shard position is the factor with the greatest influence on the fitness, and will not have a significant impact on the fitness due to changes in the subject scenario or the area of ​​use information. It is also conducive to reducing the number of iterations, improving the calculation speed, and obtaining the target individual faster.

[0141] The present application also provides an electronic device, such as Figure 5 As shown, it includes a processor 501 , a communication interface 502 , a memory 503 and a communication bus 504 , wherein the processor 501 , the communication interface 502 , and the memory 503 communicate with each other via the communication bus 504 .

[0142] Memory 503, used for storing computer programs;

[0143] The processor 501 is used to execute the program stored in the memory 503, and implements the following steps:

[0144] Determine the source text, and generate a target segmentation strategy for the source text using a genetic algorithm based on the source text and a preset corpus model, wherein the target segmentation strategy for the source text includes segmentation position information, semantic scene information, and length limit information; the semantic scene information includes: subject information to which the source text belongs and area of ​​use of the source text; the length limit information represents the maximum character length that the translation model can accept;

[0145] The source text is fragmented based on the fragmentation location information contained in the target fragmentation strategy.

[0146] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0147] The communication interface is used for communication between the above electronic device and other devices.

[0148] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0149] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0150] In another embodiment provided by the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned text segmentation methods and text translation methods are implemented.

[0151] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of any of the text segmentation methods and text translation methods in the above embodiments.

[0152] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0153] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0154] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0155] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A text segmentation method, characterized in that: The method comprises: Determine a source text, and generate a target segmentation strategy for the source text using a genetic algorithm based on the source text and a preset corpus model, wherein the target segmentation strategy for the source text includes segmentation position information; the corpus model is used to evaluate the fitness of the segmentation position information based on semantic scene information and length limit information; the semantic scene information includes: subject information to which the source text belongs and area of ​​use information of the source text; the length limit information represents the maximum character length that can be accepted by the translation model; The source text is segmented based on the segmentation location information included in the target segmentation strategy.

2. The method according to claim 1, characterized in that The step of generating a target segmentation strategy for the source text using a genetic algorithm based on the source text and a preset corpus model comprises: Obtain an initial population, the population including multiple individuals, each individual representing a sharding strategy, the sharding strategy including sharding location information, semantic scene information and length limit information, and the sharding strategy represented by each individual is randomly selected from a preset sharding strategy set; The method comprises the steps of calculating the fitness of each individual according to the sharding strategy represented by each individual in the population, the source text, and the corpus model; screening the individuals in the population according to the fitness of each individual to obtain target individuals; performing cross-breeding and mutation using the target individuals to obtain the next round of population; in response to detecting that the termination condition of the genetic algorithm is not currently met, returning to the step of calculating the fitness of each individual according to the sharding strategy represented by each individual in the population, the source text, and the corpus model for the next round of population; and in response to detecting that the termination condition of the genetic algorithm is currently met, determining the sharding strategy obtained when the termination condition of the genetic algorithm is met as the target sharding strategy.

3. The method according to claim 2, characterized in that The step of calculating the fitness of each individual according to the sharding strategy represented by each individual in the population, the source text, and the corpus model includes: Inputting the segmentation strategy represented by each of the individuals and the segmentation source text obtained under the segmentation strategy into the corpus model, so that the corpus model generates a predicted text based on the segmentation source text under the semantic scene information contained in the segmentation strategy, and outputs a cross entropy calculated based on the segmentation source text and the predicted text, and a first length difference between the segmentation source text and the predicted text; Calculate basic fitness according to the cross entropy, the first length difference, and the first length of the segment source text; the basic fitness is negatively correlated with the cross entropy and the first length difference; the basic fitness is positively correlated with the first length; Based on the preset first weighted weight, second weighted weight and third weighted weight, the basic fitness is weightedly calculated to obtain the fitness of the individual; the first weighted weight is used to characterize the influence of the subject information on the fitness, the second weighted weight is used to characterize the influence of the usage area information on the fitness, and the third weighted weight is used to characterize the influence of the length limit information on the fitness.

4. The method according to claim 3, characterized in that After calculating the basic fitness according to the cross entropy, the length difference, and the first length of the segment source text, the method further includes: A length influencing factor is superimposed on the basic fitness, wherein the length influencing factor is negatively correlated with a second length difference; the second length difference is a difference between a maximum character length acceptable to the translation model and the first length; The step of performing a weighted operation on the basic fitness based on the preset first weighted weight, the second weighted weight and the third weighted weight to obtain the fitness of the individual comprises: The basic fitness is weighted based on the first weighted weight and the second weighted weight, and the basic fitness after the length influencing factor is superimposed is weighted based on the third weighted weight to obtain the fitness of the individual.

5. A text translation method, characterized in that: The method comprises: inputting a segmented source text into a translation model to obtain a translation result, wherein the source text is obtained by segmenting according to any one of the methods of claims 1-4.

6. A text segmentation device, characterized in that: The device comprises: A generation module is used to determine a source text, and generate a target segmentation strategy for the source text using a genetic algorithm based on the source text and a preset corpus model, wherein the target segmentation strategy for the source text includes segmentation position information; the corpus model is used to evaluate the fitness of the segmentation position information based on semantic scene information and length limit information; the semantic scene information includes: subject information to which the source text belongs and area of ​​use information of the source text; the length limit information represents the maximum character length that can be accepted by the translation model; The segmentation module is used to segment the source text based on the segmentation position information included in the target segmentation strategy.

7. A text translation device, characterized in that: The device comprises: a translation module, which is used to input the segmented source text into a translation model to obtain a translation result, wherein the source text is obtained by segmenting according to any one of the methods of claims 1-4.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 5 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 5 are implemented.

10. A computer program product, characterized in that The method comprises a computer program which, when executed by a processor, implements the method steps according to any one of claims 1 to 5.

Images