Space processing method, device and electronic equipment

Abstract

The present application discloses a space processing method, device and electronic device, belonging to the field of information processing technology. The method comprises: connecting two first sentences according to P preset connection methods to obtain P candidate sentences corresponding to the P preset connection methods, wherein the two first sentences are two adjacent sentences at the position of the first space in the first text, and P is an integer greater than 1; obtaining semantic vectors of the P candidate sentences; obtaining the space score of the first space based on the semantic vector; and performing a target operation on the first space based on the space score.

Description

Technical Field

[0001] The present application belongs to the field of information processing technology, and specifically relates to a space processing method, device and electronic equipment. Background Art

[0002] Nowadays, many users usually use spaces instead of punctuation marks in text when creating or expressing. Even in many formal scenarios, there are cases where spaces are used incorrectly, which will result in many invalid or irregular spaces in the text.

[0003] Currently, spaces in text are usually deleted or uniformly replaced with certain punctuation marks such as commas. These processing methods may cause deviations in sentence separation in the text and the processing effect is relatively poor. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a space processing method, device and electronic device, which can solve the problem that the processing method of spaces in text may cause deviations in sentence separation in the text, resulting in poor space processing effect.

[0005] In a first aspect, an embodiment of the present application provides a space processing method, the method comprising:

[0006] Connecting the two first clauses according to P preset connection modes to obtain P candidate sentences corresponding to the P preset connection modes, wherein the two first clauses are two clauses adjacent to each other at the position of the first space in the first text, and P is an integer greater than 1;

[0007] Obtaining semantic vectors of the P candidate sentences;

[0008] Acquire a space score of the first space based on the semantic vector;

[0009] A target operation is performed on the first space based on the space score.

[0010] In a second aspect, an embodiment of the present application provides a space processing device, the device comprising:

[0011] a connection module, configured to connect the two first clauses in accordance with P preset connection modes to obtain P candidate sentences corresponding to the P preset connection modes, wherein the two first clauses are two clauses adjacent to each other at the position of the first space in the first text, and P is an integer greater than 1;

[0012] A first acquisition module, used to acquire the semantic vectors of the P candidate sentences;

[0013] A second acquisition module, configured to acquire a space score of the first space based on the semantic vector;

[0014] A target operation module is used to perform a target operation on the first space based on the space score.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0018] In an embodiment of the present application, two first clauses are connected in accordance with P preset connection methods to obtain P candidate sentences corresponding to the P preset connection methods, wherein the two first clauses are two clauses adjacent to each other at the position of the first space in the first text; semantic vectors of the P candidate sentences are obtained; a space score of the first space is obtained based on the semantic vector; and a target operation is performed on the first space based on the space score. In this way, the space score that the space meets can be determined based on the two clauses adjacent to each other at the position of each space in the first text, and the space can be processed accordingly based on the space score, so that the spaces in the text can be correctly processed, which helps to correctly understand the text, and further improves the processing effect of the spaces in the text. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of a space processing method provided in an embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of the structure of the Transformer model;

[0021] Figure 3 It is a schematic diagram of the syntactic structure relationship between the words in the first clause of the two second clauses;

[0022] Figure 4 It is a schematic diagram of the syntactic structure relationship between the words in the latter clause of the two second clauses;

[0023] Figure 5is a structural diagram of a space processing device provided in an embodiment of the present application;

[0024] Figure 6 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0025] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] The following is a detailed description of the space processing provided in the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0029] Figure 1 is a flow chart of a space processing method provided in an embodiment of the present application, such as Figure 1 As shown, the following steps are included:

[0030] Step 101, connect two first clauses according to P preset connection methods to obtain P candidate sentences corresponding to the P preset connection methods, wherein the two first clauses are two clauses adjacent to each other at the position of the first space in the first text.

[0031] Wherein, P is an integer greater than 1.

[0032] In this step, the first text may be any text, a long text or a short text, which is not specifically limited here.

[0033] The first text may include at least one space, and the at least one space may separate the first text into a plurality of sentences. Of course, the first text may also include punctuation marks, which are not specifically limited here.

[0034] The first space may be any space in the first text. For any space, two adjacent clauses before and after the space may be obtained to obtain two first clauses.

[0035] For example, the first text is "Hong Kong Stock Changes | Company (1801.) rose nearly 5% and cooperated with Corporation V to introduce the next generation P inhibitor", and the first text includes multiple spaces, each of which can separate the first text into different sentences. Accordingly, the two adjacent sentences at each space position constitute two first sentences.

[0036] The first text can be segmented based on the position of each space to obtain a set of sentences arranged in a front-to-back order. For example, the first text is segmented into a set of sentences ["Hong Kong stock market changes" "|" "Company (1801.) rose nearly 5%" "cooperate with Corporation" "V" "introduce the next generation P inhibitor"], and any two adjacent sentences in the set of sentences are two first sentences. The spaces in the first text can be processed according to the order of the spaces in the first text to determine whether the spaces in the first text are deleted or replaced.

[0037] Specifically, based on P preset connection methods, two adjacent first sentences corresponding to the first space in the sentence set can be connected in a front-to-back order to obtain P candidate sentences corresponding to the P preset connection methods. The P preset connection methods may include two or three, which are not specifically limited here.

[0038] In the case where the P kinds of preset connection methods include two, the two sentences may be connected by a head-to-tail splicing method and the two sentences may be connected by a preset punctuation mark, and the preset punctuation mark may be “,”.

[0039] When the P preset connection methods include three types, they may be the first connection method, the second connection method and the third connection method. The first connection method connects two sentences through a space, the second connection method connects two sentences through head-to-tail splicing, and the third connection method connects two sentences through preset punctuation marks.

[0040] Step 102: Obtain the semantic vectors of the P candidate sentences.

[0041] The semantic vector of each candidate sentence can be determined to obtain P semantic vectors. Specifically, each candidate sentence can be input into the language model for vector representation to obtain the semantic vector of the candidate sentence.

[0042] Step 103: Obtain a space score of the first space based on the semantic vector.

[0043] The space score can indicate whether the first space has a separation function, that is, whether the first space is redundant, that is, it can indicate the effectiveness of text segmentation. For example, when the space score is greater than a preset threshold, it can be determined that the first space is redundant, that is, the first space is invalid for text segmentation and has no practical significance. When the space score is less than or equal to the preset threshold, it can be determined that the first space is not redundant, that is, the first space is valid for text segmentation and has a separation function.

[0044] When the P preset connection methods include two, it is possible to determine which candidate sentence is more in line with the semantic logic based on the space score, so that the appropriate connection method for the two first clauses can be determined from the above two connection methods, and then it is possible to determine whether the space connecting the two first clauses is redundant based on the determined connection method.

[0045] In the case where the P preset connection methods include three types, the similarity distance between the semantic vector of the candidate sentence corresponding to the first connection method and the semantic vector of the candidate sentence corresponding to the second connection method, as well as the similarity distance between the semantic vector of the candidate sentence corresponding to the first connection method and the semantic vector of the candidate sentence corresponding to the third connection method can be determined respectively, and based on these two similarity distances, the space score of the first space can be determined to determine whether the candidate sentence corresponding to the first connection method is more similar to the candidate sentence corresponding to the second connection method, or whether the candidate sentence corresponding to the first connection method is more similar to the candidate sentence corresponding to the third connection method, so as to determine the appropriate connection method for the two first clauses from the second connection method and the third connection method, and then determine whether the space connecting the two first clauses is redundant based on the determined connection method.

[0046] Step 104: perform a target operation on the first space based on the space score.

[0047] Based on the space score, a target connection mode can be determined from P preset connection modes, and the target connection mode can indicate whether the first space is redundant. Accordingly, a target operation can be performed on the first space in the same connection mode as the target connection mode.

[0048] Among them, when the target connection method is to connect two sentences by splicing the head and the tail, it can be determined that the first space is redundant. When the target connection method is to connect two sentences by preset punctuation marks, it can be determined that the first space is not redundant and has a separation function.

[0049] Afterwards, the target operation can be performed on the first space in the same connection mode as the target connection mode. For example, when the target connection mode is the second connection mode, it can be determined that the first space is redundant, and a deletion operation is performed accordingly, that is, the first space in the first text is deleted. When the target connection mode is the third connection mode, it can be determined that the first space is not redundant, and when the first space does not meet the preset text specification, a replacement operation is performed accordingly, that is, the first space in the first text is replaced. When the first space meets the preset text specification, no operation can be performed.

[0050] In a specific implementation, the first text can be segmented based on the position of each space to obtain a plurality of first sentences arranged in a front-to-back order; based on P preset connection methods, two adjacent first sentences corresponding to the first space in the plurality of first sentences are connected in a front-to-back order to obtain P candidate sentences corresponding to the P preset connection methods, where P is an integer greater than 1; the semantic vector of each candidate sentence is determined to obtain P semantic vectors; based on the P semantic vectors, a space score of the first space is determined to select a target connection method from the P preset connection methods based on the space score, and the target connection method is used to characterize the effectiveness of the first space for text segmentation; based on the target connection method, a target operation to be performed on the first space is determined.

[0051] In this embodiment, two first clauses are connected in accordance with P preset connection modes to obtain P candidate sentences corresponding to the P preset connection modes, wherein the two first clauses are two clauses adjacent to each other at the position of the first space in the first text; semantic vectors of the P candidate sentences are obtained; a space score of the first space is obtained based on the semantic vector; and a target operation is performed on the first space based on the space score. In this way, the space score that the space meets can be determined based on the two clauses adjacent to each other at the position of each space in the first text, and the space can be processed accordingly based on the space score, so that the spaces in the text can be correctly processed, which helps to correctly understand the text, and further improves the processing effect of the spaces in the text.

[0052] Optionally, the P preset connection modes include a first connection mode, a second connection mode and a third connection mode, the first connection mode connects two clauses by a space, the second connection mode connects two clauses by a head-to-tail splicing mode, and the third connection mode connects two clauses by a preset punctuation mark;

[0053] The step 103 specifically includes:

[0054] Based on the semantic vector, determining a first similarity distance and a second similarity distance, wherein the first similarity distance is a similarity distance between a candidate sentence corresponding to the first connection mode and a candidate sentence corresponding to the second connection mode, and the second similarity distance is a similarity distance between a candidate sentence corresponding to the first connection mode and a candidate sentence corresponding to the third connection mode;

[0055] Based on the first similarity distance and the second similarity distance, a space score of the first space is determined.

[0056] In this implementation manner, the space score of the first space may be used to characterize the similarity between the first connection mode and other connection modes in the P preset connection modes.

[0057] Suppose that the first clause of the two adjacent first clauses corresponding to the first space is A, and the second clause is B. The candidate sentence corresponding to the first connection method can be expressed as "AB", and there is a space between the two clauses. The candidate sentence corresponding to the second connection method can be expressed as "AB", and there is no space between the two clauses. The candidate sentence corresponding to the third connection method can be expressed as "A, B", and the two clauses are connected by punctuation mark ",".

[0058] The semantic vectors of these three candidate sentences are represented by e AB 、e AB and e A，B It can be based on e AB and e AB , use the distance calculation formula to determine the first similarity distance, and use cos_sim(e AB , e AB ) indicates that, based on e AB and e A，B , use the distance calculation formula to determine the second similarity distance, and use cos_sim(e AB , e A，B )express.

[0059] The following formula (1) may be used to determine the space score of the first space based on the first similarity distance and the second similarity distance.

[0060]

[0061] Where V is the space score and penalty is the penalty coefficient, which can be greater than or equal to 1.

[0062] The larger the space score, the more AB With e AB Closer, you can choose the second connection mode, otherwise, characterize e AB With e A，B To get closer, you can choose the third connection method.

[0063] The penalty coefficient is a parameter designed for the case where there is a punctuation mark at the end of A or the beginning of sentence B. It means that there is already a punctuation mark between A and B. In this case, the space is more likely to be meaningless. In other words, "AB" should be closer to "AB". For example, A is "Hong Kong Stock Market Movements" and B is "|". Since B begins with a punctuation mark, in this case, the space between A and B is more likely to be a redundant space.

[0064] Of course, other formulas may also be used to determine the space score based on the first similarity distance and the second similarity distance, which is not specifically limited here.

[0065] In this embodiment, a first similarity distance and a second similarity distance are determined based on the semantic vector, wherein the first similarity distance is the similarity distance between the candidate sentence corresponding to the first connection mode and the candidate sentence corresponding to the second connection mode, and the second similarity distance is the similarity distance between the candidate sentence corresponding to the first connection mode and the candidate sentence corresponding to the third connection mode; based on the first similarity distance and the second similarity distance, the space score of the first space is determined. In this way, the space score of the first space can be determined by comparing the similarity of the candidate sentences, so that the appropriate connection mode of the two first clauses can be determined based on the space score, and then it can be determined whether the space connecting the two first clauses is redundant.

[0066] Optionally, step 104 specifically includes:

[0067] Determining target attribute information of the first space based on the space score;

[0068] When the target attribute information indicates that the first space is invalid for text segmentation, deleting the first space;

[0069] When the target attribute information indicates that the first space is effective for text segmentation and does not meet a preset text specification, a replacement operation is performed on the first space.

[0070] In this embodiment, the first text may include spaces of at least one attribute. For example, it may include spaces that are invalid for text segmentation. For example, the space between "Hong Kong Stock Movements" and the symbol "|" in "Hong Kong Stock Movements|" is a space that is invalid for text segmentation. It can be called a redundant space, which refers to a useless space. Deleting the space will not cause difficulty in understanding the text.

[0071] For example, spaces that are effective for text segmentation can be included. For example, the space between two English words such as "CorporationV" is an effective space for text segmentation. For example, the space in "Come to my house tomorrow and I'll treat you to dinner" is an effective space for text segmentation. Spaces with this attribute can be called non-redundant spaces and cannot be deleted. Deleting them may make it difficult to understand the text.

[0072] When spaces are effective for text segmentation, the properties of spaces can be divided into two types. The first type can be spaces that meet preset text specifications. For example, the space in "Corporation V" is a space that meets the preset text specifications. Spaces that meet the preset text specifications mean that the space can correctly segment different elements in the text.

[0073] The second type may be spaces that do not meet the preset text specifications. For example, the space in "Come to my house tomorrow and I'll treat you to a meal" does not meet the preset text specifications. The space is not a symbol that normally separates the two sentences, and the punctuation mark that normally separates the two sentences should be "," .

[0074] Correspondingly, the target attribute information can represent the effectiveness of the first space for text segmentation and whether it meets the preset text specification, and the attribute represented by the target attribute information of the first space can be one of the above three attributes. Specifically, the first case can represent that it is invalid for text segmentation, the second case can represent that it is valid for text segmentation and meets the preset text specification, and the third case can represent that it is valid for text segmentation and does not meet the preset text specification.

[0075] The target attribute information of the first space can be determined based on the space score. Specifically, when the space score is determined based on the above formula (1), the above formula (1) actually determines the role of the space by determining the connection between the sentences between the spaces, thereby determining the attribute of the space. When the space score is greater than the preset threshold, it can be determined that the first space is invalid for text segmentation. When the space score is less than or equal to the preset threshold, it can be determined that the first space is valid for text segmentation. As for whether the first space meets the preset text specification, it needs to be further determined, which is described in detail in the following implementation.

[0076] Correspondingly, when the target attribute information indicates that the first space is invalid for text segmentation, the first space can be deleted; and when the target attribute information indicates that the first space is valid for text segmentation and does not meet the preset text specification, the first space is replaced.

[0077] In this implementation, the target attribute information of the first space is determined based on the space score; if the target attribute information indicates that the first space is invalid for text segmentation, the first space is deleted; if the target attribute information indicates that the first space is valid for text segmentation and does not meet the preset text specification, the first space is replaced. In this way, the target operation to be performed on the first space can be determined based on the space score.

[0078] Optionally, determining the target attribute information of the first space based on the space score includes:

[0079] When the space score is greater than a preset threshold, determining that the first space is the target attribute information that is invalid for text segmentation;

[0080] When the space score is less than or equal to the preset threshold, the first space is determined to be the target attribute information effective for text segmentation.

[0081] In this implementation, when the space score of the first space is determined based on the above formula (1), a threshold value, i.e., a preset threshold value, can be set. When the space score is greater than the preset threshold value, it is determined that "AB" is more similar to "AB", and the second connection mode is determined as the target connection mode. The second connection mode indicates that the first space has no actual meaning and is redundant, and can be omitted, that is, the first space is determined to be invalid for text segmentation. Among them, the preset threshold value can be set according to actual conditions and is not specifically limited here.

[0082] When the space score is less than or equal to the preset threshold, it is judged that "AB" is more similar to "A, B", and the third connection method is determined as the target connection method. The third connection method indicates that the first space has a separation function and is not redundant, that is, the first space is determined to be effective for text segmentation.

[0083] In this implementation, when the space score is greater than a preset threshold, the first space is determined to be the target attribute information that is invalid for text segmentation; when the space score is less than or equal to the preset threshold, the first space is determined to be the target attribute information that is valid for text segmentation. In this way, it is very simple to determine whether a space is redundant.

[0084] Optionally, determining the target attribute information of the first space based on the space score includes:

[0085] When the space score is less than or equal to a preset threshold, obtaining the last text unit in the first sentence of the two first sentences and the first text unit in the second sentence of the two first sentences;

[0086] In a case where the type of the last text unit is the same as that of the first text unit and is a preset type, determining that the first space is the target attribute information that is valid for text segmentation and meets a preset text specification;

[0087] When the type of the last text unit is different from that of the first text unit, or the type of the last text unit is the same as that of the first text unit but is not the preset type, the first space is determined to be the target attribute information that is valid for text segmentation and does not meet the preset text specification.

[0088] In this embodiment, when it is determined that the first space is effective for text segmentation, it is necessary to further determine whether the first space meets the preset text specification. Specifically, it can be judged based on whether the first space meets the last text unit of the first sentence and the first text unit of the second sentence in the corresponding two adjacent first sentences. Among them, the text unit can refer to an element in a sentence, such as text, numbers or words.

[0089] In the case where the type of the last text unit is the same as that of the first text unit and is a preset type, it is determined that the first space meets the preset text specification. The preset type may include English words, numbers, or words in other languages. In other words, if the last text unit and the first text unit are both English words or numbers, it is determined that the first space meets the preset text specification, and no punctuation is needed.

[0090] When the type of the last text unit is different from that of the first text unit, or the type of the last text unit is the same as that of the first text unit but is not the preset type, it is determined that the first space does not meet the preset text specification, and punctuation marks need to be added here to separate two adjacent sentences.

[0091] In this implementation, whether the first space meets the preset text specification can be determined by the type of the last text unit of the first sentence and the first text unit of the second sentence in two adjacent sentences.

[0092] Optionally, in the case where a replacement operation is performed on the first space based on the space score, the method further includes:

[0093] Predicting the punctuation mark at the first space position to obtain a target punctuation mark;

[0094] The first space is replaced with the target punctuation mark.

[0095] In this embodiment, when the first space is replaced based on the space score, that is, the attribute of the first space is determined to be valid for text segmentation and does not meet the preset text specification. In this scenario, it is necessary to predict the punctuation mark at the position of the first space for replacement, so as to correctly separate the two adjacent sentences at the position of the first space.

[0096] The punctuation mark at the first space position may be predicted based on the two first sentences, or may be predicted based on the two second sentences, which is not specifically limited here.

[0097] Among them, the two second sentences can be two adjacent sentences at the position of the first space in the second text, and the second text can be determined based on the first text and the target attribute information of the space determined by the space score, that is, the spaces in the first text that are invalid for text segmentation can be deleted to obtain the second text.

[0098] The punctuation mark at the first space position can be predicted based on a deep learning model, such as a Transformer model based on an attention mechanism to predict the punctuation mark at the first space position.

[0099] In this embodiment, by predicting the punctuation mark at the position of the first space to obtain a target punctuation mark when performing a replacement operation on the first space based on the space score, the first space is replaced with the target punctuation mark. In this way, spaces that are effective for text segmentation and do not meet the preset text specification can be correctly processed, further improving the processing effect of spaces.

[0100] Optionally, predicting the punctuation mark at the first space position to obtain a target punctuation mark includes:

[0101] Dividing the two second sentences according to words respectively to obtain a first word sequence corresponding to a first sentence of the two second sentences and a second word sequence corresponding to a second sentence of the two second sentences, wherein the two second sentences are two sentences adjacent to each other at the position of the first space in the second text, and the second text is at least a part of the text determined based on the space score in the first text;

[0102] For each word in the target word sequence, based on the relationship between the word and other words in the target word sequence, determine the relevance weight information corresponding to the word; based on the relevance weight information, fuse the first feature information of each word in the target word sequence to obtain the target feature information of the word, and the target word sequence is the first word sequence or the second word sequence;

[0103] Based on the target feature information of the words in the first word sequence and the target feature information of the words in the second word sequence, the punctuation mark at the first space position is predicted to obtain a target punctuation mark.

[0104] Specifically, before making a prediction, the spaces that are invalid for text segmentation in the first text can be deleted to obtain the second text. For example, the first text is "Hong Kong Stock Market Changes | Company (1801.) rose nearly 5% and cooperated with Corporation V to introduce the next generation of P inhibitors", and after deleting the spaces that are invalid for text segmentation in the first text, the second text "Hong Kong Stock Market Changes | Company (1801.) rose nearly 5% and cooperated with Corporation V to introduce the next generation of P inhibitors" is obtained, that is, the spaces before and after "|" and the spaces between "cooperation" and "introduction" are deleted respectively.

[0105] In addition, if the second text includes a space that is effective for text segmentation and meets the preset text specification, the space can be retained and is not used as a basis for sentence segmentation, that is, the space can be used to connect the two elements before and after, so that they are different elements of the same sentence, that is, the two elements before and after the space are different elements of the same sentence. For example, "with Corporation" and "cooperate with V to introduce the next generation of P inhibitors" can be combined to finally obtain the sentence "cooperate with Corporation V to introduce the next generation of P inhibitors".

[0106] When the first text does not include a space that is invalid for text segmentation, the second text is the first text.

[0107] Afterwards, the second text can be segmented based on the position of the first space to obtain a set of sequentially arranged sentences, which can include multiple second sentences. The set of sentences can be arranged in order according to the order before and after the second text. For example, the second text "Hong Kong Stock Movements|Company (1801.) rose nearly 5% and cooperated with Corporation V to introduce the next generation of P inhibitors" is segmented based on the position of the first space to obtain two second sentences, namely "Hong Kong Stock Movements|Company (1801.) rose nearly 5%" and "Cooperated with Corporation V to introduce the next generation of P inhibitors".

[0108] Afterwards, for each two adjacent second sentences, the punctuation mark at the first space position between the two adjacent second sentences can be predicted based on the relationship between the two adjacent second sentences, so as to replace the first space with a suitable punctuation mark and correctly separate the sentences in the second text.

[0109] Among them, the punctuation mark at the first space position between the two adjacent second sentences can be predicted based on the semantic relationship between the two adjacent second sentences, or based on the syntactic structure relationship between the two adjacent second sentences, or based on the semantic relationship and syntactic structure relationship between the two adjacent second sentences, without specific limitation here.

[0110] Taking the Transformer model predicting the punctuation mark at the first space position between two adjacent second sentences as an example, before prediction, the Transformer model needs to be trained. A large amount of text with punctuation marks that meet the preset text specifications can be collected, and split into two sentences before and after according to the position of the punctuation marks to construct a training data set, in which the two adjacent sentences are used as input and the punctuation marks are used as prediction targets.

[0111] Usually, the omission of punctuation marks often occurs between sentences with incomplete grammatical structures. Therefore, the punctuation marks that are replaced by spaces can be determined, that is, the prediction target can be set to any one of the punctuation marks set to be replaced {“,”, “,”, “:”, “——”}, and all text corpora are divided into sentences. Then, the punctuation marks in the punctuation marks set to be replaced divide the text into clauses with finer granularity, and construct triples. (T a , T b , Y) indicates that T a Indicates the previous clause at the punctuation mark, T b It indicates the next clause after the punctuation mark, Y∈{“,”,”,”,”:””——”}, which indicates the connector between the previous and next clauses.

[0112] The twin tower model can be used to train the two sentences in the training data set separately, then connect the features of the two sentences, predict the punctuation between the two sentences, and compare the predicted punctuation with the labels in the training data set to update the parameters of the Transformer model, and finally realize the parameters of the Transformer model. Its training process is similar to the supervised learning training process of the deep learning model, so it will not be elaborated here.

[0113] The choice of punctuation marks can usually be determined by the semantic logical relationship between the two clauses. For example, "," usually indicates a parallel relationship between the two clauses, while ":" indicates a complementary or explanatory relationship between the two clauses. By using the semantic logical relationship between the two clauses, punctuation marks with obvious differences in usage can be distinguished, while "," and "、", which can also be used for parallel relationships, are difficult to distinguish.

[0114] Therefore, we can further consider the grammatical structure of the two clauses, that is, the syntactic structure. For example, "," indicates a long pause, and "," is used for a short pause. The main difference is that "," is used between short phrases, while the structure of the two clauses before and after "," is more complex. To this end, we can predict the punctuation marks between the two clauses based on the identification of the semantic logical relationship between the two clauses and the grammatical structure features of the two clauses. That is, based on the semantic relationship and syntactic structure relationship between the two clauses, we can predict the punctuation marks between the two clauses.

[0115] After the training is completed, the punctuation mark at the first space position between each two adjacent second clauses can be predicted based on the Transformer model. Specifically, each of the two adjacent second clauses can be segmented and annotated, and then the word sequence and annotation information can be input into the Transformer model. Based on a series of prediction processing of the Transformer model, the punctuation mark at the first space position can be obtained. The specific prediction processing process is described as follows.

[0116] Based on the Transformer model, a double-tower structure can be used to represent the features of two adjacent second sentences respectively.

[0117] The structure diagram of the Transformer model is as follows Figure 2 As shown, the left structure can be used to characterize the former of the two second clauses, and the right structure can be used to characterize the latter of the two second clauses.

[0118] Specifically, the first clause of the two second clauses and the second clause of the two second clauses may be divided according to words to obtain a first word sequence corresponding to the first clause of the two second clauses and a second word sequence corresponding to the second clause of the two second clauses.

[0119] For example, the first sentence of the two second sentences is "Hong Kong stock movement|Company (1801.) rose nearly 5%", and the second sentence of the two second sentences is "cooperate with Corporation V to introduce the next generation P inhibitor". After word division, the first word sequence is shown in Table 1 below, and the second word sequence is shown in Table 2 below.

[0120] Table 1. The first word sequence and part-of-speech tagging results

[0121] Hong Kong stocks Changes | company ( 1801. ) rise close 5％ noun verb punctuation noun punctuation numeral punctuation verb adjective numeral

[0122] Table 2 The second word sequence and part-of-speech tagging results

[0123] and Corporation V cooperate Introduction Next Generation P Inhibitors preposition noun noun verb verb adjective noun noun

[0124] Afterwards, for each word in the target word sequence, the relevance weight information corresponding to the word can be determined based on the semantic relationship and / or syntactic structure relationship between the word and other words in the target word sequence; based on the relevance weight information, the first feature information of each word in the target word sequence is fused to obtain the target feature information of the word.

[0125] In an optional implementation, the relevance weight information corresponding to the word can be determined based on the semantic relationship between the word and other words in the target word sequence. The Transformer model can specifically represent the text semantics by learning the semantic relationship between words, and fuse the context features by calculating the semantic association weights between words.

[0126] In another optional implementation, the relevance weight information corresponding to the word may be determined based on the semantic relationship and syntactic structure relationship between the word and other words in the target word sequence, wherein the syntactic structure relationship may characterize the dependency relationship between words in a sentence, and the specific steps are as follows:

[0127] The determining, based on the relationship between the word and other words in the target word sequence, the relevance weight information corresponding to the word includes:

[0128] Acquire the syntactic structure relationship between the word and other words in the target word sequence;

[0129] Performing feature mapping on the syntactic structure relationship to obtain second feature information representing the syntactic structure relationship;

[0130] Determine the relevance weight information corresponding to the word based on the second feature information, the third feature information of the word, and the fourth feature information of other words in the target word sequence;

[0131] The first feature information, the third feature information and the fourth feature information are obtained by performing different transformations on the initial feature information of the word.

[0132] In this embodiment, the semantics of words in the target word sequence and the grammatical features between words can be characterized. The grammatical features may include features that characterize syntactic structural relationships, and may also include features that characterize syntactic structural relationships, namely syntactic features, and features that characterize parts of speech, namely lexical features. The semantic features and grammatical features are then fused to obtain the target feature information of the words.

[0133] Specifically, the first word sequence can be input into the model of the left structure, and the second word sequence can be input into the model of the right structure. Since the processing methods on both sides are the same, the feature characterization of the first word sequence will be used as an example for explanation.

[0134] like Figure 2 As shown, the input of the Transformer model may include the text word token in the first word sequence, the features of the word position and the part-of-speech tagging results of the words, as shown in Table 1, and the second row in the table is the part-of-speech tagging results of each word.

[0135] Each input can be mapped into the feature space by embedding to obtain the features of each input, such as Figure 2 As shown, B w1 The first row of features corresponds to the features of the part-of-speech tagging results of each word, P w1 The first row of features corresponds to the position features of each word, T w1 The first row of features corresponds to the semantic features of each word.

[0136] For each word in the first word sequence, the above three features can be combined in an addition or concatenation manner to obtain initial feature information of the word.

[0137] Afterwards, the initial feature information is subjected to three different linear transformations to obtain the first feature information, the third feature information and the fourth feature information of the word.

[0138] In addition, the input of the Transformer model may also include the syntactic structure relationship between the words in the first word sequence, and the syntactic structure relationship between the words in the first word sequence can be obtained by syntactic analysis. The syntactic structure relationship between the words in the first sentence of the two second sentences is as follows: Figure 3 As shown, the syntactic structure relationship between the words in the latter of the two second clauses is as follows Figure 4 shown.

[0139] The syntactic structure relationship between words in the first word sequence can also be mapped to the feature space by means of vector representation, and then integrated into the calculation structure of the Transformer. The specific calculation formulas are shown in the following equations (2), (3), (4) and (5).

[0140] r ij =E(R ij ) (2)

[0141]

[0142]

[0143]

[0144] Among them, in the above formulas (2) to (5), R ij can be the syntactic structure relationship between words in the first word sequence, r ij can be a feature vector representing the syntactic structure relationship. In the above formula (3), α ij It can be the correlation weight between the i-th word and the j-th word, is the normalized correlation weight between the i-th word and the j-th word, q i is the third feature information of the i-th word, k j is the fourth feature information of the jth word, v j is the first feature information of the jth word, d is a constant greater than 0, and the first feature information, the third feature information and the fourth feature information all represent the semantic features of the word.

[0145] The above formula (3) indicates that when characterizing the features of the i-th word, it is necessary to consider the semantic and syntactic structure relationship between the word and all other words at the same time to quantify the relevance weight.

[0146] The above formula (5) indicates that if the semantic relationship and syntactic structure relationship between words are considered at the same time, the first feature information of each word in the target word sequence and the feature information representing the syntactic structure relationship need to be fused based on the relevance weight information to obtain the target feature information of the word. That is, the first feature information of the jth word and the feature information representing the syntactic structure relationship between the ith word and the jth word are fused by adding, and then the context features are weighted fused based on the relevance weight information to obtain the target feature information of the ith word.

[0147] The target feature information of each word in the first word sequence can be fused by adding or concatenating to obtain the feature information of the first sentence of the two second sentences, and the target feature information of each word in the second word sequence can be fused by using the Transformer model of the right structure to obtain the feature information of the second sentence of the two second sentences.

[0148] Afterwards, the feature information of the first sentence of the two second sentences and the feature information of the second sentence of the two second sentences are concatenated, and the classification layer is used to output the prediction result, which may include the target punctuation mark and the prediction confidence. If the prediction confidence is high (such as reaching the threshold), the first space can be replaced with the target punctuation mark.

[0149] In this implementation, the punctuation mark at the first space position is predicted by utilizing the relationship between two adjacent second sentences and the relationship between words in each of the two adjacent second sentences, thereby improving the prediction accuracy of the punctuation mark.

[0150] Furthermore, by obtaining the syntactic structure relationship between the word and other words in the target word sequence; performing feature mapping on the syntactic structure relationship, obtaining the second feature information representing the syntactic structure relationship; based on the second feature information, the third feature information of the word and the fourth feature information of other words in the target word sequence, determining the correlation weight information corresponding to the word; wherein the first feature information, the third feature information and the fourth feature information are obtained by performing different transformations on the initial feature information of the word respectively. In this way, when predicting the punctuation mark at the first space position, not only the semantic relationship between the words in the sentence is taken into account, but also the syntactic structure relationship between the words in the sentence is taken into account, thereby further improving the prediction accuracy of the punctuation mark.

[0151] Optionally, the first characteristic information of the word is obtained by the following steps:

[0152] Acquire target information of the word, wherein the target information includes a part-of-speech tagging result of the word;

[0153] Performing feature mapping on the target information to obtain initial feature information of the word;

[0154] The initial feature information is transformed to obtain first feature information of the word.

[0155] In this embodiment, the target information may include the text word token in the first word sequence, the features of the word position and the part-of-speech tagging results of the word. Each input may be mapped to the feature space by embedding to obtain the features of each input, such as Figure 2 As shown, B w1 The first row of features corresponds to the features of the part-of-speech tagging results of each word, P w1 The first row of features corresponds to the position features of each word, T w1 The first row of features corresponds to the semantic features of each word.

[0156] For each word in the first word sequence, the above three features can be combined in an addition or concatenation manner to obtain initial feature information of the word.

[0157] Afterwards, the initial feature information is subjected to three different linear transformations to obtain the first feature information, the third feature information and the fourth feature information of the word.

[0158] In this implementation, when predicting punctuation marks, the lexical features of each word in the sentence are taken into consideration, thereby further improving the prediction accuracy of punctuation marks.

[0159] It should be noted that the space processing method provided in the embodiment of the present application can be executed by a space processing device or a control module in the space processing device for executing the space processing method. In the embodiment of the present application, the space processing device provided in the embodiment of the present application is described by taking the space processing device executing the space processing method as an example.

[0160] See also Figure 5 , Figure 5 is a structural diagram of a space processing device provided in an embodiment of the present application, such as Figure 5 As shown, the space processing device 500 includes:

[0161] A connection module 501 is used to connect two first clauses according to P preset connection modes to obtain P candidate sentences corresponding to the P preset connection modes, wherein the two first clauses are two clauses adjacent to each other at the position of the first space in the first text, and P is an integer greater than 1;

[0162] A first acquisition module 502 is used to acquire semantic vectors of the P candidate sentences;

[0163] A second acquisition module 503, configured to acquire a space score of the first space based on the semantic vector;

[0164] The target operation module 504 is used to perform a target operation on the first space based on the space score.

[0165] Optionally, the P preset connection modes include a first connection mode, a second connection mode and a third connection mode, the first connection mode connects two clauses by a space, the second connection mode connects two clauses by a head-to-tail splicing mode, and the third connection mode connects two clauses by a preset punctuation mark; the second acquisition module 503 is specifically used to:

[0166] Based on the semantic vector, determining a first similarity distance and a second similarity distance, wherein the first similarity distance is a similarity distance between a candidate sentence corresponding to the first connection mode and a candidate sentence corresponding to the second connection mode, and the second similarity distance is a similarity distance between a candidate sentence corresponding to the first connection mode and a candidate sentence corresponding to the third connection mode;

[0167] Based on the first similarity distance and the second similarity distance, a space score of the first space is determined.

[0168] Optionally, the target operation module 504 includes

[0169] A first determining unit, configured to determine target attribute information of the first space based on the space score;

[0170] A deletion operation unit, configured to perform a deletion operation on the first space when the target attribute information indicates that the first space is invalid for text segmentation;

[0171] A replacement operation unit is used to perform a replacement operation on the first space when the target attribute information indicates that the first space is effective for text segmentation and does not meet a preset text specification.

[0172] Optionally, the first determining unit is specifically configured to:

[0173] When the space score is greater than a preset threshold, determining that the first space is the target attribute information that is invalid for text segmentation;

[0174] When the space score is less than or equal to the preset threshold, the first space is determined to be the target attribute information effective for text segmentation.

[0175] Optionally, the first determining unit is specifically configured to:

[0176] When the space score is less than or equal to a preset threshold, obtaining the last text unit in the first sentence of the two first sentences and the first text unit in the second sentence of the two first sentences;

[0177] In a case where the type of the last text unit is the same as that of the first text unit and is a preset type, determining that the first space is the target attribute information that is valid for text segmentation and meets a preset text specification;

[0178] When the type of the last text unit is different from that of the first text unit, or the type of the last text unit is the same as that of the first text unit but is not the preset type, the first space is determined to be the target attribute information that is valid for text segmentation and does not meet the preset text specification.

[0179] Optionally, the device further comprises:

[0180] A prediction module, used for predicting the punctuation mark at the first space position to obtain a target punctuation mark;

[0181] A symbol replacement module is used to replace the first space with the target punctuation mark.

[0182] Optionally, the prediction module includes:

[0183] a dividing unit, configured to divide the two second sentences according to words respectively, to obtain a first word sequence corresponding to a first sentence of the two second sentences and a second word sequence corresponding to a second sentence of the two second sentences, wherein the two second sentences are two sentences adjacent to each other at the position of the first space in the second text, and the second text is at least a part of the text determined based on the space score in the first text;

[0184] A second determination unit is configured to determine, for each word in a target word sequence, relevance weight information corresponding to the word based on the relationship between the word and other words in the target word sequence; and to fuse first feature information of each word in the target word sequence based on the relevance weight information to obtain target feature information of the word, wherein the target word sequence is the first word sequence or the second word sequence;

[0185] A prediction unit is used to predict the punctuation mark at the first space position based on the target feature information of the words in the first word sequence and the target feature information of the words in the second word sequence to obtain a target punctuation mark.

[0186] Optionally, the second determining unit is specifically configured to:

[0187] Acquire the syntactic structure relationship between the word and other words in the target word sequence;

[0188] Performing feature mapping on the syntactic structure relationship to obtain second feature information representing the syntactic structure relationship;

[0189] Determine the relevance weight information corresponding to the word based on the second feature information, the third feature information of the word, and the fourth feature information of other words in the target word sequence;

[0190] The first feature information, the third feature information and the fourth feature information are obtained by performing different transformations on the initial feature information of the word.

[0191] Optionally, the first characteristic information of the word is obtained by the following steps:

[0192] Acquire target information of the word, wherein the target information includes a part-of-speech tagging result of the word;

[0193] Performing feature mapping on the target information to obtain initial feature information of the word;

[0194] The initial feature information is transformed to obtain first feature information of the word.

[0195] In this embodiment, the connection module 501 connects two first clauses according to P preset connection modes to obtain P candidate sentences corresponding to the P preset connection modes, wherein the two first clauses are two clauses adjacent to each other at the position of the first space in the first text; the semantic vectors of the P candidate sentences are obtained by the first acquisition module 502; the space score of the first space is obtained based on the semantic vector by the second acquisition module 503; and the target operation module 504 performs a target operation on the first space based on the space score. In this way, the space score that the space meets can be determined based on the two clauses adjacent to each other at the position of each space in the first text, and the space can be processed accordingly based on the space score, so that the space in the text can be correctly processed, which is helpful for the correct understanding of the text, and thus the processing effect of the space in the text can be improved.

[0196] The space processing device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0197] The space processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0198] The space processing device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0199] Alternatively, if Figure 6As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned space processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0200] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0201] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0202] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0203] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0204] The processor 710 is used for:

[0205] Connecting the two first clauses according to P preset connection modes to obtain P candidate sentences corresponding to the P preset connection modes, wherein the two first clauses are two clauses adjacent to each other at the position of the first space in the first text, and P is an integer greater than 1;

[0206] Obtaining semantic vectors of the P candidate sentences;

[0207] Acquire a space score of the first space based on the semantic vector;

[0208] A target operation is performed on the first space based on the space score.

[0209] In this embodiment, the processor 710 connects two first clauses according to P preset connection modes to obtain P candidate sentences corresponding to the P preset connection modes, wherein the two first clauses are two clauses adjacent to each other at the position of the first space in the first text; obtains semantic vectors of the P candidate sentences; obtains the space score of the first space based on the semantic vector; and performs a target operation on the first space based on the space score. In this way, the space score that the space meets can be determined based on the two clauses adjacent to each other at the position of each space in the first text, and the space is processed accordingly based on the space score, so that the space in the text can be correctly processed, which helps to correctly understand the text, and further improves the processing effect of the space in the text.

[0210] Optionally, the P preset connection modes include a first connection mode, a second connection mode, and a third connection mode, the first connection mode connects two sentences by a space, the second connection mode connects two sentences by a head-to-tail splicing mode, and the third connection mode connects two sentences by a preset punctuation mark; the processor 710 is further configured to:

[0211] Based on the semantic vector, determining a first similarity distance and a second similarity distance, wherein the first similarity distance is a similarity distance between a candidate sentence corresponding to the first connection mode and a candidate sentence corresponding to the second connection mode, and the second similarity distance is a similarity distance between a candidate sentence corresponding to the first connection mode and a candidate sentence corresponding to the third connection mode;

[0212] Based on the first similarity distance and the second similarity distance, a space score of the first space is determined.

[0213] Optionally, the processor 710 is further configured to:

[0214] Determining target attribute information of the first space based on the space score;

[0215] When the target attribute information indicates that the first space is invalid for text segmentation, deleting the first space;

[0216] When the target attribute information indicates that the first space is effective for text segmentation and does not meet a preset text specification, a replacement operation is performed on the first space.

[0217] Optionally, the processor 710 is further configured to:

[0218] When the space score is greater than a preset threshold, determining that the first space is the target attribute information that is invalid for text segmentation;

[0219] When the space score is less than or equal to the preset threshold, the first space is determined to be the target attribute information effective for text segmentation.

[0220] Optionally, the processor 710 is further configured to:

[0221] When the space score is less than or equal to a preset threshold, obtaining the last text unit in the first sentence of the two first sentences and the first text unit in the second sentence of the two first sentences;

[0222] In a case where the type of the last text unit is the same as that of the first text unit and is a preset type, determining that the first space is the target attribute information that is valid for text segmentation and meets a preset text specification;

[0223] When the type of the last text unit is different from that of the first text unit, or the type of the last text unit is the same as that of the first text unit but is not the preset type, the first space is determined to be the target attribute information that is valid for text segmentation and does not meet the preset text specification.

[0224] Optionally, when a replacement operation is performed on the first space based on the space score, the processor 710 is further configured to:

[0225] Predicting the punctuation mark at the first space position to obtain a target punctuation mark;

[0226] The first space is replaced with the target punctuation mark.

[0227] Optionally, the processor 710 is further configured to:

[0228] Dividing the two second sentences according to words respectively to obtain a first word sequence corresponding to a first sentence of the two second sentences and a second word sequence corresponding to a second sentence of the two second sentences, wherein the two second sentences are two sentences adjacent to each other at the position of the first space in the second text, and the second text is at least a part of the text determined based on the space score in the first text;

[0229] For each word in the target word sequence, based on the relationship between the word and other words in the target word sequence, determine the relevance weight information corresponding to the word; based on the relevance weight information, fuse the first feature information of each word in the target word sequence to obtain the target feature information of the word, and the target word sequence is the first word sequence or the second word sequence;

[0230] Based on the target feature information of the words in the first word sequence and the target feature information of the words in the second word sequence, the punctuation mark at the first space position is predicted to obtain a target punctuation mark.

[0231] Optionally, the processor 710 is further configured to:

[0232] Acquire the syntactic structure relationship between the word and other words in the target word sequence;

[0233] Performing feature mapping on the syntactic structure relationship to obtain second feature information representing the syntactic structure relationship;

[0234] Determine the relevance weight information corresponding to the word based on the second feature information, the third feature information of the word, and the fourth feature information of other words in the target word sequence;

[0235] The first feature information, the third feature information and the fourth feature information are obtained by performing different transformations on the initial feature information of the word.

[0236] Optionally, the processor 710 is further configured to:

[0237] Acquire target information of the word, wherein the target information includes a part-of-speech tagging result of the word;

[0238] Performing feature mapping on the target information to obtain initial feature information of the word;

[0239] The initial feature information is transformed to obtain first feature information of the word.

[0240] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processor (Graphics Processing Unit, GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 709 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 can integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 710.

[0241] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned space processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0242] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0243] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned space processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0244] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0245] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0246] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0247] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A space processing method, characterized in that: include: Connecting the two first clauses according to P preset connection modes to obtain P candidate sentences corresponding to the P preset connection modes, wherein the two first clauses are two clauses adjacent to each other at the position of the first space in the first text, and P is an integer greater than 1; the P preset connection modes include a first connection mode, a second connection mode, and a third connection mode, wherein the first connection mode connects the two clauses through a space, the second connection mode connects the two clauses through head-to-tail splicing, and the third connection mode connects the two clauses through preset punctuation marks; Obtaining semantic vectors of the P candidate sentences; Based on the semantic vector, determining a first similarity distance and a second similarity distance, wherein the first similarity distance is a similarity distance between a candidate sentence corresponding to the first connection mode and a candidate sentence corresponding to the second connection mode, and the second similarity distance is a similarity distance between a candidate sentence corresponding to the first connection mode and a candidate sentence corresponding to the third connection mode; Determining a space score of the first space based on the first similarity distance and the second similarity distance; the space score represents the effectiveness of the first space for text segmentation; Determining target attribute information of the first space based on the space score; When the target attribute information indicates that the first space is invalid for text segmentation, deleting the first space; When the target attribute information indicates that the first space is effective for text segmentation and does not meet a preset text specification, a replacement operation is performed on the first space.

2. The method according to claim 1, characterized in that The determining, based on the space score, target attribute information of the first space includes: When the space score is greater than a preset threshold, determining that the first space is the target attribute information that is invalid for text segmentation; When the space score is less than or equal to the preset threshold, the first space is determined to be the target attribute information effective for text segmentation.

3. The method according to claim 1, characterized in that: The determining, based on the space score, target attribute information of the first space includes: When the space score is less than or equal to a preset threshold, obtaining the last text unit in the first sentence of the two first sentences and the first text unit in the second sentence of the two first sentences; In a case where the type of the last text unit is the same as that of the first text unit and is a preset type, determining that the first space is the target attribute information that is valid for text segmentation and meets a preset text specification; When the type of the last text unit is different from that of the first text unit, or the type of the last text unit is the same as that of the first text unit but is not the preset type, the first space is determined to be the target attribute information that is valid for text segmentation and does not meet the preset text specification.

4. The method according to claim 1, characterized in that In the case where a replacement operation is performed on the first space based on the space score, the method further includes: Predicting the punctuation mark at the first space position to obtain a target punctuation mark; The first space is replaced with the target punctuation mark.

5. The method according to claim 4, characterized in that The step of predicting the punctuation mark at the first space position to obtain a target punctuation mark includes: Dividing the two second sentences according to words respectively to obtain a first word sequence corresponding to a first sentence of the two second sentences and a second word sequence corresponding to a second sentence of the two second sentences, wherein the two second sentences are two sentences adjacent to each other at the position of the first space in the second text, and the second text is at least a part of the text determined based on the space score in the first text; For each word in the target word sequence, based on the relationship between the word and other words in the target word sequence, determine the relevance weight information corresponding to the word; based on the relevance weight information, fuse the first feature information of each word in the target word sequence to obtain the target feature information of the word, and the target word sequence is the first word sequence or the second word sequence; Based on the target feature information of the words in the first word sequence and the target feature information of the words in the second word sequence, the punctuation mark at the first space position is predicted to obtain a target punctuation mark.

6. The method according to claim 5, characterized in that The determining, based on the relationship between the word and other words in the target word sequence, the relevance weight information corresponding to the word includes: Acquire the syntactic structure relationship between the word and other words in the target word sequence; Performing feature mapping on the syntactic structure relationship to obtain second feature information representing the syntactic structure relationship; Determine the relevance weight information corresponding to the word based on the second feature information, the third feature information of the word, and the fourth feature information of other words in the target word sequence; The first feature information, the third feature information and the fourth feature information are obtained by performing different transformations on the initial feature information of the word.

7. The method according to claim 5, characterized in that The first characteristic information of the word is obtained by the following steps: Acquire target information of the word, wherein the target information includes a part-of-speech tagging result of the word; Performing feature mapping on the target information to obtain initial feature information of the word; The initial feature information is transformed to obtain first feature information of the word.

8. A space processing device, characterized in that: include: a connection module, configured to connect two first clauses according to P preset connection modes to obtain P candidate sentences corresponding to the P preset connection modes, wherein the two first clauses are two clauses adjacent to each other at the position of a first space in the first text, and P is an integer greater than 1; the P preset connection modes include a first connection mode, a second connection mode, and a third connection mode, wherein the first connection mode connects the two clauses through a space, the second connection mode connects the two clauses through a head-to-tail splicing mode, and the third connection mode connects the two clauses through preset punctuation marks; A first acquisition module, used to acquire the semantic vectors of the P candidate sentences; A second acquisition module, configured to acquire a space score of the first space based on the semantic vector; the space score represents the effectiveness of the first space for text segmentation; a target operation module, configured to perform a target operation on the first space based on the space score; The second acquisition module is specifically used for: Based on the semantic vector, determining a first similarity distance and a second similarity distance, wherein the first similarity distance is a similarity distance between a candidate sentence corresponding to the first connection mode and a candidate sentence corresponding to the second connection mode, and the second similarity distance is a similarity distance between a candidate sentence corresponding to the first connection mode and a candidate sentence corresponding to the third connection mode; Determine a space score of the first space based on the first similarity distance and the second similarity distance; The target operation module includes A first determining unit, configured to determine target attribute information of the first space based on the space score; A deletion operation unit, configured to perform a deletion operation on the first space when the target attribute information indicates that the first space is invalid for text segmentation; A replacement operation unit is used to perform a replacement operation on the first space when the target attribute information indicates that the first space is effective for text segmentation and does not meet a preset text specification.

9. The device according to claim 8, characterized in that The first determining unit is specifically configured to: When the space score is greater than a preset threshold, determining that the first space is the target attribute information that is invalid for text segmentation; When the space score is less than or equal to the preset threshold, the first space is determined to be the target attribute information effective for text segmentation.

10. The device according to claim 8, characterized in that The first determining unit is specifically configured to: When the space score is less than or equal to a preset threshold, obtaining the last text unit in the first sentence of the two first sentences and the first text unit in the second sentence of the two first sentences; In a case where the type of the last text unit is the same as that of the first text unit and is a preset type, determining that the first space is the target attribute information that is valid for text segmentation and meets a preset text specification; When the type of the last text unit is different from that of the first text unit, or the type of the last text unit is the same as that of the first text unit but is not the preset type, the first space is determined to be the target attribute information that is valid for text segmentation and does not meet the preset text specification.

11. The device according to claim 8, characterized in that The device also includes: A prediction module, used for predicting the punctuation mark at the first space position to obtain a target punctuation mark; A symbol replacement module is used to replace the first space with the target punctuation mark.

12. The device according to claim 11, characterized in that The prediction module comprises: a dividing unit, configured to divide the two second sentences according to words respectively, to obtain a first word sequence corresponding to a first sentence of the two second sentences and a second word sequence corresponding to a second sentence of the two second sentences, wherein the two second sentences are two sentences adjacent to each other at the position of the first space in the second text, and the second text is at least a part of the text determined based on the space score in the first text; A second determination unit is configured to determine, for each word in a target word sequence, relevance weight information corresponding to the word based on the relationship between the word and other words in the target word sequence; and to fuse first feature information of each word in the target word sequence based on the relevance weight information to obtain target feature information of the word, wherein the target word sequence is the first word sequence or the second word sequence; A prediction unit is used to predict the punctuation mark at the first space position based on the target feature information of the words in the first word sequence and the target feature information of the words in the second word sequence to obtain a target punctuation mark.

13. The device according to claim 12, characterized in that The second determining unit is specifically configured to: Acquire the syntactic structure relationship between the word and other words in the target word sequence; Performing feature mapping on the syntactic structure relationship to obtain second feature information representing the syntactic structure relationship; Determine the relevance weight information corresponding to the word based on the second feature information, the third feature information of the word, and the fourth feature information of other words in the target word sequence; The first feature information, the third feature information and the fourth feature information are obtained by performing different transformations on the initial feature information of the word.

14. The device according to claim 12, characterized in that The first characteristic information of the word is obtained by the following steps: Acquire target information of the word, wherein the target information includes a part-of-speech tagging result of the word; Performing feature mapping on the target information to obtain initial feature information of the word; The initial feature information is transformed to obtain first feature information of the word.

Images