Text processing method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to artificial intelligence, and provides a text processing method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: representing at least two texts as corresponding vectors; in response to unequal dimensions of the at least two vectors, adjusting the dimensions of the at least two vectors to equal dimensions based on an adjustment rule; and determining the text similarity between the at least two texts according to the adjusted vectors. According to the method, the problem that the text similarity cannot be determined due to different dimensions of at least two vectors can be solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a text processing method, device, electronic device, storage medium, and program product. Background Art

[0002] With the development of natural language technology, tasks such as text matching and information retrieval can be achieved by calculating text similarity. In related technologies, texts are usually represented as corresponding vectors, and the similarity between texts is then calculated based on the corresponding vectors. Summary of the Invention

[0003] The present application provides a text processing method, device, electronic device, storage medium and program product to solve the technical problem that text similarity cannot be calculated in some scenarios.

[0004] A first aspect of an embodiment of the present application provides a text processing method, comprising: representing at least two texts as corresponding vectors; in response to unequal dimensions of at least two vectors, adjusting the dimensions of the at least two vectors to equal dimensions based on an adjustment rule; and determining the text similarity between the at least two texts based on the adjusted vectors.

[0005] A second aspect of an embodiment of the present application provides a text processing device, comprising: a characterization unit for characterizing at least two texts as corresponding vectors; an adjustment unit for adjusting the dimensions of the at least two vectors to equal dimensions based on an adjustment rule in response to the dimensions of the at least two vectors being unequal; and a determination unit for determining the text similarity between the at least two texts based on the adjusted vectors.

[0006] A third aspect of an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method provided in the first aspect when executing the computer program.

[0007] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the first aspect are implemented.

[0008] A fifth aspect of an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps in the method provided in the first aspect.

[0009] In the text processing method of this embodiment, when the dimensions of the vectors corresponding to at least two texts are unequal, adjusting the dimensions of the at least two vectors to equal dimensions can resolve the problem of being unable to determine text similarity due to the different dimensions of the at least two vectors. Furthermore, by adjusting the rules, the dimensions of the at least two vectors can be reasonably adjusted, thereby preventing the adjusted vectors from losing important information from the text and ensuring the accuracy of text similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 This is a schematic diagram of an application scenario of a text processing method provided by an embodiment of the present application; Figure 2 is a flowchart of a text processing method provided by an embodiment of the present application; Figure 3 is a flowchart of a text processing method provided by another embodiment of the present application; Figure 4 is a flowchart of a text processing method provided by another embodiment of the present application; Figure 5 is a flowchart of a text processing method provided by another embodiment of the present application; Figure 6 This is a functional module diagram of a text processing device provided by an embodiment of the present application; Figure 7 It is a structural diagram of an electronic device for implementing a text processing method provided in one embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0013] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application. It should be understood that, unless otherwise specified in this application, " / " means or. For example, A / B can mean A or B. "And / or" in this application is merely a way to describe the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. "At least one" means one or more. "Multiple" means two or more than two. For example, at least one of a, b or c can mean: a, b, c, a and b, a and c, b and c, a, b and c.

[0015] Some terminology explanations: Pre-trained language model (Bidirectional Encoder Representation from Transformers, BERT): The pre-trained language model is a bidirectional encoding model connected using the Transformer Encoder block. The pre-trained language model uses a masked language model (MLM) to generate deep bidirectional language representations.

[0016] Term Frequency–Inverse Document Frequency (TF-IDF): is a weighting technique used in information retrieval and data mining.

[0017] Edit distance, also known as Levenshtein distance, refers to the minimum number of edit operations required to convert one string into another. The greater the distance between two strings, the more different the two strings are.

[0018] The Word2Vec language model is a model that learns semantic knowledge in an unsupervised manner and is widely used in natural language processing. Word vectors can be generated using the Word2Vec model.

[0019] With the development of natural language technology, text similarity calculations can be used to achieve tasks such as text matching, text classification, information retrieval, and cluster analysis. Text matching tasks can be divided into short-text-short-text matching tasks, short-text-long-text matching tasks, and long-text-long-text matching tasks based on the length of the text.

[0020] For example, in a search engine scenario, since the text length of the user question (query) is relatively short and the text length of the web page body (content) is relatively long, it is necessary to use the short text-long text matching task to determine the similarity between the query and the content.

[0021] In related solutions, the language model can be fine-tuned using the texts in the short text library and the long text library to obtain a representation model, which can then be used to represent the short text and the long text as vectors of the same dimension. Finally, the similarity between the two vectors is calculated through cosine similarity, Euclidean distance, etc. to obtain the similarity between the short text and the long text.

[0022] For example, in the above solution, the representation model can be used to represent short text and long text as 128-dimensional vectors respectively. However, since long text contains more content, the 128-dimensional vector cannot accurately represent the semantics of the long text, resulting in the inability to accurately determine the similarity between short text and long text, affecting the search accuracy of the search engine scenario.

[0023] For example, in the above solution, the representation model can be used to represent short text and long text as 512-dimensional vectors respectively. However, due to the high dimensionality of the vectors obtained by representing short text and long text, it is impossible to quickly determine the similarity between short text and long text, which affects the search efficiency of the search engine scenario.

[0024] To solve the above technical problems, an embodiment of the present application provides a text processing method that can improve the efficiency and accuracy of determining similarity.

[0025] refer to Figure 1 As shown, Figure 1: This is a schematic diagram of an application scenario of a text processing method provided by an embodiment of the present application. In the application scenarios provided by some embodiments of the present application, the electronic device 100 is connected to a server where the first text library 200 is located and a server where the second text library 201 is located (hereinafter, the first text library 200 is used as an example). Through the communication connection between the electronic device 100 and the server where the first text library 200 is located, the electronic device 100 can train the language model through the text obtained from the first text library 200. The electronic device 100 can also determine the adjustment rules based on the text obtained from the first text library 200, wherein the adjustment rules may include dimension adjustment rules and position adjustment rules. The dimension adjustment rules can be used to adjust the dimension of the representation vector or representation matrix of the language model, for example, using the dimension adjustment rules to perform dimensionality reduction processing. The position adjustment rules can be used to adjust the position of the element value in the representation vector or representation matrix of the language model. The position adjustment rules can also be used to adjust the position of the element value in the vector or matrix after dimensionality reduction.

[0026] The electronic device 100 can also be connected to the terminal device 300 corresponding to the user. In the search scenario, the terminal device 300 can send an information retrieval request to the electronic device 100, and the terminal device 300 can receive the retrieval results sent by the electronic device 100. In the question-and-answer scenario, the terminal device 300 can send a question-and-answer request to the electronic device 100, and the terminal device 300 can receive the reply results sent by the electronic device 100.

[0027] The electronic device 100 in this application may include a network device and / or a user device. A network device includes, but is not limited to, a single network electronic device, a group of multiple network electronic devices, or a cloud based on cloud computing consisting of a large number of hosts or network electronic devices. The electronic device 100 may also be any electronic product capable of human-computer interaction with a user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), interactive network television (IPTV), etc.

[0028] The terminal device 300 in the present application can be a station (STAION, ST) in a WLAN, and can be a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a vehicle networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, a wireless modem card, a TV set-top box (STB), a customer premises equipment (CPE) and / or other devices for communicating on a wireless system and a next-generation communication system, for example, a terminal device in a 5G network or a terminal device in a future evolved public land mobile network (PLMN) network, etc.

[0029] As an example and not a limitation, when the terminal device is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not just hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are full-featured, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0030] In another embodiment, the terminal device 300 can be integrated into the electronic device 100 so that the electronic device 100 can implement some or all of the functions of the terminal device 300. For example, the electronic device 100 receives an information retrieval request input by a user, determines the language model and adjustment rules through the text in the first text library 200 and the second text library 201, calculates the text similarity between the request text in the information retrieval request and the web page text, and based on the text similarity, selects a retrieval result that matches the request text from multiple web page texts.

[0031] In another embodiment, the first text library 200 and the second text library 201 may be external databases (e.g., cloud storage) of the electronic device 100, or the first text library 200 and the second text library 201 may be built-in databases of the electronic device 100. For example, the first text library 200 and the second text library 201 may be built in a storage device of the electronic device 100.

[0032] In another embodiment, the terminal device 300 can communicate with the electronic device 100 through a network. The network can be a wireless network or a wired network. The wireless network includes, but is not limited to: a cellular network, a wireless local area network (Wireless Fidelity, Wi-Fi), an infrared network or a Bluetooth network. The wired network includes, but is not limited to: Ethernet, a universal serial bus (USB) or a controller area network.

[0033] It should be noted that Figure 1 The number and connection relationship of each device are only for illustration and do not constitute a limitation to the embodiments of the present application.

[0034] See Figure 2 FIG. 1 is a flowchart of a text processing method provided by an embodiment of the present application. The text processing method is applied in electronic devices, for example, Figure 1 The electronic device 100. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0035] S201: Represent at least two texts as corresponding vectors.

[0036] In at least one embodiment of the present application, the text lengths of at least two texts may be equal, or the text lengths of at least two texts may be unequal. In a search scenario, the at least two texts may include a user question (query) and a webpage text (content). For example, the user question (query) may be "How to use Python to convert a list", and the webpage text (content) may be "How to use the split() method to split a string into a list, for example: #Example string string = "apple,banana,cherry" #Use split() method to convert a string into a list list_from_string = string.split(',') print(list_from_string) # Output: ['apple', 'banana', 'cherry']".

[0037] In the intelligent question-and-answer scenario, at least two texts may include a query statement (query) input by the user and a question text (question) in the question library. For example, the query statement (query) may be "I don't know why my bank card was frozen and cannot be used", and the question text (question) may be "Bank card frozen", "Bank card locked", "Bank card unlocked", etc.

[0038] In at least one embodiment of the present application, the electronic device may represent at least two texts using the same language model (eg, BERT model) to obtain vectors corresponding to the at least two texts.

[0039] In other embodiments, the electronic device can represent texts of different lengths using different language models. Before annotating the text, the electronic device can pre-train a first model and a second model, which can be used to represent texts of different lengths. For example, the first model is used to represent texts whose length is less than a preset length, and the second model is used to represent texts whose length is greater than or equal to the preset length. The preset length can be set and adjusted according to actual needs.

[0040] In some embodiments, the electronic device may train a first model using text from a first text library and may train a second model using text from a second text library. The length of the text in the first text library is different from the length of the text in the second text library, for example, the length of the text in the first text library is less than the length of the text in the second text library.

[0041] In some embodiments, the following description is made using an example of at least two texts including a first text and a second text, but the actual application is not limited thereto. The text length of the first text is not equal to the text length of the second text. The electronic device can characterize the first text through a first model to obtain a first vector, and the electronic device can characterize the second text through a second model to obtain a second vector. The dimension of the first vector is not equal to the dimension of the second vector. For example, the first text can be represented as a first vector [0.156 0.563 0.259] through the first model, and the dimension of the first vector is 3; the second text can be represented as a second vector [0.459 0.166 0.242 0.995 0.879 0.601] through the second model, and the dimension of the second vector is 6.

[0042] This embodiment uses the first model and the second model to represent texts of different lengths, respectively, to obtain a first vector corresponding to the first text and a second vector corresponding to the second text. Because the vectors obtained by representing texts of different lengths have different dimensions, the representation accuracy of the first and second vectors can be improved.

[0043] S202 : In response to the dimensions of at least two vectors being unequal, adjusting the dimensions of the at least two vectors to be equal based on an adjustment rule.

[0044] In at least one embodiment of the present application, the adjustment rules may include dimension adjustment rules and position adjustment rules. The dimension adjustment rules may be used to perform dimensionality reduction processing on a vector or matrix. The dimension adjustment rules may include column vectors that need to be deleted. For example, the dimension adjustment rules may include: deleting the first column, the fourth column, and the fifth column. The position adjustment rules may be used to adjust the position of element values ​​in a vector or matrix. The position adjustment rules may include adjustment rules for column vectors. For example, the position adjustment rules may include: the position of the first column remains unchanged, and the position of the second column is swapped with the position of the third column.

[0045] The adjustment rules can be obtained through text training corresponding to different fields. The adjustment rules corresponding to different fields may be different. For example, the dimension adjustment rules corresponding to the financial field may include: deleting the first column, the fourth column, and the fifth column. The dimension adjustment rules corresponding to the medical field may include: deleting the first column, the fourth column, and the eighth column. For another example, the position adjustment rules corresponding to the financial field may include: the position of the first column remains unchanged, the position of the second column is swapped with the position of the third column. The position adjustment rules corresponding to the medical field may include: the position of the first column remains unchanged, the position of the second column is swapped with the position of the third column, the position of the fifth column is swapped with the position of the sixth column, and the position of the seventh column remains unchanged. The adjustment rules corresponding to different fields may also be the same. For example, the dimension adjustment rules corresponding to the financial field may include: deleting the first column, the fourth column, and the fifth column. The dimension adjustment rules corresponding to the e-commerce field may include: deleting the first column, the fourth column, and the fifth column.

[0046] In at least one embodiment of the present application, the electronic device identifies the field corresponding to the first text or the second text, and determines the corresponding adjustment rule based on the field corresponding to the first text or the second text. In response to the unequal dimensions of at least two vectors, the electronic device can adjust the dimensions of at least two vectors to equal dimensions based on the dimension adjustment rule in the adjustment rule. This embodiment determines the corresponding adjustment rule through the field corresponding to the first text or the second text, and can obtain a reasonable adjustment rule. By adjusting the dimensions of at least two vectors to equal dimensions through the dimension adjustment rule in the adjustment rule, it can be ensured that the adjusted second vector has the same dimension as the first vector, thereby solving the problem of being unable to determine text similarity due to the different dimensions of at least two vectors.

[0047] In some embodiments, the at least two vectors may include a first vector and a second vector. In response to the dimension of the second vector being greater than the dimension of the first vector, the electronic device may reduce the dimension of the second vector based on a dimension adjustment rule, and the dimension of the second vector after dimension reduction is equal to the dimension of the first vector. For example, if the first vector is represented as [0.18 0 0.77] and the second vector is represented as [0.459 0.166 0.242 0.995 0.879 0.601], the dimension adjustment rule includes the first column, the fourth column, and the fifth column. The dimension of the second vector is reduced by the dimension adjustment rule, and the second vector after dimension reduction is [0.1660.242 0.601].

[0048] In some embodiments, the positions of elements in the reduced-dimensional second vector are adjusted according to a position adjustment rule within the adjustment rule to obtain an adjusted second vector. Continuing with the above example, if the position adjustment rule includes: the position of the first column remains unchanged, and the positions of the second column and the third column are swapped, the positions of the elements in the reduced-dimensional second vector [0.166, 0.242, 0.601] are adjusted according to the position adjustment rule to obtain an adjusted second vector [0.166, 0.242, 0.601]. The dimension of the adjusted second vector [0.166, 0.242, 0.601] is equal to the dimension of the first vector [0.156, 0.563, 0.259].

[0049] For example, assume that the first vector has three dimensions, and the features corresponding to the three dimensions in the first vector are, in order: age, gender, and income; the second vector has six dimensions, and the features corresponding to the six dimensions in the second vector are, in order: gender, age, January income, February income, March income, and April income. By performing dimensionality reduction on the second vector, a reduced second vector is obtained. The reduced second vector has three dimensions, and the features corresponding to the three dimensions in the reduced second vector are, in order: gender, age, and March income. Obviously, the position of the feature corresponding to each dimension in the reduced second vector is different from the position of the corresponding feature in the first vector. Therefore, using a position adjustment rule, the features corresponding to each dimension in the reduced second vector can be adjusted to obtain an adjusted second vector. The features corresponding to the three dimensions in the adjusted second vector are, in order: age, gender, and March income. Therefore, the position of the feature corresponding to each dimension in the adjusted second vector is the same as the position of the corresponding feature in the first vector.

[0050] This embodiment uses the dimension adjustment rule within the adjustment rule to accurately remove duplicate information from the second vector, thereby increasing the discreteness between features in the reduced-dimensional second vector. Furthermore, by adjusting the position adjustment rule within the adjustment rule to adjust the positions of elements in the reduced-dimensional second vector, it ensures that similar features in the adjusted second vector and the first vector are in the same position, thereby improving similarity calculation.

[0051] In other embodiments, in response to the dimension of the second vector being smaller than the dimension of the first vector, the electronic device performs dimension upgrading on the second vector, the dimension of the second vector after dimension upgrading is equal to the dimension of the first vector, and adjusts the positions of the elements in the second vector after dimension upgrading to obtain the adjusted second vector. Specifically, in the process of the electronic device upgrading the second vector, the electronic device can upgrade the second vector based on the configuration value, and the configuration value can be set to 0. The actual application is not limited to this. For example, if the first vector is represented as [0.459 0.166 0.242 0.995 0.879 0.601] and the second vector is represented as [0.18 0 0.77], the electronic device performs dimension upgrading on the second vector, the dimension of the second vector after dimension upgrading is equal to the dimension of the first vector, and the second vector after dimension upgrading can be represented as [0.18 0 0.77 0 0 0]. This embodiment can avoid the loss of information in the second text in the second vector by upgrading the second vector, thereby improving the accuracy of text similarity.

[0052] In other embodiments, the electronic device adjusts the dimensions of the first vector and the second vector to predetermined dimensions, where the adjusted dimensions of the first vector are equal to the adjusted dimensions of the second vector. For example, the electronic device may simultaneously perform dimension increase on the first vector and the second vector. For another example, the electronic device may simultaneously perform dimension reduction on the first vector and the second vector. For another example, the electronic device may perform dimension increase on the first vector and dimension reduction on the second vector.

[0053] The electronic device adjusts the positions of the elements in the dimensionally adjusted first vector, and adjusts the positions of the elements in the dimensionally adjusted second vector. In one example, the electronic device may adjust the positions of the elements in the dimensionally adjusted first vector, and adjust the positions of the elements in the dimensionally adjusted second vector, based on a preset arrangement order of a plurality of features. The preset arrangement order may be set and adjusted according to actual needs. For example, the plurality of features include: age, gender, and income, and the preset arrangement order may be set to: gender, age, and income. In another example, the electronic device may adjust the positions of the elements in the dimensionally adjusted second vector based on the dimensionally adjusted first vector, so that the features in the adjusted second vector are aligned with the features in the dimensionally adjusted first vector. In another example, the electronic device may adjust the positions of the elements in the dimensionally adjusted first vector based on the dimensionally adjusted second vector, so that the features in the adjusted first vector are aligned with the features in the dimensionally adjusted second vector.

[0054] S203: Determine the text similarity between the texts based on the adjusted vectors.

[0055] In at least one embodiment of the present application, the electronic device can calculate the text similarity between the first text and the second text by means of cosine similarity, Euclidean distance, etc. The electronic device determines the top n first texts or second texts with the highest text similarity to respond. For example, in a search scenario, the first text is a user question (query), and the second text is a web page text (content). The electronic device will respond with the top n second texts with the highest text similarity to the first text. For another example, in an intelligent question-and-answer scenario, the second text is a query statement (query) input by the user, and the first text is a question text (question) in the question library. The electronic device will respond with the top n first texts with the highest text similarity to the second text.

[0056] In multiple embodiments of the present application, when the dimensions of the vectors corresponding to at least two texts are unequal, adjusting the dimensions of the at least two vectors to equal dimensions can resolve the problem of being unable to determine text similarity due to the different dimensions of the at least two vectors. At the same time, by adjusting the rules, the dimensions of the at least two vectors can be reasonably adjusted, thereby preventing the adjusted vectors from losing important information in the text and ensuring the accuracy of text similarity. Furthermore, because the dimension of the adjusted second vector is smaller than that of the first vector, the computational efficiency of text similarity can be improved.

[0057] like Figure 3 As shown in FIG, it is a flowchart of a text processing method provided by another embodiment of the present application. Figure 3 As shown, it specifically includes steps S301-S303: S301 : Characterize a plurality of first texts to obtain a first characterization matrix, and characterize a plurality of second texts to obtain a second characterization matrix.

[0058] In at least one embodiment of the present application, the text length of the second text is greater than the text length of the first text. The dimension of each row vector in the second representation matrix is ​​greater than the dimension of each row vector in the first representation matrix.

[0059] In at least one embodiment of the present application, the electronic device characterizes K first texts using a first model to obtain a characterization vector corresponding to each first text. The electronic device constructs a first characterization matrix based on the characterization vectors corresponding to the K first texts, and the number of rows of the first characterization matrix is ​​equal to K. For example, if the characterization vector corresponding to each first text is N-dimensional, the first characterization matrix with K rows and N columns is obtained by characterizing the K first texts using the first model. .

[0060] In at least one embodiment of the present application, the electronic device characterizes K second texts using a second model to obtain a characterization vector corresponding to each second text. The electronic device constructs a second characterization matrix based on the characterization vectors corresponding to the K second texts, and the number of rows of the second characterization matrix is ​​equal to K. For example, if the characterization vector corresponding to each second text is M-dimensional, the second texts are characterized using the second model to obtain a second characterization matrix with K rows and M columns. , M is greater than N.

[0061] This embodiment uses the first model and the second model to represent texts of different lengths respectively. Since the dimensions of vectors obtained by representing texts of different lengths are different, the representation accuracy of the first representation matrix and the second representation matrix can be improved.

[0062] S302: Perform matrix transformation on the first characterization matrix to obtain a third characterization matrix, and perform matrix transformation on the second characterization matrix to obtain a fourth characterization matrix.

[0063] In at least one embodiment of the present application, in order to quantify the commonality between column vectors in a first characterization matrix and column vectors in a second characterization matrix, and to quantify the commonality between every two column vectors in the second characterization matrix, the electronic device needs to perform matrix conversion on the first characterization matrix and the second characterization matrix.

[0064] In at least one embodiment of the present application, the electronic device calculates an element mean of each vector in the first representation matrix, updates the element value of each vector in the first representation matrix based on the element mean of each vector, and obtains a third representation matrix based on the updated element value of each vector.

[0065] The element mean of each vector can be the average value of the element values ​​of each column vector in the first characterization matrix. For example, the first characterization matrix is , then the element mean of the first column vector in the first characterization matrix is ​​0.65, and the element mean of the second column vector in the first characterization matrix is ​​0.3.

[0066] The electronic device compares the element value of each column vector with the corresponding element mean. The electronic device updates the element value greater than or equal to the element mean based on the first preset value, and updates the element value less than the element mean based on the second preset value. The first preset value is greater than the second preset value. For example, the first preset value can be set to 1, and the second preset value can be set to 0. Continuing with the above example, since the element value 0.5 in the first column vector is less than the element mean 0.65 of the first column vector, the element value 0.5 in the first column vector is updated to 0; since the element value 0.8 in the first column vector is greater than the element mean 0.65 of the first column vector, the element value 0.8 in the first column vector is updated to 1. In this way, the element value 0.2 in the second column vector is updated to 0, and the element value 0.4 in the second column vector is updated to 1.

[0067] The electronic device constructs a third representation matrix based on the element position of the element value in the first representation matrix and the updated element value. Can be converted into the third representation matrix .

[0068] This embodiment updates the element value of each vector by the element mean of each vector in the first characterization matrix, and can reasonably update the element value, thereby improving the rationality of the third characterization matrix.

[0069] In at least one embodiment of the present application, the manner in which the electronic device performs matrix conversion on the second characterization matrix is ​​similar to the manner in which the electronic device performs matrix conversion on the first characterization matrix, and this application will not repeat the description thereof.

[0070] S303: Determine a dimension adjustment rule according to the dimension difference between the fourth representation matrix and the third representation matrix, and the first similarity between every two vectors in the fourth representation matrix.

[0071] In at least one embodiment of the present application, every two vectors in the fourth characterization matrix include a third vector and a fourth vector, and the electronic device determines the first similarity based on the comparison result of each element value in the third vector and the element value at the corresponding position in the fourth vector, as well as the number of elements in the third vector.

[0072] The comparison result can be determined by an indicator function, and the formula for determining the comparison result can be expressed as: , Can indicate the The first column vector The element value of the element, Can indicate the The first column vector The element value of the element, Can be expressed and The comparison results.

[0073] The calculation formula of the first similarity can be expressed as: .in, It can be expressed in the fourth representation matrix The column vector and the fourth characterization matrix The first similarity of vectors, Can be expressed and The comparison result shows that the number of elements in each column vector of the fourth characterization matrix is ​​equal. It can represent the number of elements of each column vector in the fourth characterization matrix.

[0074] This embodiment can accurately quantify the similarity between every two column vectors in the fourth representation matrix by comparing the values ​​of elements at the same position in every two vectors and the number of elements in each column vector in the fourth representation matrix.

[0075] In at least one embodiment of the present application, the electronic device determines a dimensional difference between the fourth representation matrix and the third representation matrix based on the number of elements in the row vectors in the fourth representation matrix and the number of elements in the row vectors in the third representation matrix. For example, if the number of elements in the row vectors in the fourth representation matrix is ​​M and the number of elements in the row vectors in the third representation matrix is ​​N, the dimensional difference between the fourth representation matrix and the third representation matrix may be MN.

[0076] In at least one embodiment of the present application, the electronic device determines a similar vector corresponding to each vector in the fourth representation matrix based on a first similarity between every two vectors in the fourth representation matrix, where each vector in the fourth representation matrix and the corresponding similar vector constitute a first vector pair. The electronic device determines second vector pairs from the plurality of first vector pairs based on the first similarities of the plurality of first vector pairs, where the number of second vector pairs is determined based on the dimensionality difference. The electronic device determines a dimensionality adjustment rule based on the position information of any vector in the second vector pair in the fourth representation matrix.

[0077] For each vector in the fourth representation matrix, the electronic device determines another vector with the highest first similarity as the similar vector corresponding to each vector in the fourth representation matrix. For example, the first similarity between vector 1 and vector 2 is 0.8, the first similarity between vector 1 and vector 3 is 0.6, and the first similarity between vector 1 and vector 4 is 0.7. Since vector 1 has the highest first similarity with vector 2, vector 2 is determined as the similar vector to vector 1.

[0078] The number of pairs of second vector pairs is equal to the dimension difference. The second vector pairs are determined by taking the first MN first vector pairs with the highest first similarity as the second vector pairs based on the first similarity of each first vector pair, where MN can represent the dimension difference and is a positive integer. For example, if the first similarity between vector 1 and vector 2 is 0.8, the first similarity between vector 2 and vector 3 is 0.7, the first similarity between vector 5 and vector 20 is 0.6, and the first similarity between vector 5 and vector 30 is 0.5, and MN is 3, then the second vector pairs are determined to include: vector 1 and vector 2, vector 2 and vector 3, and vector 5 and vector 20.

[0079] The dimension adjustment rule may include the column vector to be deleted. The dimension adjustment rule includes the position information of any vector in the second vector pair in the fourth representation matrix. Continuing with the above example, the second vector pairs include vector 1 and vector 2, vector 2 and vector 3, and vector 5 and vector 20. The dimension adjustment rule may include the position of vector 2 in the fourth representation matrix, the position of vector 3 in the fourth representation matrix, and the position of vector 20 in the fourth representation matrix.

[0080] This embodiment accurately determines first vector pairs by using the first similarity between each two vectors in the fourth representation matrix, and then determines second vector pairs from multiple first vector pairs, thereby screening for similar features in the fourth representation matrix. By using dimensional differences, second vector pairs are determined from multiple first vector pairs, allowing for the determination of a reasonable number of second vector pairs. Furthermore, dimensionality adjustment rules are determined based on the position information of any vector in the second vector pair in the fourth representation matrix, ensuring that the representation vector obtained after subsequent dimensionality reduction based on the dimensionality adjustment rules has the same dimension as the representation vector of the short text, thereby facilitating the calculation of text similarity.

[0081] In multiple embodiments of the present application, by characterizing texts of different lengths, a first characterization matrix corresponding to a plurality of first texts and a second characterization matrix corresponding to a plurality of second texts are obtained. By performing matrix conversion on the first characterization matrix and the second characterization matrix, it is beneficial to the commonality between the column vectors in the first characterization matrix and the column vectors in the second characterization matrix, and to quantify the commonality between every two column vectors in the second characterization matrix. By using the dimensional difference between the fourth characterization matrix and the third characterization matrix, and the first similarity between every two vectors in the fourth characterization matrix, not only can it be ensured that the characterization vector obtained after subsequent dimensionality reduction based on the dimensionality adjustment rule is the same as the dimension of the characterization vector of the short text, but it can also be ensured that the useful information lost after subsequent dimensionality reduction based on the dimensionality adjustment rule is minimal, thereby improving the rationality of the dimensionality adjustment rule.

[0082] like Figure 4 As shown in FIG, it is a flowchart of a text processing method provided by another embodiment of the present application. Figure 4 As shown, it specifically includes steps S401-S405: S401 : Characterize a plurality of first texts to obtain a first characterization matrix, and characterize a plurality of second texts to obtain a second characterization matrix.

[0083] S402: Perform matrix transformation on the first characterization matrix to obtain a third characterization matrix, and perform matrix transformation on the second characterization matrix to obtain a fourth characterization matrix.

[0084] S403 : Determine a dimension adjustment rule according to the dimension difference between the fourth representation matrix and the third representation matrix, and the first similarity between every two vectors in the fourth representation matrix.

[0085] The details of steps S401 to S403 can be found above. Figure 3 The detailed description of steps S301 to S303 is not repeated here.

[0086] S404: Based on the dimension adjustment rule, reduce the dimension of each vector in the fourth representation matrix to obtain a fifth representation matrix.

[0087] In at least one embodiment of the present application, the electronic device removes the corresponding column vectors in the fourth representation matrix based on a dimensionality adjustment rule to obtain a fifth representation matrix, wherein the dimension of each row vector in the fifth representation matrix is ​​equal to the dimension of each row vector in the third representation matrix. This embodiment reduces the dimensionality of each vector in the fourth representation matrix using the dimensionality adjustment rule to ensure that the dimension of each row vector in the fifth representation matrix is ​​equal to the dimension of each row vector in the third representation matrix.

[0088] S405 : Determine a position adjustment rule corresponding to each vector in the fifth representation matrix based on a second similarity between any vector in the third representation matrix and multiple vectors in the fifth representation matrix.

[0089] In at least one embodiment of the present application, the position adjustment rule may include position adjustment rules corresponding to all column vectors in the fifth characterization matrix.

[0090] In at least one embodiment of the present application, the electronic device calculates a second similarity between any vector in the third representation matrix and multiple vectors in the fifth representation matrix based on a comparison result between any vector in the third representation matrix and multiple vectors in the fifth representation matrix, as well as the number of elements in any vector in the third representation matrix. The electronic device uses the vector in the fifth representation matrix corresponding to the highest second similarity as the target vector corresponding to any vector in the third representation matrix. The electronic device determines a position adjustment rule corresponding to the target vector based on the position of the any vector in the third representation matrix and the position of the target vector in the fifth representation matrix.

[0091] The calculation method of the second similarity is similar to the calculation method of the first similarity, and this application will not repeat the description. The position adjustment rule corresponding to the target vector can be: adjust the position of the target vector in the fifth representation matrix to the position of any vector in the third representation matrix.

[0092] For example, first, determine any vector from the third representation matrix in sequence: the first column vector in the third representation matrix , the plurality of vectors in the fifth representation matrix include: N column vectors in the fifth representation matrix. The electronic device calculates the column vector The second similarity with the N column vectors in the fifth representation matrix. If the 5th column vector in the fifth representation matrix and the first column vector in the third representation matrix Most similar, determine the column vector As a column vector The target vector of (any vector in the third representation matrix). The electronic device is based on the column vector The position in the third representation matrix, and the column vector The position in the fifth representation matrix determines the column vector The corresponding position adjustment rule is: adjust from the 5th column position to the 1st column position in the fifth representation matrix.

[0093] Furthermore, the electronic device sequentially determines any vector from the third characterization matrix as: the second column vector in the third characterization matrix , the multiple vectors in the fifth characterization matrix include: The electronic device calculates any vector (column vector) in the third representation matrix ) and multiple vectors in the fifth characterization matrix (except for the fifth characterization matrix If the 7th column vector in the fifth representation matrix and the second column vector in the third representation matrix Most similar, determine the column vector As a column vector The target vector of (any vector in the third representation matrix). The electronic device is based on the column vector The position in the third representation matrix, and the column vector The position in the fifth representation matrix determines the column vector The corresponding position adjustment rule is: adjust from the 7th column position to the 2nd column position in the fifth representation matrix.

[0094] Similarly, the position adjustment rules corresponding to all column vectors in the fifth representation matrix are determined.

[0095] This embodiment calculates the second similarity between any vector in the third representation matrix and multiple vectors in the fifth representation matrix, thereby avoiding repeated second similarity calculations for vectors for which position adjustment rules have already been determined. By using the vector in the fifth representation matrix corresponding to the highest second similarity as the target vector for any vector in the third representation matrix, and determining the position adjustment rule corresponding to the target vector based on the position of any vector in the third representation matrix and the position of the target vector in the fifth representation matrix, similar column vectors can be placed in the same position in different matrices, thereby improving the accuracy of subsequent text similarity calculations.

[0096] In multiple embodiments of the present application, by reducing the dimensionality of each vector in the fourth representation matrix through a dimensionality adjustment rule, it is possible to not only ensure that the fifth representation matrix has the same dimension as the third representation matrix, but also to ensure that the fifth representation matrix loses minimal useful information, thereby improving the computational efficiency of text similarity while ensuring the accuracy of text similarity. By determining the position adjustment rule corresponding to each vector in the fifth representation matrix through the second similarity between any vector in the third representation matrix and multiple vectors in the fifth representation matrix, it is possible to ensure that similar features are in the same position in different vectors or matrices after subsequent position adjustment based on the position adjustment rule, thereby improving the computational accuracy of subsequent text similarity.

[0097] like Figure 5 As shown in FIG, it is a flowchart of a text processing method provided by another embodiment of the present application. Figure 5 As shown, it specifically includes steps S501-S509: S501: Use a first training text to train a language model to obtain a first model, and use a second training text to train the language model to obtain a second model.

[0098] In at least one embodiment of the present application, the electronic device uses the first training text to adjust the network parameters in the language model until the preset conditions are met to obtain the first model, wherein the preset conditions may include but are not limited to: the number of times the network parameters in the language model are adjusted is greater than the preset number, and the representation accuracy of the adjusted language model is greater than the preset accuracy. The method for determining the representation accuracy of the adjusted language model may include: representing the first training text based on the adjusted language model to obtain a representation vector corresponding to the first training text, and calculating the representation accuracy based on the representation vector and the standard vector corresponding to the first training text. The standard vector corresponding to the first training text may be a vector set by the user, and the calculation formula for the representation accuracy may be: ,in, It can be expressed as the characterization accuracy, It can represent the number of identical elements at the same position in the representation vector and the standard vector. It can represent the number of elements in a representation vector or a standard vector.

[0099] In at least one embodiment of the present application, the length of the first training text is different from the length of the second training text. The electronic device uses the second training text to adjust the network parameters in the language model to obtain a second model. The electronic device uses the second training text to adjust the network parameters in the language model in a manner similar to the manner in which the electronic device uses the first training text to adjust the network parameters in the language model, and this application will not repeat this description.

[0100] S502: Using a first model, characterize a plurality of first texts to obtain a first characterization matrix, and using a second model, characterize a plurality of second texts to obtain a second characterization matrix.

[0101] S503: Perform matrix transformation on the first characterization matrix to obtain a third characterization matrix, and perform matrix transformation on the second characterization matrix to obtain a fourth characterization matrix.

[0102] S504: Determine a dimension adjustment rule according to the dimension difference between the fourth representation matrix and the third representation matrix, and the first similarity between every two vectors in the fourth representation matrix.

[0103] S505 : Based on the dimension adjustment rule, reduce the dimension of each vector in the fourth representation matrix to obtain a fifth representation matrix.

[0104] S506: Determine a position adjustment rule corresponding to each vector in the fifth representation matrix based on the second similarity between any vector in the third representation matrix and multiple vectors in the fifth representation matrix.

[0105] The details of steps S502 to S506 can be found above. Figure 4 The detailed description of steps S401 to S405 is not repeated here.

[0106] S507: Represent at least two texts as corresponding vectors.

[0107] S508 : In response to the dimensions of the at least two vectors being unequal, adjusting the dimensions of the at least two vectors to be equal based on an adjustment rule.

[0108] S509: Determine the text similarity between at least two texts based on the adjusted vector.

[0109] The details of steps S507 to S509 can be found above. Figure 2 The detailed description of steps S201 to S203 is not repeated here.

[0110] This embodiment fine-tunes the language model through different training texts, and can quickly obtain the first model and the second model.

[0111] like Figure 6 The figure shows a functional module diagram of a text processing device provided by an embodiment of the present application. The text processing device 11 includes a representation unit 110, an adjustment unit 111, a determination unit 112, and a conversion unit 113. The module / unit referred to in the present application refers to a type of unit that can be processed by a processor (e.g. Figure 7 The processor 1001 shown in FIG. 100 is obtained and is capable of performing a series of computer-readable instruction segments that are stored in a memory (eg, Figure 7 1002).

[0112] The representation unit 110 is used to represent at least two texts as corresponding vectors; the adjustment unit 111 is used to adjust the dimensions of the at least two vectors to equal dimensions based on an adjustment rule in response to the unequal dimensions of the at least two vectors; and the determination unit 112 is used to determine the text similarity between the at least two texts based on the adjusted vectors.

[0113] In one embodiment, the adjustment rules include dimension adjustment rules and position adjustment rules. The at least two vectors include a first vector and a second vector. The adjustment unit 111 is specifically used to: in response to the dimension of the second vector being greater than the dimension of the first vector, reduce the dimension of the second vector based on the dimension adjustment rule, and the dimension of the second vector after dimension reduction is equal to the dimension of the first vector; according to the position adjustment rule, adjust the position of the elements in the second vector after dimension reduction to obtain the adjusted second vector.

[0114] In one embodiment, in response to the fact that the dimensions of at least two vectors are not equal, the characterization unit 110 is further used to characterize multiple first texts to obtain a first characterization matrix, and to characterize multiple second texts to obtain a second characterization matrix, wherein the dimension of each row vector in the second characterization matrix is ​​greater than the dimension of each row vector in the first characterization matrix; the conversion unit 113 is used to perform matrix conversion on the first characterization matrix to obtain a third characterization matrix, and to perform matrix conversion on the second characterization matrix to obtain a fourth characterization matrix; the determination unit 112 is further used to determine a dimension adjustment rule based on the dimension difference between the fourth characterization matrix and the third characterization matrix, and the first similarity between each two vectors in the fourth characterization matrix.

[0115] In one embodiment, the conversion unit 113 is specifically configured to: calculate an element mean of each vector in the first characterization matrix; update an element value of each vector in the first characterization matrix based on the element mean of each vector; and obtain a third characterization matrix based on the updated element value of each vector.

[0116] In one embodiment, the determination unit 112 is specifically used to: determine, based on the first similarity, a similar vector corresponding to each vector in the fourth characterization matrix, where each vector in the fourth characterization matrix and the corresponding similar vector constitute a first vector pair; determine, based on the first similarities of the multiple first vector pairs, multiple second vector pairs from the multiple first vector pairs, where the number of the second vector pairs is determined based on the dimension difference; and determine a dimension adjustment rule based on the position information of any vector in the second vector pair in the fourth characterization matrix.

[0117] In one embodiment, every two vectors in the fourth representation matrix include a third vector and a fourth vector, and the determination unit 112 is further configured to determine the first similarity based on a comparison result of each element value in the third vector with an element value at a corresponding position in the fourth vector, and the number of elements in the third vector.

[0118] In one embodiment, the adjustment rule includes a position adjustment rule. The adjustment unit 111 is further used to perform dimension adjustment on each vector in the fourth representation matrix based on the dimension adjustment rule to obtain a fifth representation matrix. The determination unit 112 is further used to determine the position adjustment rule corresponding to each vector in the fifth representation matrix based on the second similarity between any vector in the third representation matrix and multiple vectors in the fifth representation matrix.

[0119] In one embodiment, the determination unit 112 is configured to: calculate a second similarity between any vector in the third representation matrix and multiple vectors in the fifth representation matrix based on a comparison result between any vector in the third representation matrix and multiple vectors in the fifth representation matrix, and the number of elements in any vector in the third representation matrix; use the vector in the fifth representation matrix corresponding to the highest second similarity as the target vector corresponding to any vector in the third representation matrix; and determine a position adjustment rule corresponding to the target vector based on the position of any vector in the third representation matrix and the position of the target vector in the fifth representation matrix.

[0120] In one embodiment, at least two vectors include a first vector and a second vector, and the adjustment unit 111 is specifically used to: in response to the dimension of the second vector being smaller than the dimension of the first vector, upgrade the second vector, so that the dimension of the upgraded second vector is equal to the dimension of the first vector, and adjust the positions of the elements in the upgraded second vector to obtain an adjusted second vector; or adjust the dimensions of the first vector and the dimensions of the second vector to preset dimensions, adjust the positions of the elements in the dimension-adjusted first vector, and adjust the positions of the elements in the dimension-adjusted second vector.

[0121] In various embodiments of the present application, when the dimensions of the vectors corresponding to at least two texts are unequal, adjusting the dimensions of the at least two vectors to equal dimensions can resolve the problem of being unable to determine text similarity due to the different dimensions of the at least two vectors. Furthermore, by adjusting the rules, the dimensions of the at least two vectors can be reasonably adjusted, thereby preventing the adjusted vectors from losing important information from the text and ensuring the accuracy of text similarity.

[0122] Figure 7 is a schematic diagram of the structure of an electronic device for implementing a text processing method provided by an embodiment of the present application. Figure 7 The electronic device 100 is used to perform Figure 2-Figure 5 The method shown.

[0123] The electronic device 100 includes at least one processor 1001 , a memory 1002 , and at least one network interface 1003 .

[0124] The processor 1001 is, for example, a general-purpose central processing unit (CPU), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the solution of the present application. For example, the processor 1001 includes an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0125] The memory 1002 may be, for example, a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Optionally, the memory 1002 exists independently and is connected to the processor 1001 via the internal connection 1004. Alternatively, the memory 1002 and the processor 1001 may be integrated together.

[0126] The network interface 1003 uses any transceiver-like device for communicating with other devices or a communication network. For example, the network interface 1003 includes at least one of a wired network interface and a wireless network interface. For example, the wired network interface is an Ethernet interface. For example, the Ethernet interface is an optical interface, an electrical interface, or a combination thereof. For example, the wireless network interface is a wireless local area network (WLAN) interface, a cellular network interface, or a combination thereof.

[0127] In some embodiments, the processor 1001 includes one or more CPUs, such as Figure 7 CPU0 and CPU1 are shown in the figure.

[0128] In some embodiments, the electronic device 100 optionally includes multiple processors, such as Figure 7 1 and 1005 are shown in FIG. Each of these processors is, for example, a single-CPU or a multi-CPU. A processor herein optionally refers to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0129] In some embodiments, electronic device 100 further includes internal connections 1004. Processor 1001, memory 1002, and at least one network interface 1003 are connected via internal connections 1004. Internal connections 1004 include pathways for transmitting information between these components. Optionally, internal connections 1004 are single boards or buses. Internal connections 1004 may be divided into address buses, data buses, control buses, and the like.

[0130] In some embodiments, the electronic device 100 further includes an input / output interface 1006 . The input / output interface 1006 is connected to the internal connection 1004 .

[0131] Optionally, the processor 1001 implements the method in the above embodiment by reading the program code 1010 stored in the memory 1002, or the processor 1001 implements the method in the above embodiment by using the program code stored internally. In the case where the processor 1001 implements the method in the above embodiment by reading the program code 1010 stored in the memory 1002, the memory 1002 stores the program code that implements the method provided in the embodiment of the present application.

[0132] For more details on how the processor 1001 implements the above functions, please refer to the descriptions in the previous method embodiments, which will not be repeated here.

[0133] This embodiment further provides a computer storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the text processing method in the above-mentioned embodiment.

[0134] This embodiment further provides a computer program product. When the computer program product is run on an electronic device, the electronic device is caused to execute the above-mentioned related steps to implement the text processing method in the above-mentioned embodiment.

[0135] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the text processing method in the above-mentioned method embodiments.

[0136] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0137] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0139] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0140] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0141] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0142] The above are only specific embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A text processing method, characterized in that: The method comprises: Representing at least two texts as corresponding vectors; In response to at least two of the vectors having unequal dimensions, adjusting the dimensions of the at least two vectors to equal dimensions based on an adjustment rule; The text similarity between the at least two texts is determined according to the adjusted vector.

2. The text processing method according to claim 1, characterized in that: The adjustment rule includes a dimension adjustment rule and a position adjustment rule, the at least two vectors include a first vector and a second vector, and adjusting the dimensions of the at least two vectors to equal dimensions based on the adjustment rule includes: In response to the dimension of the second vector being greater than the dimension of the first vector, based on the dimension adjustment rule, reducing the dimension of the second vector, so that the dimension of the reduced second vector is equal to the dimension of the first vector; According to the position adjustment rule, the positions of the elements in the second vector after dimensionality reduction are adjusted to obtain an adjusted second vector.

3. The text processing method according to claim 2, characterized in that: Before responding to the fact that the dimensions of at least two vectors are unequal, the method further comprises: Characterizing the plurality of first texts to obtain a first characterization matrix, and characterizing the plurality of second texts to obtain a second characterization matrix, wherein the dimension of each row vector in the second characterization matrix is ​​greater than the dimension of each row vector in the first characterization matrix; Performing a matrix transformation on the first characterization matrix to obtain a third characterization matrix, and performing a matrix transformation on the second characterization matrix to obtain a fourth characterization matrix; The dimension adjustment rule is determined according to a dimension difference between the fourth representation matrix and the third representation matrix, and a first similarity between every two vectors in the fourth representation matrix.

4. The text processing method according to claim 3, characterized in that: The performing matrix transformation on the first characterization matrix to obtain a third characterization matrix includes: Calculating the element mean of each vector in the first representation matrix; Based on the element mean of each vector, updating the element value of each vector in the first representation matrix; The third representation matrix is ​​obtained based on the updated element values ​​of each vector.

5. The text processing method according to claim 3, characterized in that: The determining the dimension adjustment rule according to the dimension difference between the fourth representation matrix and the third representation matrix, and the first similarity between every two vectors in the fourth representation matrix, includes: Determine, based on the first similarity, a similar vector corresponding to each vector in the fourth representation matrix, wherein each vector in the fourth representation matrix and the corresponding similar vector constitute a first vector pair; determining a plurality of second vector pairs from the plurality of first vector pairs according to first similarities of the plurality of first vector pairs, wherein the number of the second vector pairs is determined based on the dimensional differences; The dimension adjustment rule is determined according to position information of any vector in the second vector pair in the fourth representation matrix.

6. The text processing method according to claim 3, characterized in that: Every two vectors in the fourth characterization matrix include a third vector and a fourth vector, and the method further includes: The first similarity is determined according to a comparison result between each element value in the third vector and the element value at a corresponding position in the fourth vector, and the number of elements in the third vector.

7. The text processing method according to any one of claims 3 to 6, characterized in that: The method further comprises: Based on the dimension adjustment rule, dimension adjustment is performed on each vector in the fourth representation matrix to obtain a fifth representation matrix; Based on a second similarity between any vector in the third representation matrix and multiple vectors in the fifth representation matrix, a position adjustment rule corresponding to each vector in the fifth representation matrix is ​​determined.

8. The text processing method according to claim 7, characterized in that: The determining, based on the second similarity between any vector in the third representation matrix and multiple vectors in the fifth representation matrix, a position adjustment rule corresponding to each vector in the fifth representation matrix includes: Calculating a second similarity between any vector in the third representation matrix and multiple vectors in the fifth representation matrix based on a comparison result between any vector in the third representation matrix and multiple vectors in the fifth representation matrix, and the number of elements of any vector in the third representation matrix; Taking the vector in the fifth representation matrix corresponding to the highest second similarity as the target vector corresponding to any vector in the third representation matrix; Based on the position of the any one vector in the third representation matrix and the position of the target vector in the fifth representation matrix, a position adjustment rule corresponding to the target vector is determined.

9. The text processing method according to claim 1, wherein: The at least two vectors include a first vector and a second vector, and adjusting the dimensions of the at least two vectors to be equal based on the adjustment rule includes: In response to the dimension of the second vector being smaller than the dimension of the first vector, upscaling the second vector so that the dimension of the upscaled second vector is equal to the dimension of the first vector, and adjusting positions of elements in the upscaled second vector to obtain an adjusted second vector; or The dimensions of the first vector and the second vector are adjusted to predetermined dimensions, positions of elements in the dimension-adjusted first vector are adjusted, and positions of elements in the dimension-adjusted second vector are adjusted.

10. A text processing device, characterized in that: The device comprises: Representation unit, used to represent each text into a corresponding vector; an adjusting unit, configured to adjust the dimensions of the at least two vectors to be equal based on an adjustment rule in response to the dimensions of the at least two vectors being unequal; The determining unit is configured to determine the text similarity between the texts based on the adjusted vectors.

11. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the text processing method according to any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the text processing method according to any one of claims 1 to 9 is implemented.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the text processing method according to any one of claims 1 to 9 is implemented.

Images