Data processing method, apparatus, device, and medium

Abstract

Embodiments of the application disclose a data processing method and device, equipment and medium, which are applied to the technical field of data processing. The method comprises the following steps: obtaining a sample text pair, determining a first sample word segmentation set matched with a second sample word segmentation set from a first sample word segmentation set, and generating word weight marking information of each first sample word segmentation in the first sample word segmentation set according to the first sample word segmentation matched with the second sample word segmentation set; inputting each first sample word segmentation into an initial word segmentation processing model in sequence, performing word segmentation processing on each first sample word segmentation by the initial word segmentation processing model to obtain word weight prediction information of each first sample word segmentation; and iteratively training the initial word segmentation processing model through the word weight prediction information and the word weight marking information to obtain a target word segmentation processing model used for determining the word weight of text word segmentation. By adopting the embodiments of the application, the determination accuracy of the word weight of text word segmentation can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device and medium. Background Technology

[0002] Text matching is a very typical task, such as search recall and ranking, and retrieval-based question answering. Essentially, these tasks all fall under the category of text matching: given a text as a query, the task is to match the most relevant documents or answers and return them to the user. The matching degree between two texts can be determined by combining the word weights of text segmentation. Current methods for determining word weights typically rely on the co-occurrence statistics of text segments within the text. However, these co-occurrence statistics are independent of the text segmentation itself. For example, if a text segment is assigned a weight of 0.5 based on co-occurrence statistics, and this segment is replaced with another unrelated segment, the weight of the replaced segment will also remain at 0.5 because the co-occurrence statistics remain unchanged. In other words, the word weights determined by current methods do not reflect the relative importance of different text segments within the text. Therefore, improving the accuracy of determining the word weights of text segmentation is a pressing issue. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and medium that can improve the accuracy of determining word weights in text segmentation.

[0004] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0005] Obtain sample text pairs; each sample text pair includes a first sample text and a second sample text that matches the first sample text; the first sample text is the recall text in the target key domain, and the second sample text is the query text in the target key domain; the first sample text is associated with a first sample word segmentation set, and the second sample text is associated with a second sample word segmentation set; the first sample word segmentation set includes at least one first sample word segmentation; the second sample word segmentation set includes at least one second sample word segmentation.

[0006] First sample word segments that match the second sample word segment set are determined from the first sample word segment set, and word weight annotation information of each first sample word segment in the first sample word segment set is generated based on the first sample word segments that match the second sample word segment set.

[0007] Each first sample word segment is sequentially input into the initial word segmentation processing model, which then performs word segmentation processing on each first sample word segment to obtain the word weight prediction information for each first sample word segment.

[0008] The initial word segmentation model is iteratively trained using word weight prediction information and word weight annotation information to obtain the target word segmentation model used to determine the word weights of text segmentation.

[0009] On one hand, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0010] The acquisition module is used to acquire sample text pairs; the sample text pair includes a first sample text and a second sample text that matches the first sample text; the first sample text is the recall text in the target key domain, and the second sample text is the query text in the target key domain; the first sample text is associated with a first sample word segmentation set, and the second sample text is associated with a second sample word segmentation set; the first sample word segmentation set includes at least one first sample word segmentation; the second sample word segmentation set includes at least one second sample word segmentation.

[0011] The processing module is used to determine the first sample word segment that matches the second sample word segment from the first sample word segment set, and generate word weight annotation information for each first sample word in the first sample word segment set based on the first sample word segment that matches the second sample word segment set;

[0012] The processing module is also used to input each first sample word segment into the initial word segmentation processing model in sequence, and the initial word segmentation processing model performs word segmentation processing on each first sample word segment to obtain the word weight prediction information of each first sample word segment.

[0013] The processing module is also used to iteratively train the initial word segmentation processing model using word weight prediction information and word weight annotation information to obtain the target word segmentation processing model used to determine the word weights of text segmentation.

[0014] On one hand, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute some or all of the steps in the above method.

[0015] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, are used to perform some or all of the steps in the above-described method.

[0016] Accordingly, according to one aspect of this application, a computer program product or computer program is provided, which includes computer instructions that, when executed by a processor, can implement some or all of the steps in the above-described method.

[0017] In this embodiment, sample text pairs can be obtained; each sample text pair includes a first sample text and a second sample text under the target key domain, wherein the first sample text is the recall text and the second sample text is the query text; first sample words that match the second sample word set are determined from the first sample word set, and word weight annotation information for each first sample word in the first sample word set is generated based on the first sample word that matches the second sample word set; this word weight annotation information can effectively combine the relevant matching information between the first sample text and the second sample text; each first sample word is sequentially input into the initial word segmentation processing model. The initial word segmentation model segments each first sample word to obtain word weight prediction information. The initial word segmentation model is then iteratively trained using the word weight prediction information and word weight annotation information to obtain the target word segmentation model for determining the word weights of the text. This method can determine the word weights of recalled text in a specific domain. Furthermore, when training the initial word segmentation model, it can incorporate not only relevant domain information of the target key domain but also relevant feature information within the first sample text, thereby improving the accuracy of the target word segmentation model in determining the word weights of the text. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating a data processing scenario provided in an embodiment of this application;

[0020] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of a word weighting annotation scenario provided in an embodiment of this application;

[0022] Figure 4 A schematic diagram illustrating a training scenario for an initial word segmentation processing model provided in an embodiment of this application;

[0023] Figure 5 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0024] Figure 6 This is a schematic diagram illustrating an application scenario of word weighting provided in an embodiment of this application;

[0025] Figure 7This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0026] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0028] The data processing method proposed in this application is implemented in an electronic device, which can be a server or a terminal. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, etc., but is not limited to these.

[0029] A schematic diagram of a data processing scenario proposed based on this data processing method can be shown as follows: Figure 1 As shown, Figure 1 A network architecture is proposed, which may include a service server and a user terminal cluster. The user terminal cluster may include one or more user terminals; the number of user terminals in the cluster is not limited. Communication connections can exist between the user terminals in the cluster. Simultaneously, any user terminal in the cluster can have a communication connection with the service server, enabling each user terminal in the cluster to interact with the service server via this connection. The communication connection method is not limited; it can be established directly or indirectly via wired communication, wireless communication, or other methods. This application does not impose any limitations on this method. Furthermore, it is understood that the electronic devices involved in the embodiments of this application may be… Figure 1 The business server shown can also be Figure 1 Any user terminal in the user terminal cluster shown.

[0030] For example, in this embodiment, the server can obtain sample text pairs, which include a first sample text and a second sample text matching the first sample text. The first sample text is associated with a first sample word segmentation set, and the second sample text is associated with a second sample word segmentation set. The first sample word segmentation set includes at least one first sample word, and the second sample word segmentation set includes at least one second sample word. First sample words matching the second sample word segmentation set are determined from the first sample word segmentation set, and word weight annotation information for each first sample word in the first sample word segmentation set is generated based on these matching words. Each first sample word is sequentially input into an initial word segmentation processing model, which performs word segmentation processing on each first sample word to obtain word weight prediction information for each first sample word. Based on the word weight prediction information and word weight annotation information, the initial word segmentation processing model is iteratively trained to obtain a target word segmentation processing model. This target word segmentation processing model can be used to determine the word weights of text segments in a text.

[0031] In this model, the first sample text can be the recall text within the target key domain, and the second sample text can be the query text within the target key domain. Therefore, word weights can be assigned to the recall text using the query text. This allows the trained target word segmentation model to determine the word weights of the text segments within the recall text, thereby achieving word weight determination for specific domain-specific text segmentation. Furthermore, it addresses the issue of different word weights arising from the varying semantic representations of different text segments within the recall text. This improves the accuracy of word weight determination and accurately represents the importance of each text segment within the recall text.

[0032] Optionally, in some embodiments, the electronic device can execute the data processing method according to actual business needs to accurately obtain the word weights of text segmentation. The technical solution of this application can be applied to text word weight determination scenarios in any specific field. For example, the target key field can be e-commerce, where the recall text can be a product title, and the query text can be a query text used to search for e-commerce products (such as a user's social media posts). The technical solution of this application can be used to train a target word segmentation processing model, and the trained target word segmentation processing model can determine the word weight of each word in the product title. As another example, the target key field can be medical, where the recall text can be a summary of medical literature, and the query text can be a query text used to search for medical literature (such as professional terms related to medicine, text information composed of symptoms, etc.). The technical solution of this application can be used to train a target word segmentation processing model, and the trained target word segmentation processing model can determine the word weight of each word in the product title.

[0033] It is understood that the word weights determined by the text segmentation in this application can be applied to any business scenario. For example, in a product recommendation scenario, the word weights of the product title text segmentation can be used to determine products that match the query text and recommend them to the user. Similarly, it can be applied to information search scenarios, such as using the word weights of the summary text to determine medical literature that matches the query text and identifying the medical literature as the searched matching text, and so on. The application scenarios for the target key domain and word weights are not limited here.

[0034] It should be noted that in specific embodiments of this application, when scenarios involving the acquisition of user information and related data, such as acquiring first or second target text uploaded by the user, user permission or consent is required. That is, when the embodiments of this application are applied to specific products or technologies, the collection, use, and processing of relevant user data comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, prompts can be issued through an interactive interface to indicate what data will be collected or acquired. Specifically, the types and content of this data can be presented to the user through lists or other means. Further data collection and processing will only proceed after a confirmation or instruction to allow data collection is received on the interactive interface.

[0035] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0036] Based on the foregoing description, this application proposes a data processing method that can be executed by the aforementioned electronic device. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application.

[0037] like Figure 2 As shown, the data processing method of this application embodiment may include the following flow:

[0038] S101. Obtain sample text pairs.

[0039] The sample text pairs include a first sample text and a second sample text that matches the first sample text. The first sample text is the recall text within the target key domain, and the second sample text is the query text within the target key domain. It can be understood that the target key domain can be any specific business domain, such as e-commerce or healthcare. The recall text refers to the text that needs to be matched when performing a search query based on the query text.

[0040] It can be understood that recall text refers to text defined according to certain specifications within a target key domain. For example, when the target key domain is e-commerce, a search query can retrieve e-commerce products that match the query text. The recall text can be the product title, and matching the query text and the recall text can identify the e-commerce products that match the query text. Similarly, when the target key domain is healthcare, a search query can retrieve medical literature that matches the query text. The recall text can be the abstract of the medical literature, and matching the query text and the recall text can identify the medical literature that matches the query text. There are no restrictions on the specific target key domain or the specific type of recall text. Furthermore, the query text can be any business text related to the target key domain. For example, it can be the search text entered by the user (also known as a query term), or it can be user-generated content (UGC), such as a user's social media posts. There are no restrictions on the specific type of query text.

[0041] The first sample text is associated with a first sample word segmentation set, and the second sample text is associated with a second sample word segmentation set. The first sample word segmentation set includes at least one first sample word segmentation; the second sample word segmentation set includes at least one second sample word segmentation. For example, the first sample text can be segmented to obtain the first sample word segmentation set, and the second sample text can be segmented to obtain the second sample word segmentation set.

[0042] S102. Determine the first sample word segment that matches the second sample word segment from the first sample word segment set, and generate word weight annotation information for each first sample word in the first sample word segment set based on the first sample word segment that matches the second sample word segment set.

[0043] In some embodiments, determining the first sample word that matches the second sample word set can be done by selecting any first sample word in the first sample word set that matches any second sample word in the second sample word set as the first sample word that matches the second sample word set. Determining whether a first sample word and a second sample word match can be done by obtaining the word feature vectors of the first sample word and the second sample word, and determining the vector similarity between the word feature vectors of the first and second sample words; if the vector similarity is greater than or equal to a similarity threshold, then the first sample word and the second sample word are determined to match; if the vector similarity is less than the similarity threshold, then the first sample word and the second sample word are determined to not match.

[0044] In some embodiments, determining the first sample word that matches the second sample word set may involve obtaining key domain word attributes associated with the target key domain, selecting a first target sample word whose word attributes are key domain word attributes from the first sample word set, and selecting a second target sample word whose word attributes are key domain word attributes from the second sample word set; if the word semantics represented by the first target sample word matches the word semantics represented by the second target sample word, then the first target sample word is determined as the first sample word that matches the second target sample word in the second sample word set.

[0045] Among them, the key domain word segmentation attributes are related to the target key domain, and the key domain word segmentation attributes associated with different target key domains are different. For example, if the target key domain is e-commerce, the key domain word segmentation attributes can be: product category attributes, product brand attributes, product modification attributes, product long-tail attributes, etc. Specifically, text segmentation with the product category attribute represents the category of a product; text segmentation with the product brand attribute represents the brand of a product; text segmentation with the product modification attribute represents the modification of the product; and text segmentation with the product long-tail attribute represents the long-tail words of the product. For example, the sample text is: "XX brand's sales of high-top casual shoes released in spring and autumn have decreased." The sample word segmentation of the sample text is: XX brand, targeting, spring and autumn, season, release, of, high-top, casual shoes, sales volume, decrease. The sample word segmentation with the word segmentation attribute as product category attribute is casual shoes, the sample word segmentation with the word segmentation attribute as product brand attribute is XX brand, the sample word segmentation with the word segmentation attribute as product modifier attribute is high-top, and the sample word segmentation with the word segmentation attribute as product long tail attribute is spring and autumn.

[0046] For example, if the target key domain is the medical field, the key domain word segmentation attributes can be: medical category attribute, medical professional term attribute, medical symptom term attribute, medical long-tail attribute, etc. Specifically, text segmentation with the medical category attribute represents a medical category (e.g., internal medicine, surgery); text segmentation with the medical professional term attribute represents a medical professional term (e.g., E. coli); and text segmentation with the medical symptom term attribute represents a medical professional term (e.g., horseshoe kidney). The key domain word segmentation attributes differ for different target key domains and can be set by relevant business personnel based on the actual key domain. No restrictions are imposed here.

[0047] It's understandable that text segments with key domain segmentation attributes are considered relatively important within a given text. Therefore, the matching degree between two texts can be determined based on these key domain segmentation attributes. In other words, the trained target segmentation model can not only identify text segments associated with key domain segmentation attributes (i.e., text segments with key domain segmentation attributes) but also assign corresponding weights to the acquired text segments.

[0048] Specifically, a pre-trained word segmentation attribute determination model can predict the word segmentation attributes of text segments in the first and second sample word segmentation sets to determine the word segmentation attributes of each first sample word and each second sample word. The word segmentation attributes of a sample word can include key domain word segmentation attributes and non-key domain word segmentation attributes. When a sample word's word segmentation attribute is a key domain word segmentation attribute, it can be further subdivided into more detailed key domain word segmentation attributes, such as product modification attributes. Alternatively, a pre-built domain keyword library associated with the target key domain can be used to determine the word segmentation attributes of each first and second sample word. For example, the domain keyword library includes text segments associated with product modification attributes, text segments associated with product long-tail attributes, etc. By performing word segmentation queries on the first sample word (or second sample word) in the domain keyword library, the word segmentation attributes of the first sample word (or second sample word) can be determined. For example, if a sample word segmentation has the same word segmentation as other text segments associated with product modification attributes, then the word segmentation attribute of that sample word segmentation is determined to be a product modification attribute. No restrictions are placed on the method for determining the word segmentation attribute of a sample word segmentation.

[0049] In some embodiments, since the first sample text and the second sample text are two sample texts that match under the target key domain, it can be considered that the first target sample word segmentation with the key domain segmentation attribute in the first sample text will contain sample texts that match the second target sample word segmentation. The second target sample word segmentation is the second sample word segmentation with the key domain segmentation attribute in the second sample text. Specifically, the segmentation semantics represented by the first target sample word segmentation and the segmentation semantics represented by the second target sample word segmentation can be obtained. When the segmentation semantics of the first target sample word segmentation and the segmentation semantics of the second target sample word segmentation match, and the key domain segmentation attribute of the first target sample word segmentation matches the key domain segmentation attribute of the second target sample word segmentation, it indicates that the first target sample word segmentation and the second target sample word segmentation are two sample word segments that match under the target key domain.

[0050] In other words, if the semantic meaning of the first target sample word determined from the first sample word segmentation set is the same as the semantic meaning of any second target sample word determined from the second sample word segmentation set, and the key domain word segmentation attribute of the first target sample word is the same as the key domain word segmentation attribute of any second target sample word, then the first target sample word is determined to be the first sample word that matches the second target sample word in the second sample word segmentation set.

[0051] It can be understood that the word segmentation semantics can refer to the word segmentation semantic features obtained by processing the target sample word segmentation semantic features through a pre-trained semantic feature extraction model. The matching of the word segmentation semantics of two target sample word segments can be determined by assessing the feature correlation between their semantic features. For example, by processing the semantic features of the first target sample word segmentation using a semantic feature extraction model to obtain the first word segmentation semantic features, and processing the semantic features of the second target sample word segmentation to obtain the second word segmentation semantic features, based on the feature correlation between the first and second word segmentation semantic features, if the feature correlation is greater than or equal to a preset threshold, then the first and second target sample word segments are determined to match; if the feature correlation is less than the preset threshold, then the first and second target sample word segments are determined to be mismatched.

[0052] Alternatively, the semantic segmentation can refer to the standard sample segmentation obtained by standardizing the target sample segmentation. For example, a labeled keyword library associated with the target key domain can be defined. This library records a standard sample segmentation and its corresponding extended sample segmentation. The extended sample segmentation is a different description of the standard sample segmentation. For example, if the labeled sample segmentation is "apple," the extended sample segmentation could include: "apple," "apple flavor," "red apple," etc. Therefore, the first target sample segmentation can be determined from the labeled keyword library, and the second target sample segmentation can be determined from the labeled keyword library, and if the first and second standard sample segmentations are the same, they are considered to be a match.

[0053] It is understandable that the first target sample word segmentation (or the second target sample word segmentation) can first determine whether a standard sample word segmentation exists in the labeled keyword library to serve as the word segmentation semantics. If no standard sample word segmentation exists, semantic features are extracted using a pre-trained semantic feature extraction model to serve as the word segmentation semantics. Alternatively, the first target sample word segmentation (or the second target sample word segmentation) can first determine whether a standard sample word segmentation exists in the labeled keyword library, and then extract semantic features using a pre-trained semantic feature extraction model. Both the determined standard sample word segmentation and the extracted word segmentation semantic features are used as the corresponding word segmentation semantics. When determining whether the first target sample word segmentation and the second target sample word segmentation match, as long as either the standard sample word segmentation or any of the word segmentation semantic features in the word segmentation semantics satisfies a matching relationship, the first target sample word segmentation and the second target sample word segmentation can be considered to match.

[0054] Furthermore, a first reference sample word segment with a non-key domain segmentation attribute can be selected from the first sample word segmentation set, and a second reference sample word segment with a non-key domain segmentation attribute can be selected from the second sample word segmentation set. If the first reference sample word segment and the second reference sample word segment match, then the first reference sample word segment is determined as the first sample word segment that matches the second sample word segmentation set. In other words, the first sample word segment that matches the second sample word segmentation set can include first sample word segment with a key domain segmentation attribute, or it can include first sample word segment with a non-key domain segmentation attribute.

[0055] Furthermore, it can be understood that when the first sample text and the second sample word segmentation match, it means that there is a first target sample word segmentation with the word segmentation attribute of key domain in the first sample text and a second target sample word segmentation with the word segmentation attribute of key domain in the second sample text, and there is a first sample word segmentation in the first target sample word segmentation that matches the second target sample word segmentation.

[0056] It can be understood that there can be one or more first target sample words and one or more second target sample words. The key domain word segmentation attributes of different first target sample words can be different, and the key domain word segmentation attributes of different second target sample words can be different. If the word segmentation semantics of the reference target sample word among the multiple first target sample words match the word segmentation semantics of any second target sample word among the multiple second target sample words, and the key domain word segmentation attribute of the reference target sample word is the same as the key domain word segmentation attribute of any second target sample word, then the reference target sample word is determined as the first sample word that matches the second sample word set.

[0057] In some embodiments, generating word weight annotation information for each first sample word segment can be as follows: if any first sample word in the first sample word segmentation set matches a first sample word in the second sample word segmentation set, then the word weight annotation information for that first sample word segment is determined as a first word weight value; if any first sample word does not match a first sample word in the second sample word segmentation set, then the word weight annotation information for that first sample word segment is determined as a second word weight value. The first word weight value can be set to 1, and the second word weight value can be set to 0. Alternatively, if any first sample word in the first sample word segmentation set matches a first sample word in the second sample word segmentation set, then the word weight annotation information for that first sample word is determined as a word weight value matching the segmentation attribute of that first sample word; if any first sample word does not match a first sample word in the second sample word segmentation set, then the word weight annotation information for that first sample word is determined as a default word weight value (e.g., 0). For example, the importance of key domain segmentation attributes is greater than that of non-key domain segmentation attributes, and the importance of segmentation attributes in different key domains can also vary. Therefore, the word weight values ​​corresponding to different segmentation attributes can be determined based on the importance of the segmentation attributes, thereby determining the word weight annotation information of the first sample segmentation that matches the second sample segmentation set as the word weight values ​​corresponding to the respective segmentation attributes.

[0058] For example, if the first sample text is "xx brand organic high calcium infant rice cereal", the first sample word segmentation obtained by word segmentation is [xx brand, organic, high calcium, infant, rice cereal]. If the first sample word segmentation "xx brand", "high calcium", and "rice cereal" are the first sample word segments that match the second sample word segmentation set, then the word weight annotation information of the first sample word segmentation "xx brand", "high calcium", and "rice cereal" can be set as the first word weight value (e.g., 1), and the word weight annotation information of the first sample word segmentation "organic" and "infant" can be set as the second word weight value (e.g., 0). Thus, the word weight annotation information for the first sample text is [1, 0, 1, 0, 1]. Alternatively, for example, if the first sample word segmentation "xx brand", "high calcium", and "rice noodles" are the first sample word segments that match the second sample word segmentation set, and the word segmentation attribute of the first sample word segmentation "xx brand" is a product brand attribute, the word segmentation attribute of the first sample word segmentation "high calcium" is a product modification attribute, and the word segmentation attribute of the first sample word segmentation "rice noodles" is a product category attribute (e.g., the matching word weight value is 2), then the word weight annotation information of the first sample word segmentation "xx brand" can be set to the word weight value matching the product brand attribute (e.g., the matching word weight value is 1), and the word weight annotation information of the first sample word segmentation "organic" can be set to the word weight value matching the product brand attribute. The word weight annotation information for the first sample text is set to the default word weight value (e.g., 0). The word weight annotation information for the first sample word segment "high calcium" is set to the word weight value that matches the product modification attribute (e.g., the matching word weight value is 1.5). The word weight annotation information for the first sample word segment "infant" is set to the default word weight value (e.g., 0). The word weight annotation information for the first sample word segment "rice noodles" is set to the word weight value that matches the product category attribute (e.g., the matching word weight value is 2). Thus, the word weight annotation information for the first sample text is [1, 0, 1.5, 0, 2]. The method for determining the word weight annotation information is not limited here and can be determined according to the specific application scenario.

[0059] For example, such as Figure 3 As shown, Figure 3This is a schematic diagram of a word weighting annotation scenario provided in an embodiment of this application; wherein, the first sample word segmentation of the first sample text is A1, A2, ..., A8, and the second sample word segmentation of the second sample text is B1, B2, ..., B6. For example, the word segmentation attribute of the first sample word segmentation A1-A4 is a key domain word segmentation attribute, and the word segmentation attribute of the first sample word segmentation A1 is key domain word segmentation attribute 1 (e.g., represented as X1), the word segmentation attribute of the first sample word segmentation A2 is key domain word segmentation attribute 2 (e.g., represented as X2), the word segmentation attribute of the first sample word segmentation A3 is key domain word segmentation attribute 1 (e.g., represented as X1), and the word segmentation attribute of the first sample word segmentation A4 is key domain word segmentation attribute 3 (e.g., represented as X1). X3); The segmentation attributes of the first sample words A5-A8 are non-key domain segmentation attributes (e.g., represented as X4); For example, the segmentation attributes of the second sample words B3-B6 are key domain segmentation attributes, and the segmentation attributes of the second sample words B3-B4 are key domain segmentation attribute 1, and the segmentation attributes of the second sample words B5-B6 are key domain segmentation attribute 2; the segmentation attributes of the second sample words B1-B2 are non-key domain segmentation attributes; obtain the segmentation semantics of the first sample words A1-A4 and the segmentation semantics of the second sample words B3-B6, and based on the segmentation semantics and their respective segmentation attributes of the first sample words A1-A4 and the segmentation semantics and their respective segmentation attributes of the second sample words B3-B6, from the first For example, the first sample segmentation words that match the second sample segmentation words B3-B6 are determined from sample segmentation words A1-A4. For instance, based on the segmentation semantics and segmentation attributes of the first sample segmentation word A1, and the segmentation semantics and segmentation attributes of the second sample segmentation word B1, the first sample segmentation word A1 and the second sample segmentation word B1 are compared. If the segmentation semantics of the first sample segmentation word A1 and the second sample segmentation word B1 are the same, and the segmentation attributes of the first sample segmentation word A1 and the second sample segmentation word B1 are the same, then the first sample segmentation word A1 and the second sample segmentation word B1 are determined to match. This yields the first sample segmentation words that match the second sample segmentation set under each key domain segmentation attribute. For example, in... Under key domain segmentation attribute 1, the first sample segment A1 matches the second sample segment B3, and the first sample segment A2 matches the second sample segment B5; under key domain segmentation attribute 2, the first sample segment A3 matches the second sample segment B4; when labeling word weights: if the word weight values ​​associated with the key domain segmentation attribute are all 1, then the word weight labeling information generated for each first sample segment is [1,1,1,0,0,0,0,0]; or, if the word weight value associated with key domain segmentation attribute 1 is 1 and the word weight value associated with key domain segmentation attribute 2 is 2, then the word weight labeling information generated for each first sample segment is [1,2,1,0,0,0,0,0].

[0060] S103. Input each first sample word segment into the initial word segmentation processing model in sequence. The initial word segmentation processing model performs word segmentation processing on each first sample word segment to obtain the word weight prediction information of each first sample word segment.

[0061] In this process, the segmentation encoding vectors of each first sample word segmentation can be sequentially input into the initial word segmentation processing model. The initial word segmentation processing model then processes the segmentation encoding vectors of each first sample word segmentation to obtain the word weight prediction information of each first sample word segmentation, which is the word weight prediction value of each first sample word segmentation. The initial word segmentation processing model can be, for example, a transformer model (a neural network model).

[0062] S104. Iteratively train the initial word segmentation processing model using word weight prediction information and word weight annotation information to obtain the target word segmentation processing model used to determine the word weights of text segmentation.

[0063] Specifically, a multi-class loss function can be used to calculate the loss value of the word weight prediction model and the word weight annotation information to obtain the model loss value for the initial word segmentation processing model. The initial word segmentation processing model is then iteratively calculated in the direction of reducing the model loss value to obtain the target word segmentation processing model. This target word segmentation processing model can be used to determine the word weight of each text segment in the recalled text under the target key domain. Subsequently, these word weights can be used to determine the text matching degree between the recalled text and the query text under the target key domain.

[0064] Therefore, word weights can be labeled on the recalled text by matching the recalled text and the query text, thereby enabling the determination of word weights for recalled texts in a specific domain. This can identify more important word segments from the recalled text that are related to the target key domain. This not only makes the determined word weights more representative, but also improves the application effect of word weights, such as making the determined text matching degree more accurate when matching subsequent texts.

[0065] For example, such as Figure 4 As shown, Figure 4This is a schematic diagram illustrating a training scenario for an initial word segmentation processing model provided in an embodiment of this application. The process involves acquiring a first sample text and a second sample text; performing word segmentation on the first sample text to obtain a first sample text set; performing word segmentation on the second sample text to obtain a second sample word set; determining the word segmentation attributes of each first sample word and each second sample word; and matching the first and second sample word sets based on the word segmentation attributes of the first and second sample words to obtain a matched sample word set. The matched sample word set includes first sample words that match the second sample word set. The process involves generating word weight annotation information for the first sample word segmentation set based on the matching sample word segmentation set. Each first sample word segmentation in the first sample word segmentation set is then processed into a word segmentation vector (tokenizer). The word segmentation encoding vector of each first sample word segmentation is input into the initial word segmentation processing model, which outputs multi-classification results, i.e., word weight prediction information for each first sample word segmentation. Multi-classification loss is calculated using the word weight annotation information and the word weight prediction information to obtain the model loss value for the initial word segmentation processing model. This model loss value is then used to train the initial word segmentation processing model, resulting in the trained target word segmentation processing model.

[0066] Therefore, the trained target word segmentation processing model can accurately determine the word weights of text segments, and can learn the word weight information of the first sample word segment that matches the second sample word segmentation set in the target key domain during the training process. That is, the word weight annotation information determined based on the second sample word segmentation. In this way, the target word segmentation processing model can learn the text segmentation word weight information in a specific domain, thereby better determining the word weights of text segments in the target text in the target key domain.

[0067] In this embodiment, sample text pairs can be obtained; each sample text pair includes a first sample text and a second sample text under the target key domain, wherein the first sample text is the recall text and the second sample text is the query text; first sample words that match the second sample word set are determined from the first sample word set, and word weight annotation information for each first sample word in the first sample word set is generated based on the first sample word that matches the second sample word set; this word weight annotation information can effectively combine the relevant matching information between the first sample text and the second sample text; each first sample word is sequentially input into the initial word segmentation processing model. The initial word segmentation model segments each first sample word to obtain word weight prediction information. The initial word segmentation model is then iteratively trained using the word weight prediction information and word weight annotation information to obtain the target word segmentation model for determining the word weights of the text. This method can determine the word weights of recalled text in a specific domain. Furthermore, when training the initial word segmentation model, it can incorporate not only relevant domain information of the target key domain but also relevant feature information within the first sample text, thereby improving the accuracy of the target word segmentation model in determining the word weights of the text.

[0068] Please see Figure 5 , Figure 5 This is a flowchart illustrating a data processing method provided in an embodiment of this application. This method can be executed by the aforementioned electronic device. Figure 5 As shown, the data processing method in this embodiment may include the following flow:

[0069] S201, Obtain the first target text.

[0070] The first target text is the recall text within the target key domain, such as product titles.

[0071] The first target text is associated with a text segmentation set, which includes at least one text segment.

[0072] S202. Input at least one text segmentation word into the target segmentation processing model in sequence. The target segmentation processing model performs segmentation processing on the at least one text segmentation word to obtain the word weight prediction information of each text segmentation word.

[0073] Specifically, the word segmentation encoding vectors of each text segment are obtained, and the word segmentation encoding vectors of at least one text segment are sequentially input into the target word segmentation processing model. The target word segmentation processing model performs word segmentation processing on the word segmentation encoding vectors of at least one text segment to obtain the word weight prediction information of each text segment, that is, the word weight prediction value of each text segment.

[0074] S203. Obtain the second target text, and determine the text matching degree between the first target text and the second target text based on the word weight prediction information of each text segmentation.

[0075] The second target text can be query text within the target key domain. For example, it could be a user-inputted query term or social media content posted by the user online. There are no restrictions on the type or source of the second target text. The text matching degree between the first and second target texts can be applied to recommendation scenarios targeting the first target text; that is, the recommendation strategy for the first target text can be determined through the text matching degree.

[0076] In some embodiments, determining the text matching degree between the first target text and the second target text may involve identifying a text segment that matches the second target text from at least one text segmentation; and determining the text matching degree between the first target text and the second target text based on the word weight prediction information of the text segment that matches the second target text. Specifically, identifying a text segment that matches the second target text from at least one text segmentation may involve using the text segment that appears in the second target text from at least one text segmentation as the text segment that matches the second target text. Alternatively, at least one text segment of the first target text may be used as the first text segmentation, and the second target text may be segmented to obtain at least one text segmentation of the second target text. If the semantic meaning of the target text segmentation in the first text segmentation matches the semantic meaning of the target text segmentation in the second text segmentation, then the target text segmentation in the first text segmentation is used as the text segment that matches the second target text; the target text segmentation is any text segmentation from the first text segmentation. For the method of determining the semantics of a text segmentation, please refer to the method of determining the semantics of the sample segmentation mentioned above.

[0077] The process of determining the text matching degree between the first and second target texts based on the word weight prediction information of the text segmentation matching the second target text can be as follows: Obtain the text feature vectors of the first and second target texts, and determine the text similarity between them. Then, determine the similarity correction parameter based on the word weight prediction information of the text segmentation matching the second target text, and correct the text similarity using the similarity correction parameter to obtain the text matching degree between the first and second target texts. The text similarity can be understood as the initial text matching degree between the first and second target texts. Correcting the initial text matching degree using word weight prediction information yields a more accurate text matching degree. The correction of text similarity can be achieved by weighting the text similarity using the similarity correction parameter and using the weighted text similarity as the initial text matching degree. Alternatively, the sum of the similarity correction parameter and the text similarity value can be used as the text matching degree. No specific limitation is imposed here.

[0078] In this context, there can be K word segments that match the second target text. The similarity correction parameter is determined using the word weight prediction information of these K matching word segments. Specifically, the target word weight value can be determined based on the word weight prediction information of the K matching word segments, and the similarity correction parameter can be determined based on the target word weight value. Alternatively, the average of the word weight values ​​indicated by the word weight prediction information of the K word segments can be used as the target word weight value. The similarity correction parameter corresponding to the target word weight value can be determined based on the mapping relationship between word weight values ​​and similarity correction parameters. This mapping relationship can be set based on empirical values.

[0079] In some embodiments, there may be one or more first target files. Determining the text matching degree between the first target text and the second target text may involve determining the word segmentation importance of each matching word segment of the first target text and the second target text based on the word weight prediction information of the matching word segments of each first target text and the second target text; and determining the text matching degree between each first target text and the second target text separately using the word segmentation importance of each matching word segment of the first target text and the second target text. Specifically, the word segmentation importance of each matching word segment of the first target text and the second target text can be determined based on the mapping relationship between word weight prediction information and word segmentation importance. Alternatively, the summation of the word segmentation importance of each matching word segment of a first target text and a second target text can be used as the text matching degree between each first target text and the second target text.

[0080] Furthermore, the word weights determined by the technical solution of this application can be applied to information search or information recommendation scenarios. Therefore, the first target texts to be recalled can be selected from multiple first target texts according to the text matching degree between each first target text and the second target text. For example, if the first target text is a product title, the product corresponding to the first target text to be recalled is identified as the product to be recalled and pushed to the target user associated with the second target text. As another example, if the first target text is a medical literature abstract, the medical literature corresponding to the first target text to be recalled can be identified as the medical literature to be recalled and returned to the target user associated with the second target text.

[0081] For example, such as Figure 6 As shown, Figure 6 This is a schematic diagram illustrating an application scenario of word weighting provided in an embodiment of this application. The user terminal sends a second target text (e.g., a query term entered by the user), and the electronic device acquires multiple first target texts (e.g., product titles), such as those obtained from a business database. The device determines the word weight of each first target text using a target word segmentation processing model, determines the text matching degree between each first target text and the second target text based on the word weight of each first target text, determines the first target text to be recalled from the multiple first target texts based on the text matching degree, and returns the recall business data associated with the first target text to be recalled (e.g., product data associated with the first target text to be recalled obtained from the business database) to the user terminal.

[0082] In this embodiment, a first target text can be obtained, and at least one text segmentation word can be sequentially input into a target segmentation processing model. The target segmentation processing model performs segmentation processing on the at least one text segmentation word to obtain the word weight prediction information of each text segmentation word. A second target text can then be obtained, and the text matching degree between the first target text and the second target text can be determined based on the word weight prediction information of each text segmentation word. The first target text is the recall text under the target key domain, and the second target text is the query text under the target key domain. This can achieve accurate determination of the word weight of the recall text under a specific domain. Subsequently, the word weight can be used to determine the matching degree between the recall text and the query text under the target key domain. That is, different word weights can be used to correct the initial matching degree of the recall text and the query text to obtain a more accurate text matching degree. For example, the word weight of the text segmentation word that matches the second target text can be used to weight or penalize the initial matching degree, thereby achieving the effect of noise reduction.

[0083] Please see Figure 7 , Figure 7 This is a schematic diagram of a data processing device provided in this application. It should be noted that... Figure 7The data processing apparatus shown is used to execute this application. Figure 2 and Figure 5 The methods in the illustrated embodiments are shown only in the parts relevant to the embodiments of this application for ease of explanation; specific technical details are not disclosed. Reference to this application is required. Figure 2 and Figure 5 The illustrated embodiment. The data processing device 700 may include: an acquisition module 701 and a processing module 702. Wherein:

[0084] The acquisition module 701 is used to acquire sample text pairs; the sample text pair includes a first sample text and a second sample text that matches the first sample text; the first sample text is the recall text in the target key domain, and the second sample text is the query text in the target key domain; the first sample text is associated with a first sample word segmentation set, and the second sample text is associated with a second sample word segmentation set; the first sample word segmentation set includes at least one first sample word segmentation; the second sample word segmentation set includes at least one second sample word segmentation.

[0085] The processing module 702 is used to determine the first sample word that matches the second sample word set from the first sample word set, and generate word weight annotation information for each first sample word in the first sample word set based on the first sample word that matches the second sample word set.

[0086] The processing module 702 is also used to input each first sample word segment into the initial word segmentation processing model in sequence, and the initial word segmentation processing model performs word segmentation processing on each first sample word segment to obtain the word weight prediction information of each first sample word segment.

[0087] The processing module 702 is also used to iteratively train the initial word segmentation processing model through word weight prediction information and word weight annotation information to obtain the target word segmentation processing model used to determine the word weights of text segmentation.

[0088] Specifically, when the processing module 702 determines the first sample word that matches the second sample word set from the first sample word set, it is used to:

[0089] Obtain the key domain word segmentation attributes associated with the target key domain;

[0090] The first target sample word segmentation is selected from the first sample word segmentation set, and the second target sample word segmentation is selected from the second sample word segmentation set, where the word segmentation attribute is the key domain word segmentation attribute.

[0091] If the semantic representation of the first target sample word segment matches the semantic representation of the second target sample word segment, then the first target sample word segment is determined as the first sample word segment that matches the second target sample word segment in the second sample word segment set.

[0092] Specifically, when the processing module 702 generates word weight annotation information for each first sample word in the first sample word set based on the first sample word segmentation that matches the second sample word segmentation set, it is used for:

[0093] If any first sample word in the first sample word segmentation set is a first sample word segmentation that matches the second sample word segmentation set, then the word weight annotation information of the first sample word segmentation is determined as the first word weight value.

[0094] If any of the first sample word segments is not a first sample word segment that matches the second sample word segment set, then the word weight annotation information of any of the first sample word segments is determined as the second word weight value.

[0095] The processing module 702 is also used for:

[0096] Obtain a first target text; the first target text is a recall text in the target key domain; the first target text is associated with a text segmentation set, and the text segmentation set includes at least one text segmentation;

[0097] The at least one text segment is sequentially input into the target segmentation processing model, which then performs segmentation processing on the at least one text segment to obtain word weight prediction information for each text segment.

[0098] The processing module 702 is also used for:

[0099] Obtain the second target text; the second target text is the query text in the target key domain;

[0100] Determine the text segment that matches the second target text from the at least one text segment;

[0101] The text matching degree between the first target text and the second target text is determined based on the word weight prediction information of the text segmentation that matches the second target text.

[0102] Specifically, when the processing module 702 determines the text matching degree between the first target text and the second target text based on the word weight prediction information of the text segmentation that matches the second target text, it is used for:

[0103] Obtain the text feature vector of the first target text and the text feature vector of the second target text, and determine the text similarity between the first target text and the second target text based on the text feature vector of the first target text and the text feature vector of the second target text;

[0104] The similarity correction parameters are determined based on the word weight prediction information of the text segmentation that matches the second target text, and the text similarity is corrected according to the similarity correction parameters to obtain the text matching degree between the first target text and the second target text.

[0105] The first target text can be multiple; when the processing module 702 determines the text matching degree between the first target text and the second target text based on the word weight prediction information of the text segmentation that matches the second target text, it is specifically used for:

[0106] The word importance of the text segments that match each first target text with the second target text is determined based on the word weight prediction information of the text segments.

[0107] The text matching degree between each first target text and the second target text is determined by the importance of the word segmentation of each first target text and the second target text.

[0108] The method further includes:

[0109] The first target text to be recalled is selected from multiple first target texts based on the text matching degree between each first target text and the second target text.

[0110] The specific implementation methods of the acquisition module and the processing module can be found in the description of the above embodiments, and will not be repeated here. It should be understood that the beneficial effects obtained by using the same method will also not be repeated here.

[0111] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8As shown, the electronic device 800 includes at least one processor 801 and a memory 802. Optionally, the electronic device may also include a network interface. The processor 801, memory 802, and network interface can exchange data. The network interface, controlled by the processor 801, is used to send and receive messages. The memory 802 stores computer programs, including program instructions. The processor 801 executes the program instructions stored in the memory 802. The processor 801 is configured to invoke the program instructions to execute the aforementioned method.

[0112] The memory 802 may include volatile memory, such as random-access memory (RAM); the memory 802 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the memory 802 may also include a combination of the above types of memory.

[0113] Processor 801 may be a central processing unit (CPU). In one embodiment, processor 801 may also be a graphics processing unit (GPU). Processor 801 may also be a combination of a CPU and a GPU. Processor 801 can be used to invoke device control applications stored in memory 802 to perform the above-described tasks. Figure 2 and Figure 5 The description of the data processing method in the corresponding embodiments can also be executed as described above. Figure 7 The description of the data processing apparatus in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0114] In specific implementations, the devices, processors, memory, etc., described in the embodiments of this application can execute the implementation methods described in the above method embodiments, or they can execute the implementation methods described in the embodiments of this application, which will not be repeated here.

[0115] This application also provides a computer-readable storage medium storing a computer program. The computer program includes program instructions, which, when executed by a processor, enable the processor to perform some or all of the steps described in the above method embodiments. Optionally, the computer storage medium can be volatile or non-volatile. The computer-readable storage medium may primarily include a program storage area and a data storage area. The program storage area may store an operating system, at least one application program required for a given function, etc.; the data storage area may store data created based on the use of blockchain nodes, etc.

[0116] This application provides a computer program product, which may include a computer program. When the computer program is executed by a processor, it can implement some or all of the steps in the above method, which will not be elaborated here.

[0117] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0118] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer storage medium, which can be a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0119] The above-disclosed embodiments are merely some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments, and equivalent changes made in accordance with the claims of this application, still fall within the scope of this application.

Claims

1. A data processing method, characterized by, The method includes: Obtain sample text pairs; the sample text pairs include a first sample text and a second sample text that matches the first sample text; the first sample text is a recall text in the target key domain, and the second sample text is a query text in the target key domain; the first sample text is associated with a first sample word segmentation set, and the second sample text is associated with a second sample word segmentation set; the first sample word segmentation set includes at least one first sample word segmentation; the second sample word segmentation set includes at least one second sample word segmentation. First sample words that match the second sample word set are determined from the first sample word set, and word weight annotation information of each first sample word in the first sample word set is generated based on the first sample word that matches the second sample word set. Each of the first sample word segments is sequentially input into the initial word segmentation processing model, and the initial word segmentation processing model performs word segmentation processing on each of the first sample word segments to obtain the word weight prediction information of each of the first sample word segments; The initial word segmentation model is iteratively trained using the word weight prediction information and the word weight annotation information to obtain the target word segmentation model used to determine the word weights of text segmentation.

2. The method of claim 1, wherein, The step of determining the first sample word segment that matches the second sample word segment from the first sample word segment set includes: Obtain the key domain word segmentation attributes associated with the target key domain; The first target sample word segmentation is selected from the first sample word segmentation set, and the second target sample word segmentation is selected from the second sample word segmentation set, where the word segmentation attribute is the key domain word segmentation attribute. If the semantic representation of the first target sample word segment matches the semantic representation of the second target sample word segment, then the first target sample word segment is determined as the first sample word segment that matches the second target sample word segment in the second sample word segment set.

3. The method of claim 1, wherein, The step of generating word weight annotation information for each first sample word in the first sample word set based on the first sample word segmentation that matches the second sample word segmentation set includes: If any first sample word in the first sample word segmentation set is a first sample word segmentation that matches the second sample word segmentation set, then the word weight annotation information of the first sample word segmentation is determined as the first word weight value. If any of the first sample word segments is not a first sample word segment that matches the second sample word segment set, then the word weight annotation information of any of the first sample word segments is determined as the second word weight value.

4. The method of claim 1, wherein, The method further includes: Obtain a first target text; the first target text is a recall text in the target key domain; the first target text is associated with a text segmentation set, and the text segmentation set includes at least one text segmentation; The at least one text segment is sequentially input into the target segmentation processing model, which then performs segmentation processing on the at least one text segment to obtain word weight prediction information for each text segment.

5. The method of claim 4, wherein, The method further includes: Obtain the second target text; the second target text is the query text in the target key domain; Determine the text segment that matches the second target text from the at least one text segment; The text matching degree between the first target text and the second target text is determined based on the word weight prediction information of the text segmentation that matches the second target text.

6. The method of claim 5, wherein, Determining the text matching degree between the first target text and the second target text based on the word weight prediction information of the text segmentation that matches the second target text includes: Obtain the text feature vector of the first target text and the text feature vector of the second target text, and determine the text similarity between the first target text and the second target text based on the text feature vector of the first target text and the text feature vector of the second target text; The similarity correction parameters are determined based on the word weight prediction information of the text segmentation that matches the second target text, and the text similarity is corrected according to the similarity correction parameters to obtain the text matching degree between the first target text and the second target text.

7. The method of claim 5, wherein, The first target text can be multiple; the step of determining the text matching degree between the first target text and the second target text based on the word weight prediction information of the text segmentation that matches the second target text includes: The word importance of the text segments that match each first target text with the second target text is determined based on the word weight prediction information of the text segments. The text matching degree between each first target text and the second target text is determined by the importance of the word segmentation of each first target text and the second target text. The method further includes: The first target text to be recalled is selected from multiple first target texts based on the text matching degree between each first target text and the second target text.

8. A data processing apparatus, characterized by, The device includes: An acquisition module is used to acquire sample text pairs; the sample text pair includes a first sample text and a second sample text that matches the first sample text; the first sample text is a recall text in the target key domain, and the second sample text is a query text in the target key domain; the first sample text is associated with a first sample word segmentation set, and the second sample text is associated with a second sample word segmentation set; the first sample word segmentation set includes at least one first sample word segmentation; the second sample word segmentation set includes at least one second sample word segmentation. The processing module is used to determine the first sample word that matches the second sample word set from the first sample word set, and generate word weight annotation information for each first sample word in the first sample word set based on the first sample word that matches the second sample word set; The processing module is further configured to input each of the first sample word segments into the initial word segmentation processing model in sequence, and the initial word segmentation processing model performs word segmentation processing on each of the first sample word segments to obtain word weight prediction information for each of the first sample word segments; The processing module is further configured to iteratively train the initial word segmentation processing model using the word weight prediction information and the word weight annotation information to obtain a target word segmentation processing model for determining the word weights of text segmentation.

9. An electronic device, comprising: The system includes a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-7.

Images