Data processing and model training method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a data processing and model training method and device, electronic equipment and storage medium, relates to the technical field of artificial intelligence, and particularly relates to the technical field of deep learning. The specific implementation scheme is: performing binary classification processing on each text data in a first text data set of a target user for a target category to obtain a confidence of each text data; obtaining confidence statistical data corresponding to the target category based on the confidence of each text data; and obtaining a category feature of the target user based on the confidence statistical data corresponding to the target category. According to the embodiment of the present disclosure, the category feature of the user can be determined through the confidence of the text data of the user belonging to the target type, and the intention of the user can be accurately mined.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, and more particularly to a data processing and model training method and device, an electronic device and a storage medium. BACKGROUND

[0002] With the continuous development of machine learning, more and more enterprises apply machine learning models to actual businesses. For example, in banks, insurance companies, securities companies and other financial companies, machine learning models are applied to insurance risk control, anti-fraud, credit scoring and other scenarios; on e-commerce platforms, machine learning models are applied to product listing, recommendation, scoring and other scenarios; in the transportation industry, machine learning models are applied to package delivery, path selection and other scenarios. A large amount of user data is generated in these scenarios, and mining these user data is beneficial to improving the use effect of models in actual businesses. SUMMARY

[0003] The present disclosure provides a data processing and model training method and device, an electronic device and a storage medium.

[0004] According to an aspect of the present disclosure, a data processing method is provided, comprising:

[0005] performing binary classification processing on each text data in a first text data set of a target user for a target category to obtain a confidence of the each text data;

[0006] obtaining confidence statistical data corresponding to the target category based on the confidence of the each text data;

[0007] obtaining a category feature of the target user based on the confidence statistical data corresponding to the target category.

[0008] According to an aspect of the present disclosure, a model training method is provided, comprising:

[0009] performing category labeling on each text data in a second text data set to obtain category information of the each text data;

[0010] training a second classification model corresponding to each category in at least one category based on the each text data and the category information of the each text data, respectively, to obtain a first classification model corresponding to the each category; wherein the first classification model is used for performing binary classification processing on text data for a category corresponding to the classification model to obtain a confidence of the text data.

[0011] According to another aspect of the present disclosure, a data processing device is provided, comprising:

[0012] a binary classification module configured to perform binary classification processing on each text data in a first set of text data of a target user with respect to a target category to obtain a confidence level of the text data;

[0013] a confidence level statistics module configured to obtain confidence level statistics data corresponding to the target category based on the confidence level of the text data;

[0014] a user feature determination module configured to obtain a category feature of the target user based on the confidence level statistics data corresponding to the target category.

[0015] According to another aspect of the present disclosure, a model training apparatus is provided, comprising:

[0016] a category labeling module configured to label each text data in a second set of text data with a category to obtain category information of the text data;

[0017] a model training module configured to train a second classification model corresponding to each category in at least one category based on the text data and the category information of the text data to obtain a first classification model corresponding to the category; wherein the first classification model is configured to perform binary classification processing on text data with respect to a category corresponding to the classification model to obtain a confidence level of the text data.

[0018] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein

[0021] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method in any embodiment of the present disclosure.

[0022] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, the computer instructions being used to cause a computer to perform the method in any embodiment of the present disclosure.

[0023] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method in any embodiment of the present disclosure.

[0024] According to the technology of the present disclosure, the binary classification processing of each text data in the first text data set of the target user is performed for the target category, the confidence of each text data is obtained, and the confidence statistical data corresponding to the target category is obtained based on the confidence of each text data. Then, based on the confidence statistical data corresponding to the target category, the category feature of the target user is obtained. The category feature obtained in this way can reflect the characteristics of the target user in handling business in this target category, that is, the technology of the present disclosure can accurately mine the intention of the user in a specified category.

[0025] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings are used to better understand the present scheme and do not constitute a limitation on the present disclosure. Among them:

[0027] Figure 1 is a flowchart of the data processing method of the first embodiment of the present disclosure;

[0028] Figure 2 is a flowchart of the model training method of the second embodiment of the present disclosure;

[0029] Figure 3 is a structural block diagram of the data processing device of the third embodiment of the present disclosure;

[0030] Figure 4 is a structural block diagram of the data processing device of the fourth embodiment of the present disclosure;

[0031] Figure 5 is a structural block diagram of the model training device of the fifth embodiment of the present disclosure;

[0032] Figure 6 is a structural block diagram of the model training device of the sixth embodiment of the present disclosure;

[0033] Figure 7 is a block diagram of an electronic device implementing the data processing method and the model training method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0035] Figure 1 is a flowchart of a data processing method of a first embodiment of the present disclosure. As shown in the figure, the data processing method can include the following steps: Figure 1

[0036] S110, performing binary classification processing on each text data in the first text data set of the target user for the target category to obtain the confidence of each text data;

[0037] S120, obtaining confidence statistical data corresponding to the target category based on the confidence of each text data;

[0038] S130, obtaining the category feature of the target user based on the confidence statistical data corresponding to the target category.

[0039] Exemplarily, the target user can be one user or multiple users, for example, a user of a certain age group, a user of a certain region, a user in a certain scene in a certain past time period.

[0040] Exemplarily, the first text data set is a data set for obtaining the category feature of the target user. It can include multiple text data, which are text data related to the target user generated in actual application scenarios. For example, in an online shopping platform, the text data generated by the target user when searching for goods. For another example, in a search engine, the text data generated by the target user when searching online in a certain time period. The first text data set can also specify the text data generated by the target user in one or more scenes in a certain time period. For example, in the past three months, in the past six months.

[0041] Exemplarily, the target category can include disease, drug, loan, insurance, consumption, food consumption, etc. Using the method of the present embodiment, the category feature of each category of the target user can be obtained.

[0042] Exemplarily, the confidence can be the possibility that the category of the text data is the target category. The higher the confidence, the higher the possibility that the category of the text data is the target category; the lower the confidence, the lower the possibility that the category of the text data is the target category. In the case where the confidence is higher than the set threshold, it can be determined that the category of the text data is the target category. In the case where the confidence is lower than the set threshold, it can be determined that the category of the text data is not the target category.

[0043] Exemplarily, the statistical data of the confidence can include: the maximum value, the minimum value, the mean value, the median, the variance, the standard deviation, etc. in multiple confidences. It can also include the number of text data with confidence higher than the set threshold, the number of text data with confidence lower than the set threshold, etc.

[0044] ​Exemplarily, in the step S120, the statistical data of the confidence degree can include: the maximum value, the minimum value, the mean value and the variance of the confidence degree of the text data in the first text data set, and the number of text data whose confidence degree is higher than the set threshold. By counting these confidence degree data, the category characteristics reflected by the target user on the target category can be obtained, which can truly reflect the user's intention.

[0045] For example, in the finance category, the number of text data whose confidence degree is higher than 0.8 is 10, and the first text data set includes 12 text data, so the proportion of the text data whose confidence degree is higher than 0.8 is high, which reflects that the user has strong tendency on the finance category and is more interested in finance.

[0046] Exemplarily, the category characteristics of the user include the characteristics of each category, which can be composed of the confidence degree statistical data, for example, each category corresponds to a confidence degree feature vector, which represents the category characteristics of the user on this category.

[0047] In the embodiments of the present disclosure, the confidence degree of each text data is determined for the target category by fully utilizing the text data set provided by the target user, and then the confidence degree statistical data is used to mine the potential information about the target user in the text data, so that the obtained category characteristics of the target user can fully reflect the real intention of the target user. Training the machine learning model in the actual application scenario based on the category characteristics of the target user can improve the accuracy of the machine learning model.

[0048] In some embodiments, the text data can be processed by the first classification model corresponding to the target type to obtain the confidence degree of the text data. The first classification model is a trained model, for example, the accuracy of the model meets the set requirements after training.

[0049] In some embodiments, each word in each text data in the text data set of the target user can also be analyzed or calculated to obtain the confidence degree of each text data belonging to the target category.

[0050] Exemplarily, in the step S110, each text data in the text data set of the target user is processed by the binary classification for the target category to obtain the confidence degree of each text data, including:

[0051] Tokenizing each text data in the first text data set of the target user;

[0052] Determining the weight of each word in each text data based on the positive sample rate of each word in each text data;

[0053] The semantic and weight of each word in each text data are used to determine the confidence of each text data belonging to the target category.

[0054] In this example, the positive sample rate of a word can be determined based on the number of times the word is searched and the total amount of data in the first set of text data.

[0055] For example, the positive sample rate of each word in each text data can be calculated as follows:

[0056] Suppose the first set of text data includes 10 text data, for the word a in text data A, if it is used or searched by a user, it is considered 1, if it is not used or searched by a user, it is considered 0. The text data in which the word a is considered 1 in 10 text data includes text data A and text data B, that is, the number of times considered 1 is 2, and the positive sample rate of the word a is 2 / 10 = 20%.

[0057] For example, the higher the positive sample rate of a word, the higher the weight of the word in the text data. In addition to the positive sample rate of each word, the total number of words obtained by tokenizing the text data can also be used to determine the weight of each word in the text data.

[0058] In the tokenization process, prepositions, modal words and other words can be filtered. For repeated words, de-duplication can be performed, and the weight of the word can be further determined based on the number of repetitions of the word in the text data.

[0059] After obtaining the semantic and weight of each word in the text data, the semantic of each word can be mapped to obtain a numerical value or vector representing the semantic of each word. Then, the weight of each word is multiplied by the numerical value or vector representing the semantic, and the sum is obtained to obtain the confidence of the text data.

[0060] In this example, the weight of each word in each text data is determined based on the positive sample rate of each word in the text data, and then the semantic and weight of each word in each text data are used to obtain the accurate confidence of each text data. Since the weight of the word is related to the positive sample rate of each word searched or used by the user, and the confidence of the text data is determined by combining such weight, the intention of the user in the target category can be accurately reflected.

[0061] In some embodiments, the text data can be converted into a text sentence vector first, and then input into the first classification model corresponding to the target type to obtain the confidence of the text data belonging to the target type.

[0062] Exemplarily, the determining the confidence of each text data belonging to the target category based on the semantics and the weight of each word in each text data can include:

[0063] converting each text data into a text sentence vector based on the semantics and the weight of each word in each text data;

[0064] performing binary classification processing on each text data for the target category by inputting the text sentence vector corresponding to each text data into the first classification model corresponding to the target category, to obtain the confidence of each text data belonging to the target category output by the first classification model.

[0065] The first classification model is a trained model corresponding to the target type, and the accuracy or degree of accuracy of the model meets a set condition.

[0066] In the present example, the trained classification model is used to identify the category of the text data and output the confidence of the text data belonging to the target category. This calculation method is beneficial in improving the accuracy of determining the confidence of the text data and quickly outputting the confidence of a large amount of text data, thereby improving efficiency.

[0067] The first classification model can be pre-trained, or updated and trained before use, or updated and trained after use. The first classification model can also be updated at intervals or when a set condition is met. The set condition can include a set time point or a model accuracy lower than a set threshold.

[0068] In the present embodiment, there are multiple types, and the first classification model of each category can be pre-trained. When the first classification model corresponding to the target type is needed, it can be called according to the target type.

[0069] Figure 2 is a flowchart of the model training method of the second embodiment of the present disclosure. As shown in Figure 2 The model training method can include:

[0070] S210, classifying each text data in the second text data set to obtain the category information of each text data;

[0071] S220, training the second classification model corresponding to each category in at least one category based on each text data and the category information of each text data, to obtain the first classification model corresponding to each category; wherein the first classification model is used to perform binary classification processing on the text data for the category corresponding to the classification model, to obtain the confidence of the text data.

[0072] The second classification model is a model before training, and the first classification model is a trained model.

[0073] Exemplarily, the second set of text data is a data set for training the second classification model. The second set of text data can include a plurality of text data, which are text data related to any user in an actual application scenario. For example, in an online shopping platform, text data generated by a user when searching for goods. For another example, in a search engine, text data generated by a user when searching online in a certain time period. The first set of text data can also specify text data generated by a user in one or more scenarios in a certain time period. For example, in the past three months, in the past six months, in the past year, and the like. It should be noted that the second set of text data is not text data of a specific user, and can include text data of any user.

[0074] The category information of the text data can include a category to which the text data belongs, for example, a category of finance, medicine, visual recognition, animals, food, and the like.

[0075] In the training process of the above step S220, each text data and the category information of each text data are input into the second classification model, and the second classification model outputs the confidence of each text data; in a case where the accuracy of the confidence of each text data does not meet a set condition, the parameters of the second classification model are adjusted, and the step of inputting each text data and the category information of each text data into the second classification model and outputting the confidence of each text data by the second classification model is returned; in a case where the accuracy of the confidence of each text data does not meet the set condition, the adjustment of the parameters of the second classification model is stopped, and the second classification model at the time of stopping the adjustment is determined as the first classification model. Thus, the training of the model is completed.

[0076] In the present example, a set of data is used to train a classification model corresponding to each of a plurality of categories, which can improve the classification accuracy of model training and also improve the speed of model training.

[0077] In some embodiments, part of the text data can be labeled first, and then the distance or similarity degree between the part of the text data and the unlabeled text data is compared to achieve the classification effect. For a large amount of text data, not only the accuracy of labeling can be improved, but also the labeling efficiency can be improved.

[0078] Exemplarily, in the above step S210, the category annotation of each text data in the second set of text data is performed to obtain the category information of each text data, which can include:

[0079] The first part of text data in the second text data set is labeled with a category to obtain category information of the first part of text data;

[0080] Based on the category information of the first part of text data and the distance between the first part of text data and the second part of text data in the second text data set, the second part of text data is labeled to obtain category information of the second part of text data.

[0081] For example, the first part of text data can be randomly extracted from the second text data set for category labeling, such as text data A and text data B. The category information of text data A is category D1, and the category of text data B is category D2.

[0082] The second part of text data is data in the second text data set that has not been labeled.

[0083] For category D1, based on the distance between text data A and each unlabeled text data in the second text data set, the possibility of each unlabeled text data belonging to category D1 can be determined. The closer the distance, the greater the possibility of the unlabeled text data belonging to category D1. For example, if the distance is greater than a set threshold, it can be determined that the unlabeled text data corresponding to the distance belongs to category D1.

[0084] For category D2, based on the distance between text data B and each unlabeled text data in the second text data set, the possibility of each unlabeled text data belonging to category D2 can be determined. The closer the distance, the greater the possibility of the unlabeled text data belonging to category D2. For example, if the distance is greater than a set threshold, it can be determined that the unlabeled text data corresponding to the distance belongs to category D2.

[0085] For a certain category, for example, the first category, if there are multiple text data labeled with the first category, the center point of these text data can be determined, and the distance between the center point and each unlabeled text data in the set can be calculated. The distance represents the possibility of the category of the unlabeled text data corresponding to the distance being the first category.

[0086] In some embodiments, the above-mentioned category labeling of part of the text data can also be performed by manual labeling, which can further improve the accuracy of the labeling.

[0087] In some embodiments, before training the second classification model corresponding to each category in the at least one category, each text data in the second text data set can be converted into a text sentence vector, which is beneficial to improve the training accuracy of the model.

[0088] The conversion of the text data into the text sentence vector can be performed before or after the text data is labeled.

[0089] In the step S220, the second classification model corresponding to each of the at least one category is trained based on each of the text data and the category information of each of the text data, including:

[0090] Converting each of the text data in the second text data set into a text sentence vector;

[0091] Training the second classification model corresponding to each of the at least one category based on the text sentence vector and the category information of each of the text data.

[0092] In some embodiments, the process of converting the text data into the text sentence vector can include:

[0093] Converting each of the text data into the text sentence vector based on the semantics and the weight of each of the words in each of the text data.

[0094] In this example, the weight value can be pre-set or determined based on the positive sample rate of each word in the second text data set.

[0095] In some embodiments, the process of converting the text data into the text sentence vector can include:

[0096] Segmenting each of the text data in the second text data set;

[0097] Determining the weight of each of the words in each of the text data based on the positive sample rate of each of the words of each of the text data;

[0098] Converting each of the text data into the text sentence vector based on the semantics and the weight of each of the words in each of the text data.

[0099] The first classification model trained by the model training method of the embodiments of the present disclosure can be applied to the data processing method of the embodiments of the present disclosure to determine the confidence of the text data, which is beneficial to improve the accuracy of the confidence.

[0100] Figure 3 is a structural block diagram of a data processing apparatus of a third embodiment of the present disclosure. As shown in Figure 3 The processing apparatus can include:

[0101] The binary classification module 310 is configured to perform binary classification processing on each of the text data in the first text data set of the target user for the target category, to obtain the confidence of each of the text data.

[0102] The confidence statistics module 320 is configured to obtain confidence statistics data corresponding to the target category based on the confidence of each text data.

[0103] The user feature determination module 330 is configured to obtain the category feature of the target user based on the confidence statistics data corresponding to the target category.

[0104] Figure 4 is a structural block diagram of a data processing apparatus of the fourth embodiment of the present disclosure. As shown in Figure 4 The data processing apparatus includes a binary classification module 410, a confidence statistics module 420, a user feature determination module 430, and the binary classification module 310, the confidence statistics module 320, and the user feature determination module 330 in the above embodiments have the same functions, and will not be described in detail here.

[0105] In some embodiments, as shown in Figure 4 The binary classification module 410 includes:

[0106] The word segmentation unit 411 is configured to perform word segmentation on each text data in the first text data set of the target user.

[0107] The weight determination unit 412 is configured to determine the weight of each word in each text data based on the positive sample rate of each word in the text data.

[0108] The confidence determination unit 413 is configured to perform binary classification processing on each text data for the target category based on the semantics and weight of each word in the text data, and determine the confidence of the text data belonging to the target category.

[0109] In some embodiments, the confidence determination unit 413 is configured to:

[0110] convert each text data into a text sentence vector based on the semantics and weight of each word in the text data;

[0111] input the text sentence vector corresponding to each text data into a first classification model corresponding to the target category to perform binary classification processing on each text data for the target category, and obtain the confidence of each text data belonging to the target category output by the first classification model.

[0112] In some embodiments, the confidence statistics data includes at least one of the maximum value, the minimum value, the mean value, and the variance value of the confidence of the text data in the first text data set, and / or the number of text data with a confidence greater than a set threshold.

[0113] Figure 5is a structural block diagram of a model training apparatus of a fifth embodiment of the disclosure. As shown in Figure 5 The model training apparatus can include:

[0114] The category labeling module 510 is configured to label each text data in the second text data set with a category to obtain category information of the text data.

[0115] The model training module 520 is configured to train each second classification model corresponding to each category in the at least one category based on the text data and the category information of the text data to obtain a first classification model corresponding to the category. The first classification model is configured to perform binary classification processing on the text data for the category corresponding to the classification model to obtain a confidence score of the text data.

[0116] Figure 6 is a structural block diagram of a model training apparatus of a sixth embodiment of the disclosure. As shown in Figure 6 The model training apparatus includes a category labeling module 610 and a model training module 620. The category labeling module 610 and the model training module 620 have the same functions as the category labeling module 510 and the model training module 520 in the above embodiments, and will not be described in detail here.

[0117] In some embodiments, as shown in Figure 6 The category labeling module 610 includes:

[0118] The first labeling unit 611 is configured to label a first part of text data in the second text data set with a category to obtain category information of the first part of text data.

[0119] The second labeling unit 612 is configured to label a second part of text data in the second text data set based on the category information of the first part of text data and a distance between the first part of text data and the second part of text data to obtain category information of the second part of text data.

[0120] In some embodiments, as shown in Figure 6 The model training module 620 includes:

[0121] The vector conversion unit 621 is configured to convert each text data in the second text data set into a text sentence vector.

[0122] The training unit 622 is configured to train each second classification model corresponding to each category in the at least one category based on the text sentence vector and the category information of the text data.

[0123] The functions of each unit, module or sub-module in each device of the embodiments of the present disclosure can be referred to the corresponding description in the method embodiments described above, which will not be repeated here.

[0124] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0125] According to embodiments of the present disclosure, the present disclosure also provides a network device, which can include the electronic device in the embodiments of the present disclosure.

[0126] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0127] As shown in Figure 7 The electronic device 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0128] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, a speaker, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0129] The computing unit 701 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs various methods and processes described above, such as the data processing method or the model training method. For example, in some embodiments, the data processing method and the model training method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded onto the RAM 703 and executed by the computing unit 701, one or more steps of the data processing method and the model training method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the data processing method and the model training method by any other appropriate means, such as by means of firmware.

[0130] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0131] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0132] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0134] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0135] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0136] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0137] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method, comprising: Perform binary classification processing on each text data in the first text data set of the target user for the target category, and obtain the confidence score of each text data belonging to the target category; Based on the confidence level of each text data belonging to the target category, confidence statistics corresponding to the target category are obtained, wherein the confidence statistics include: the maximum, minimum, mean, and variance of the confidence level of text data belonging to the target category in the first text data set, and the number of text data with a confidence level greater than a set threshold; The confidence statistics corresponding to the target category constitute the category features of the target user in the target category, wherein the category features are used to reflect the user intent of the target user; Based on the category features of the target user in the target category, a machine learning model is trained for a real-world application scenario, where the application scenario includes online shopping platforms and search engines.

2. The method according to claim 1, wherein, The step of performing binary classification on each text data in the target user's text data set to obtain the confidence score of each text data belonging to the target category includes: Perform word segmentation on each text data in the first text data set of the target user; The weight of each word in each text data is determined based on the positive sample rate of each word in each text data. Based on the semantics and weight of each word in each text data, a binary classification process is performed on each text data for the target category to determine the confidence level of each text data belonging to the target category.

3. The method according to claim 2, wherein, The step of performing binary classification processing on each text data based on the semantics and weight of each word in each text data to determine the confidence level of each text data belonging to the target category includes: Based on the semantics and weight of each word in each text data, each text data is converted into a text sentence vector; The text sentence vector corresponding to each text data is input into the first classification model corresponding to the target category to perform binary classification processing on each text data for the target category, and the confidence score of each text data belonging to the target category is obtained by the first classification model.

4. A model training method, comprising: Each text data in the second text data set is categorized to obtain the category information of each text data; Based on each text data and the category information of each text data, a second classification model corresponding to each category in at least one category is trained to obtain a first classification model corresponding to each category; wherein, the first classification model is used to perform binary classification processing on the text data in the first text dataset in the data processing method of claims 1-3, for the category corresponding to the first classification model, to obtain the confidence that the text data belongs to the category.

5. The method according to claim 4, wherein, The step of classifying each text data in the second text data set to obtain the class information of each text data includes: The first part of the text data in the second text data set is labeled with categories to obtain the category information of the first part of the text data. Based on the category information of the first part of the text data and the distance between the first part of the text data and the second part of the text data in the second text data set, the second part of the text data is labeled to obtain the category information of the second part of the text data.

6. The method according to claim 4, wherein, The step of training a second classification model for each category in at least one category based on each text data and the category information of each text data includes: Convert each text data in the second text data set into a text sentence vector; Based on the text sentence vectors and the category information of each text data, a second classification model corresponding to each category in at least one category is trained.

7. A data processing apparatus, comprising: The binary classification module is used to perform binary classification processing on each text data in the first text data set of the target user for the target category, and obtain the confidence score of each text data belonging to the target category; The confidence statistics module is used to obtain confidence statistics data corresponding to the target category based on the confidence level of each text data belonging to the target category. The confidence statistics data includes: the maximum value, minimum value, mean and variance of the confidence level of text data belonging to the target category in the first text data set, and the number of text data with a confidence level greater than a set threshold. The user feature determination module is used to compose the category features of the target user in the target category by the confidence statistics data corresponding to the target category, wherein the category features are used to reflect the user intent of the target user; Based on the category features of the target user in the target category, a machine learning model is trained for a real-world application scenario, where the application scenario includes online shopping platforms and search engines.

8. The apparatus according to claim 7, wherein, The binary classification module includes: The word segmentation unit is used to segment each text data in the first text data set of the target user into words; The weight determination unit is used to determine the weight of each word in each text data based on the positive sample rate of each word in each text data. The confidence determination unit is used to perform binary classification processing on each text data for the target category based on the semantics and weight of each word in each text data, and to determine the confidence level of each text data belonging to the target category.

9. The apparatus according to claim 8, wherein, The confidence level determination unit is used for: Based on the semantics and weight of each word in each text data, each text data is converted into a text sentence vector; The text sentence vector corresponding to each text data is input into the first classification model corresponding to the target category to perform binary classification processing on each text data for the target category, and the confidence score of each text data belonging to the target category is obtained by the first classification model.

10. A model training device, comprising: The category labeling module is used to label each text data in the second text data set with a category, thereby obtaining the category information of each text data. The model training module is used to train a second classification model corresponding to each category in at least one category based on each text data and the category information of each text data, so as to obtain a first classification model corresponding to each category; wherein, the first classification model is used to perform binary classification processing on the text data in the first text data set in the data processing method of claims 1-3, for the category corresponding to the first classification model, to obtain the confidence that the text data belongs to the category.

11. The apparatus according to claim 10, wherein, The category labeling module includes: The first annotation unit is used to perform category annotation on the first part of the text data in the second text data set to obtain the category information of the first part of the text data. The second annotation unit is used to annotate the second part of the text data based on the category information of the first part of the text data and the distance between the first part of the text data and the second part of the text data in the second text data set, so as to obtain the category information of the second part of the text data.

12. The apparatus according to claim 10, wherein, The model training module includes: A vector transformation unit is used to convert each text data in the second text data set into a text sentence vector; The training unit is used to train a second classification model for each category in at least one category, based on the text sentence vector and the category information of each text data.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1-6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images