A data fusion method, device and storage medium

Abstract

The present invention discloses a data fusion method, device and storage medium provided by the present invention, by obtaining a first label group, each label of the first label group includes at least: attribute results and channels; determining the classification of each label according to the attribute results and channels of each label; determining the attribute weight of each attribute result according to the attribute results of each label; obtaining channel weights of different channels; determining the first result probability of each type of label according to the attribute weights and channel weights; determining the attribute results and channels of a type of label with the largest first result probability as the attribute results and channels of the fused label. The present invention can determine the attribute results and channels of the fused label according to the attribute results and channels of multiple labels in the first label group and use the fused label as the fusion result of the first label group, and is applicable to any label, has a wide range of applications, and has certain universal applicability.

Description

Technical Field

[0001] The present invention relates to the field of data fusion, and in particular to a data fusion method, device and storage medium. Background Art

[0002] Insurance companies have a large amount of voice communication data and WeChat text communication data between insurance consultants and customers. A customer may have several voice communications and several WeChat communications with a consultant, register on the company's website, and finally place an order. Each conversation will contain certain basic customer information, such as gender, age, occupation, whether there is social security, etc. Extracting these basic information tags can be used for data fusion, and then building a user portrait system in the insurance field to improve the accuracy of customer information.

[0003] Currently, there are several multimodal technology solutions for data fusion: (1) Rule-based approach. The rule-based approach refers to the use of templates to fuse data from different data sources into a usable data. This approach is mostly used when the data formats are not very different, and the rule template can cover the data format of the existing data source.

[0004] (2) Based on deep learning. The deep learning-based approach generally uses deep learning to extract entities from the data source and classify the same or similar entities. The classified entities can be regarded as fused data.

[0005] (3) Similarity-based approach: The similarity-based approach is to vectorize different data sources, use different similarity calculation methods to calculate similarity, and classify and merge data sources.

[0006] The above solutions all have the disadvantages of narrow application range and lack of universality. Summary of the invention

[0007] In view of the above problems, the present invention provides a data fusion method, device and storage medium that overcome the above problems or at least partially solve the above problems.

[0008] In a first aspect, a data fusion method includes:

[0009] Obtaining a first tag group, the first tag group includes a plurality of tags, each of the tags includes at least: an attribute result and a channel;

[0010] Determine the classification of each tag according to the attribute result and channel of each tag, wherein the multiple tags of the first tag group cover different attribute results of the same attribute, tags with the same attribute results and the same channel are tags of the same category, and tags with different attribute results and / or different channels are tags of different categories;

[0011] Determine the attribute weight of each attribute result according to the attribute result of each tag;

[0012] Obtaining channel weights of different channels, wherein the channel weights are preset weights matching the channels;

[0013] Determine the first result probability of each type of the label according to the attribute weight and the channel weight;

[0014] The attribute results and channels of a type of the labels with the largest probability of the first result are determined as the attribute results and channels of a fused label, where the fused label is a data fusion result of multiple labels of the first label group.

[0015] In combination with the first aspect, in some optional implementations, each of the tags at least further includes: time, and the method further includes:

[0016] Determine the latest tag in each category of tags according to the time of each tag;

[0017] Determine the time weight of each latest tag match according to the time of each latest tag;

[0018] Determining the first result probability of each type of the label according to the attribute weight and the channel weight includes:

[0019] Determine the second result probability of each of the latest tags respectively according to the time weight, the attribute weight and the channel weight;

[0020] The step of determining the attribute result and channel of the label of a category with the largest probability of the first result as the attribute result and channel of the fused label includes:

[0021] The attribute result and channel of the latest label with the highest probability of the second result are determined as the attribute result and channel of the fused label.

[0022] In combination with the previous embodiment, in some optional embodiments, each of the time weights is a preset time weight, and different time ranges of the tags have different matching time weights, and the newer the time, the greater the time weight;

[0023] The determining the second result probability of each of the latest tags respectively according to the time weight, the attribute weight and the channel weight includes:

[0024] Calculating the product of the time weight, the attribute weight and the channel weight of each of the latest tags;

[0025] The product of each of the latest labels is determined as the second result probability.

[0026] In combination with the first aspect, in some optional implementations, determining the first result probability of each type of the tag according to the attribute weight and the channel weight includes:

[0027] Calculate the product of the attribute weight of each type of the label and the corresponding channel weight;

[0028] The product of each class of the labels is determined as the first result probability.

[0029] In combination with the first aspect, in some optional implementations, determining the attribute weight of each attribute result according to the attribute result of each tag includes:

[0030] Calculate the frequency of occurrence of each attribute result in the first label group;

[0031] The frequency of occurrence of each attribute result is normalized to determine the attribute weight of each attribute result.

[0032] In conjunction with the first aspect, in certain optional embodiments, the first tag group includes at least one new tag;

[0033] The new label is obtained through the following process:

[0034] Get text data;

[0035] Determine a channel identifier and a time identifier of the text data, wherein the channel identifier is not limited to one of the following channel identifiers: WeChat, phone, QQ, and the time identifier is the time when the corresponding conversation of the text data is initiated;

[0036] Execute the method 1, the method 2 and the method 3 on the text data, so as to obtain at least one new label;

[0037] Method 1: Perform regular expression matching on the text data, and if the text data includes formatting attributes, obtain the formatting attributes; obtain at least one new tag according to the formatting attributes, the channel identifier of the text data, and the time identifier of the text data; wherein the formatting attributes include but are not limited to at least one of the following attributes: email address, mobile phone number, and QQ number;

[0038] Method 2: Input the text data into a pre-trained binary classification model, and if the text data includes a classification attribute, obtain the classification attribute; obtain at least one new label according to the classification attribute, the channel identifier of the text data, and the time identifier of the text data; wherein the classification attribute includes but is not limited to at least one of the following attributes: whether there is social insurance, whether there is medical insurance, whether there is commercial insurance, and whether there is a loan;

[0039] Method three: the text data is input into a pre-trained entity recognition model, and if the text data includes entity attributes, the entity attributes are obtained; at least one new label is obtained according to the entity attributes, the channel identifier of the text data and the time identifier of the text data; wherein the entity attributes include but are not limited to at least one of the following attributes: age, disease and occupation.

[0040] In combination with the first aspect, in some optional implementations, the binary classification model is a fasttext model, and the training process of the fasttext model includes:

[0041] Obtaining a labeled data set, wherein the labeled data set includes at least one of the following data sets: a data set with social insurance, a data set without social insurance, a data set with medical insurance, a data set without medical insurance, a data set with commercial insurance, a data set without commercial insurance, a data set with loans, and a data set without loans;

[0042] The annotated data set is input into the fasttext classification algorithm, so as to perform binary classification training on the fasttext classification algorithm.

[0043] Optionally, in some optional implementations, the entity recognition model includes: BERT algorithm, LSTM algorithm and CRF algorithm, and the training process of the entity recognition model includes:

[0044] Inputting a pre-constructed structured training set into the BERT algorithm, wherein the format of each training corpus in the structured training set conforms to the format requirement of the BERT algorithm;

[0045] The BERT algorithm outputs semantic vectors corresponding to each of the training corpora to the LSTM algorithm, wherein the semantic vectors incorporate the full-text semantic information of the corresponding training corpora;

[0046] The LSTM algorithm performs sequence labeling on each of the semantic vectors input by the BERT algorithm, thereby obtaining a label sequence and outputting it to the CRF algorithm;

[0047] The CRF algorithm outputs the probability of the label sequence as a test result of testing the structured training set. When the test result meets the preset result requirement, the BERT algorithm, the LSTM algorithm and the CRF algorithm are jointly constructed as the entity recognition model.

[0048] In a second aspect, a data fusion device includes: a label group obtaining unit, a classification determining unit, an attribute weight determining unit, a channel weight obtaining unit, a first result unit, and a fusion determining unit;

[0049] The tag group obtaining unit is configured to obtain a first tag group, wherein the first tag group includes a plurality of tags, each of which includes at least: an attribute result and a channel;

[0050] The classification determination unit is configured to determine the classification of each of the tags according to the attribute results and channels of each of the tags, wherein the multiple tags of the first tag group cover different attribute results of the same attribute, tags with the same attribute results and the same channels are tags of the same category, and tags with different attribute results and / or different channels are tags of different categories;

[0051] The attribute weight determination unit is configured to determine the attribute weight of each attribute result according to the attribute result of each tag;

[0052] The channel weight obtaining unit is configured to obtain channel weights of different channels, wherein the channel weights are preset weights matching the channels;

[0053] The first result unit is configured to determine the first result probability of each type of the label according to the attribute weight and the channel weight;

[0054] The fusion determination unit is configured to determine the attribute results and channels of a type of the labels with the largest probability of the first result as the attribute results and channels of a fused label, where the fused label is a data fusion result of multiple labels of the first label group.

[0055] In a third aspect, a storage medium stores a program thereon, wherein the program, when executed by a processor, implements any of the above-mentioned data fusion methods.

[0056] By means of the above technical scheme, the present invention provides a data fusion method, device and storage medium, by obtaining a first label group, the first label group includes multiple labels, each of which includes at least: attribute results and channels; according to the attribute results and channels of each label, the classification of each label is determined, wherein the multiple labels of the first label group cover different attribute results of the same attribute, the labels with the same attribute results and the same channels are the same type of labels, and the labels with different attribute results and / or different channels are different types of labels; according to the attribute results of each label, the attribute weight of each attribute result is determined; the channel weight of different channels is obtained, and the channel weight is a preset weight matching the channel; according to the attribute weight and the channel weight, the first result probability of each type of label is determined respectively; the attribute result and channel of the type of label with the largest probability of the first result are determined as the attribute result and channel of the fusion label, and the fusion label is the data fusion result of multiple labels of the first label group. It can be seen from this that the present invention can determine the attribute result and channel of the fusion label according to the attribute results and channels of multiple labels in the first label group and use the fusion label as the fusion result of the first label group, which is applicable to any label, has a wide range of applications, and has certain universal applicability.

[0057] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0059] Figure 1 A flow chart of a data fusion method provided by the present invention is shown;

[0060] Figure 2 A flow chart of another data fusion method provided by the present invention is shown;

[0061] Figure 3 A flow chart of another data fusion method provided by the present invention is shown;

[0062] Figure 4 A schematic structural diagram of a data fusion device provided by the present invention is shown;

[0063] Figure 5A structural schematic diagram of an electronic device provided by the present invention is shown. DETAILED DESCRIPTION

[0064] Insurance companies have a large amount of voice communication data and WeChat text communication data between insurance consultants and customers. A customer may have several voice communications and several WeChat communications with a consultant, register on the company's website, and finally place an order. Each conversation will contain certain basic customer information, such as gender, age, occupation, whether there is social security, etc. Extracting these basic information tags can be used for data fusion, and then building a user portrait system in the insurance field to improve the accuracy of customer information.

[0065] Each conversation may involve a variety of basic information about the customer, and each conversation may also contain some useless information that is not related to the basic information of the customer. Therefore, the basic information of the customer involved in the conversation can be extracted and formed into labels, so that the basic information of the customer can be integrated by integrating the labels later.

[0066] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0067] like Figure 1 As shown, the present invention provides a data fusion method, comprising:

[0068] S100, obtaining a first tag group, wherein the first tag group includes a plurality of tags, each of which includes at least: an attribute result and a channel;

[0069] Optionally, the first tag group can be multiple tags for a specific type of attribute. For example, in a conversation, the gender of a specific customer can appear multiple times, so a group of tags can be formed for the attribute of the customer's "gender". For example, in a phone call, Zhang San's gender is mentioned as "male" 5 times and Zhang San's gender is mentioned as "female" once, then the first tag group can be understood as Zhang San's gender tag group, which includes 6 tags, namely 5 tags with the attribute result of "male" and 1 tag with the attribute result of "female".

[0070] Optionally, if the gender label of Zhang San is also involved in other conversations, such as WeChat, the gender information of Zhang San discussed in different channels can be extracted to establish the gender labels of Zhang San in different channels. Multiple gender labels of Zhang San in different channels and with different attribute results together constitute the first label group, which is not limited in the present invention.

[0071] S200, determining the classification of each tag according to the attribute result and channel of each tag, wherein the multiple tags of the first tag group cover different attribute results of the same attribute, tags with the same attribute results and the same channel are tags of the same category, and tags with different attribute results and / or different channels are tags of different categories;

[0072] Optionally, different attribute results can be understood as different results under the same attribute, because for the same attribute, there can be multiple types of results. For example, gender can be divided into male and female, but male and female are both results under the attribute of "gender".

[0073] Optionally, the same attribute results mentioned in this article can be understood as being literally identical. For example, if the attribute results of two labels are both "male", it can be understood that the attribute results are the same.

[0074] Optionally, the same category of labels mentioned in this article can be understood as: labels with the same attribute results and the same channels; there are three cases of different category labels: (1) labels with the same attribute results and different channels; (2) labels with different attribute results and the same channel; (3) labels with different attribute results and different channels.

[0075] For example, if the first label group is Zhang San's gender label group, and the channels include WeChat and QQ, then the multiple labels of the first label group can be divided into four categories of labels, namely: labels with male gender and WeChat as the channel, labels with male gender and QQ as the channel, labels with female gender and WeChat as the channel, and labels with female gender and QQ as the channel. The present invention does not limit this.

[0076] S300, determining an attribute weight of each attribute result according to the attribute result of each tag;

[0077] Optionally, the first label group corresponds to a type of attribute, and the type of attribute may have multiple results. The attribute weights of different attribute results are determined to facilitate the subsequent determination of the attribute result of the final fusion label according to the attribute weights of the different attribute results. The present invention does not limit this.

[0078] Optionally, the attribute weight of "male" and the attribute weight of "female" can be determined based on the number of "male" tags and the number of "female" tags. Figure 1 In the embodiment shown, in some optional embodiments, the step S300 includes: step 1 and step 2;

[0079] Step 1: Calculate the frequency of occurrence of each attribute result in the first tag group;

[0080] Step 2: Normalize the frequency of occurrence of each attribute result to determine the attribute weight of each attribute result.

[0081] Optionally, normalization is an existing technical solution, and the present invention does not limit this. For details, please refer to the description of normalization in the art.

[0082] Optionally, the normalized result may be directly used as the attribute weight of the corresponding attribute result, or the normalized result may be converted into other results, and then the converted result may be used as the attribute weight of the corresponding attribute result, and the present invention does not impose any limitation on this.

[0083] Optionally, the attribute weight of each attribute result will affect the subsequent fusion result. For example, if the attribute weight of "male" is larger, the probability that the attribute result of the subsequent fusion label is "male" is larger. The present invention does not limit this.

[0084] Optionally, the attribute weight of each attribute result is calculated according to the frequency of occurrence of each attribute result, which conforms to the actual situation and can improve the accuracy of the final fusion result to a certain extent. The present invention does not impose any limitation on this.

[0085] S400, obtaining channel weights of different channels, where the channel weights are preset weights matching the channels;

[0086] Optionally, the channel weights of different channels can be pre-set according to the credibility of each channel. The channel weights of different channels can be the same or different, and can be set according to actual needs, and the present invention does not limit this. For example, the channel weight of social software such as WeChat and QQ can be set to 0.6, and the channel weight of telephone voice to text can be set to 0.4.

[0087] Optionally, the greater the channel weight of a specific channel is, the greater the probability that the channel is used as the final fusion label channel is, and the present invention does not limit this.

[0088] S500, determining the first result probability of each type of the label according to the attribute weight and the channel weight;

[0089] Optionally, multiple tags of the same type have the same attribute results and the same channels, so for any type of tag, there is only one attribute weight and one channel weight. Therefore, the probability of this type of tag being the final fusion tag is determined according to the attribute weight and channel weight of this type of tag, that is, the first result probability.

[0090] Optionally, in order to improve the accuracy of the fusion result of the present invention, a weight calculation method can be used to calculate the first result probability of each type of label. Figure 1In the embodiment shown, in some optional embodiments, the step S500 includes: step three and step four;

[0091] Step 3: Calculate the product of the attribute weight of each type of the label and the corresponding channel weight;

[0092] Optionally, for each type of label, there is only one attribute weight and one channel weight. The attribute weight and channel weight of the label of this type are multiplied together to obtain a product, which can reflect the probability of the label of this type as a fusion label to a certain extent.

[0093] Step 4: Determine the product of each type of labels as the first result probability.

[0094] Optionally, the product can be directly used as the first result probability, or the product of each type of label can be normalized and then the normalized result can be used as the result probability of the corresponding label. For example, there are 4 types of labels, of which the product of the A-type label is 0.2, the product of the B-type label is 0.4, the product of the C-type label is 0.6, and the product of the D-type label is 0.8. Then, 0.2 can be used as the first result probability of the A-type label; 0.4 can be used as the first result probability of the B-type label; 0.6 can be used as the first result probability of the C-type label; and 0.8 can be used as the first result probability of the D-type label. 0.2, 0.4, 0.6 and 0.8 can also be normalized to calculate the first result probability of the A-type label: 0.2÷2=0.1, the first result probability of the B-type label is: 0.4÷2=0.2, the first result probability of the C-type label is: 0.6÷2=0.3 and the first result probability of the D-type label is: 0.8÷2=0.4, and the present invention is not limited to this.

[0095] S600: Determine the attribute results and channels of a type of the labels with the largest first result probability as the attribute results and channels of a fused label, where the fused label is a data fusion result of multiple labels of the first label group.

[0096] Optionally, the present invention determines the attribute results and channels of a category of labels with the largest probability of the first result as the attribute results and channels of the fused labels through calculation, which is more in line with objective facts and has a higher accuracy of data fusion. The present invention does not impose any limitation on this.

[0097] Optional, Figure 1 There is no necessary order of execution among steps S200, S300 and S400 in the implementation manner, and any feasible manner belongs to the protection scope of the present invention, and the present invention does not impose any limitation on this.

[0098] like Figure 2 As shown, combined Figure 1In the embodiment shown, in some optional embodiments, each of the tags at least includes: time, and the method further includes:

[0099] S700, determining the latest tag in each type of tags according to the time of each tag;

[0100] Optionally, the time of each tag may reflect the time when the conversation matching the tag was initiated. For example, if a specific tag is extracted from a WeChat conversation, the tag may carry the initiation time of the WeChat conversation, and of course, the termination time of the WeChat conversation, which is not limited in the present invention.

[0101] Optional, because each label is extracted from various conversation data, the time when the conversation is initiated can affect the credibility of the label to a certain extent. Generally speaking, the closer the conversation initiation time is to the current label, the higher the feasibility is, because some basic information of the customer may change over time.

[0102] Optionally, as mentioned above, the closer the time of a tag is to the present, the higher its credibility is. Therefore, the latest tag in each category of tags, that is, the tag in each category of tags that is closest to the present time, can be determined.

[0103] Optionally, since multiple tags of the same type have the same attribute results and the same channels, but different times, it is sufficient to subsequently determine whether the attribute results and channels of the latest tags of each type can be used as the attribute results and channels of the fused tags.

[0104] S800, determining a time weight of each latest tag match according to the time of each latest tag;

[0105] Optionally, as mentioned above, the closer the time is to the current tag, the higher the credibility is, and the time weight of each latest tag match can be determined. The credibility of each latest tag is reflected by the time weight, and the present invention does not limit this.

[0106] The aforementioned Figure 1 Based on the implementation mode, time is added, so the step S500 includes:

[0107] S510, determining the second result probability of each of the latest tags respectively according to the time weight, the attribute weight and the channel weight;

[0108] Optionally, the present invention does not limit the method for determining the probability of the first result, and any feasible implementation method belongs to the protection scope of the present invention. Figure 2In the embodiment shown, in some optional embodiments, each of the time weights is a preset time weight, and different time ranges of the tags have different matching time weights, and the newer the time, the greater the time weight;

[0109] The step S510 includes: step five and step six;

[0110] Step 5: Calculate the product of the time weight, the attribute weight and the channel weight of each of the latest tags;

[0111] Optionally, the product of each latest label can be calculated by the formula: Q=T×V×C, where Q is the product, T is the time weight, V is the attribute weight, and C is the channel weight, and the present invention does not limit this.

[0112] Step six: determine the product of each of the latest labels as the second result probability.

[0113] Optionally, the product of each latest label can be determined as the second result probability of the corresponding latest label, or the product of each latest label can be transformed and then the transformed results can be used as the second result probability of the corresponding latest label. The present invention is not limited to this.

[0114] For example, the product of each latest label may be normalized, and the normalized results may be used as the second result probability of the corresponding latest label, but the present invention does not impose any limitation on this.

[0115] The aforementioned Figure 1 Based on the implementation mode, a second result probability is added, so the step S600 includes:

[0116] S610: Determine the attribute result and channel of the latest tag with the highest probability of the second result as the attribute result and channel of the fused tag.

[0117] Optionally, the time of merging the label may be set to the time of generating the fused label, or may be set to the time of the latest label with the highest probability of the second result, which is not limited in the present invention.

[0118] like Figure 3 As shown, combined Figure 1 In the illustrated embodiment, in certain optional embodiments, the first tag group includes at least one new tag;

[0119] Optionally, the first tag group may be all historical tags of a type of attribute of a specific customer, for example, all historical tags about Zhang San's gender, including tags from various channels such as WeChat, QQ, and telephone.

[0120] Optionally, any label can be extracted from the conversation data, for example, from text data matching the conversation, and a piece of text data can be extracted to obtain multiple labels or one label, which is not limited in the present invention.

[0121] For example, if Zhang San's gender is mentioned many times in a text data, and generally speaking, the gender will not change, if Zhang San's gender is mentioned many times in the text data as male, and the number of times Zhang San's gender is mentioned as female is less than that of male, then a gender label of Zhang San can be extracted from the text data, and the attribute result of the label is "male". Of course, multiple labels with the attribute result of "male" and some labels with the attribute result of "female" can also be extracted, and the number of labels is consistent with the number of times the corresponding gender is mentioned, and the present invention does not limit this.

[0122] Optionally, the present invention may periodically obtain new text data and then extract new tags based on the new text data, or may instantly extract new tags when new text data is available, and the present invention is not limited to this.

[0123] The new label is obtained through the following process:

[0124] S1000, obtaining text data;

[0125] Optionally, the text data may be text data directly obtained from chat records such as WeChat and QQ, or may be text data converted from voice data such as telephone, WeChat and QQ, and the present invention is not limited to this.

[0126] S2000, determining a channel identifier and a time identifier of the text data, wherein the channel identifier is not limited to one of the following channel identifiers: WeChat, phone, QQ; the time identifier is the conversation initiation time corresponding to the text data;

[0127] Optionally, the channel identifier of the text data can be determined by the source thereof. For example, if the source of a specific piece of text data is a WeChat chat record, the channel identifier of the text data can be determined to be: WeChat, and the channel of the label subsequently extracted from the text data can be set to WeChat, and the present invention does not limit this.

[0128] Optionally, the time identifier of the text data can be determined by the time in the text data. For example, the chat records of social software such as WeChat and QQ generally carry time, and the time can be used as the time identifier in the text data. For another example, by converting the text data obtained by phone recording, the initiation time of the call can also be obtained when the phone recording is obtained, and the call initiation time is used as the time identifier of the text data, and the present invention is not limited to this.

[0129] S3000, executing the first method, the second method and the third method on the text data, thereby obtaining at least one new label;

[0130] Optionally, for any piece of text data, at least one of the following methods 1, 2 and 3 can be executed.

[0131] Optionally, there is no necessary order in which the method 1, the method 2 and the method 3 are executed. For any piece of text data, the method 1, the method 2 and the method 3 can be executed in any order, and the present invention does not impose any limitation on this.

[0132] Method 1: Perform regular expression matching on the text data, and if the text data includes formatting attributes, obtain the formatting attributes; obtain at least one new tag according to the formatting attributes, the channel identifier of the text data, and the time identifier of the text data; wherein the formatting attributes include but are not limited to at least one of the following attributes: email address, mobile phone number, and QQ number;

[0133] Optionally, email addresses, mobile phone numbers, QQ numbers, etc. are all attributes with fixed formats. For these attributes with fixed formats, the formats are fixed, but there can be multiple attribute results. Therefore, the corresponding attribute results can be obtained from the text data by regular matching. For example, by regular matching the email addresses in the text data, if one email address is matched, a label is generated for the email address. If multiple email addresses are matched, corresponding labels are generated for each of the multiple email addresses. The present invention does not limit this.

[0134] Method 2: Input the text data into a pre-trained binary classification model, and if the text data includes a classification attribute, obtain the classification attribute; obtain at least one new label according to the classification attribute, the channel identifier of the text data, and the time identifier of the text data; wherein the classification attribute includes but is not limited to at least one of the following attributes: whether there is social insurance, whether there is medical insurance, whether there is commercial insurance, and whether there is a loan;

[0135] Optionally, for the classification attributes such as whether there is social security, whether there is medical insurance, whether there is commercial insurance, and whether there is a loan, there are only two cases for the attribute results: yes or no. Therefore, the attribute results of the above classification attributes can be determined by a binary classification model and corresponding labels can be generated, and the present invention does not limit this.

[0136] Optionally, for each type of classification attribute, a corresponding binary classification model can be used for processing. For example, one binary classification model can be used for whether there is social security, and another binary classification model can be used for whether there is medical insurance, and so on. The present invention does not limit this.

[0137] Method three: the text data is input into a pre-trained entity recognition model, and if the text data includes entity attributes, the entity attributes are obtained; at least one new label is obtained according to the entity attributes, the channel identifier of the text data and the time identifier of the text data; wherein the entity attributes include but are not limited to at least one of the following attributes: age, disease and occupation.

[0138] Optionally, for attributes such as age, disease, and occupation, the attribute results are not in a fixed format and are variable. Therefore, the entity recognition model can be used to identify the attribute results of each entity from the text data, taking age, disease, and occupation as entities, respectively. The present invention does not limit this.

[0139] Optionally, different entities can be identified using different entity recognition models to improve the accuracy of recognition. For example, one entity recognition model can be used for age, and another entity recognition model can be used for disease, and the present invention does not limit this.

[0140] Combination Figure 3 In the illustrated implementation manner, in certain optional implementation manners, the plurality of labels of the first label group correspond to a type of attributes in the formatting attributes, or correspond to a type of attributes in the classification attributes, or correspond to a type of attributes in the entity attributes.

[0141] Optionally, the first tag group mentioned in this article can be a tag of a specific type of attribute, for example, it can be a tag about gender, a tag about mobile phone number, or a tag about age, and the present invention is not limited to this.

[0142] Combination Figure 3 In the illustrated embodiment, in certain optional embodiments, the old tag is a tag that is pre-acquired through a process of acquiring the new tag and pre-stored in a storage device.

[0143] Optionally, the first label group can be all historical labels of a specific type of attributes. New labels can be obtained through the above process, and old labels can also be obtained through the above process and stored in a storage device, so that the old labels can be directly obtained from the storage device later to improve efficiency. The present invention does not impose any restrictions on this.

[0144] Combination Figure 3 In the embodiment shown, in some optional embodiments, the binary classification model is a fasttext model, and the training process of the fasttext model includes:

[0145] Obtaining a labeled data set, wherein the labeled data set includes at least one of the following data sets: a data set with social insurance, a data set without social insurance, a data set with medical insurance, a data set without medical insurance, a data set with commercial insurance, a data set without commercial insurance, a data set with loans, and a data set without loans;

[0146] The annotated data set is input into the fasttext classification algorithm, so as to perform binary classification training on the fasttext classification algorithm.

[0147] Combination Figure 3 In the embodiment shown, in some optional embodiments, the entity recognition model includes: BERT algorithm, LSTM algorithm and CRF algorithm, and the training process of the entity recognition model includes: step one, step two, step three and step four;

[0148] Step 1: input the pre-constructed structured training set into the BERT algorithm, wherein the format of each training corpus in the structured training set meets the format requirements of the BERT algorithm;

[0149] Step 2, the BERT algorithm outputs the semantic vectors corresponding to each of the training corpora to the LSTM algorithm, wherein the semantic vectors incorporate the full-text semantic information of the corresponding training corpora;

[0150] Step 3: The LSTM algorithm performs sequence labeling on the semantic vectors input by the BERT algorithm to obtain a label sequence and output it to the CRF algorithm;

[0151] Step 4: The CRF algorithm outputs the probability of the label sequence as a test result for testing the structured training set. When the test result meets the preset result requirements, the BERT algorithm, LSTM algorithm and CRF algorithm are jointly constructed as the entity recognition model.

[0152] like Figure 4 As shown, the present invention provides a data fusion device, including: a label group obtaining unit 100, a classification determination unit 200, an attribute weight determination unit 300, a channel weight obtaining unit 400, a first result unit 500 and a fusion determination unit 600;

[0153] The tag group obtaining unit 100 is configured to obtain a first tag group, wherein the first tag group includes a plurality of tags, each of which includes at least: an attribute result and a channel;

[0154] The classification determination unit 200 is configured to determine the classification of each tag according to the attribute result and channel of each tag, wherein the multiple tags of the first tag group cover different attribute results of the same attribute, tags with the same attribute results and the same channel are tags of the same category, and tags with different attribute results and / or different channels are tags of different categories;

[0155] The attribute weight determination unit 300 is configured to determine the attribute weight of each attribute result according to the attribute result of each tag;

[0156] The channel weight obtaining unit 400 is configured to obtain channel weights of different channels, where the channel weights are preset weights that match the channels;

[0157] The first result unit 500 is configured to determine the first result probability of each type of the tag according to the attribute weight and the channel weight;

[0158] The fusion determination unit 600 is configured to determine the attribute results and channels of a type of the labels with the largest probability of the first result as the attribute results and channels of a fused label, where the fused label is a data fusion result of multiple labels of the first label group.

[0159] Combination Figure 4 In certain optional embodiments of the embodiment shown, the apparatus further comprises: a new label determination unit and a new label determination unit;

[0160] The new label determination unit is configured to determine the latest label in each type of labels according to the time of each label, wherein each label at least includes: time;

[0161] The new label determination unit is configured to determine a time weight of each latest label match according to the time of each latest label;

[0162] The first result unit 500 includes: a second result unit;

[0163] The second result unit is configured to determine the second result probability of each of the latest tags according to the time weight, the attribute weight and the channel weight;

[0164] The fusion determination unit 600 includes: a fusion unit;

[0165] The fusion unit is configured to determine the attribute result and channel of the latest label with the highest probability of the second result as the attribute result and channel of the fused label.

[0166] In combination with the previous embodiment, in some optional embodiments, the second result unit includes: a first product subunit and a second result subunit;

[0167] The first product subunit is configured to calculate the product of the time weight, the attribute weight and the channel weight of each of the latest tags, wherein each of the time weights is a preset time weight, and different time ranges of the tags have different matching time weights, and the newer the time, the greater the time weight;

[0168] The second result subunit is configured to determine the product of each of the latest labels as the second result probability.

[0169] In combination with the previous embodiment, in some optional embodiments, the first result unit 500 includes: a second product subunit and a first result subunit;

[0170] The second product subunit is configured to calculate the product of the attribute weight of each type of the tag and the corresponding channel weight;

[0171] The first result subunit is configured to determine the product of each type of the labels as the first result probability.

[0172] In combination with the previous embodiment, in some optional embodiments, the attribute weight determination unit 300 includes: a frequency determination subunit and a weight determination subunit;

[0173] The frequency determination subunit is configured to calculate the frequency of occurrence of each attribute result in the first tag group;

[0174] The weight determination subunit is configured to normalize the frequency of occurrence of each attribute result, so as to determine the attribute weight of each attribute result.

[0175] Combination Figure 4 In the illustrated embodiment, in certain optional embodiments, the apparatus further comprises: a text data obtaining unit, a first determining unit, and a new label obtaining unit;

[0176] The text data obtaining unit is configured to obtain text data;

[0177] The first determining unit is configured to determine a channel identifier and a time identifier of the text data, wherein the channel identifier is not limited to one of the following channel identifiers: WeChat, phone, QQ; the time identifier is a conversation initiation time corresponding to the text data;

[0178] The new label obtaining unit is configured to execute the first, second and third methods on the text data, thereby obtaining at least one new label;

[0179] Method 1: Perform regular expression matching on the text data, and if the text data includes formatting attributes, obtain the formatting attributes; obtain at least one new tag according to the formatting attributes, the channel identifier of the text data, and the time identifier of the text data; wherein the formatting attributes include but are not limited to at least one of the following attributes: email address, mobile phone number, and QQ number;

[0180] Method 2: Input the text data into a pre-trained binary classification model, and if the text data includes a classification attribute, obtain the classification attribute; obtain at least one new label according to the classification attribute, the channel identifier of the text data, and the time identifier of the text data; wherein the classification attribute includes but is not limited to at least one of the following attributes: whether there is social insurance, whether there is medical insurance, whether there is commercial insurance, and whether there is a loan;

[0181] Method three: the text data is input into a pre-trained entity recognition model, and if the text data includes entity attributes, the entity attributes are obtained; at least one new label is obtained according to the entity attributes, the channel identifier of the text data and the time identifier of the text data; wherein the entity attributes include but are not limited to at least one of the following attributes: age, disease and occupation.

[0182] In combination with the previous embodiment, in some optional embodiments, the device further includes: a labeling data obtaining unit and a classification training unit;

[0183] The annotated data obtaining unit is configured to obtain an annotated data set, wherein the annotated data set includes at least one of the following data sets: a data set with social insurance, a data set without social insurance, a data set with medical insurance, a data set without medical insurance, a data set with commercial insurance, a data set without commercial insurance, a data set with loans, and a data set without loans;

[0184] The classification training unit is configured to input the annotated data set into a fasttext classification algorithm, thereby performing binary classification training on the fasttext classification algorithm.

[0185] Optionally, in certain optional embodiments, the device further comprises: a first input unit, a BERT algorithm unit, an LSTM algorithm unit, and a CRF algorithm unit;

[0186] The first input unit is configured to input a pre-constructed structured training set into the BERT algorithm, wherein the format of each training corpus in the structured training set meets the format requirements of the BERT algorithm;

[0187] The BERT algorithm unit is configured to execute the BERT algorithm to output semantic vectors corresponding to each of the training corpora to the LSTM algorithm, wherein the semantic vectors incorporate the full-text semantic information of the corresponding training corpora;

[0188] The LSTM algorithm unit is configured to execute the LSTM algorithm to perform sequence labeling on each of the semantic vectors input by the BERT algorithm, thereby obtaining a label sequence and outputting it to the CRF algorithm;

[0189] The CRF algorithm unit is configured to execute the CRF algorithm to output the probability of the label sequence as a test result of testing the structured training set, and when the test result meets the preset result requirements, the BERT algorithm, the LSTM algorithm and the CRF algorithm are jointly constructed as the entity recognition model.

[0190] The present invention provides a storage medium on which a program is stored, and the program is executed by a processor to implement any of the above-mentioned data fusion methods.

[0191] like Figure 5 As shown, the present invention provides an electronic device 70, which includes at least one processor 701, and at least one memory 702 and a bus 703 connected to the processor 701; wherein the processor 701 and the memory 702 communicate with each other through the bus 703; the processor 701 is used to call the program instructions in the memory 702 to execute any of the data fusion methods described above.

[0192] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0193] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0194] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0195] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data fusion method, characterized in that: include: Obtaining a first tag group, the first tag group includes a plurality of tags, each of the tags includes at least: an attribute result and a channel, wherein basic information of the customers involved in the conversation is extracted to form the tags; Determine the classification of each tag according to the attribute result and channel of each tag, wherein the multiple tags of the first tag group cover different attribute results of the same attribute, tags with the same attribute results and the same channel are tags of the same category, and tags with different attribute results and / or different channels are tags of different categories; Determine the attribute weight of each attribute result according to the attribute result of each tag; Obtaining channel weights of different channels, wherein the channel weights are preset weights matching the channels; Determine the first result probability of each type of the label according to the attribute weight and the channel weight; The attribute results and channels of a type of the labels with the largest probability of the first result are determined as the attribute results and channels of a fused label, where the fused label is a data fusion result of multiple labels of the first label group.

2. The method according to claim 1, characterized in that Each of the tags at least includes: time, and the method further includes: Determine the latest tag in each category of tags according to the time of each tag; Determine the time weight of each latest tag match according to the time of each latest tag; Determining the first result probability of each type of the label according to the attribute weight and the channel weight includes: Determine the second result probability of each of the latest tags respectively according to the time weight, the attribute weight and the channel weight; The step of determining the attribute result and channel of the label of a category with the largest probability of the first result as the attribute result and channel of the fused label includes: The attribute result and channel of the latest label with the highest probability of the second result are determined as the attribute result and channel of the fused label.

3. The method according to claim 2, characterized in that Each of the time weights is a preset time weight, and different time ranges of the tags have different matching time weights, and the newer the time, the greater the time weight; The determining the second result probability of each of the latest tags respectively according to the time weight, the attribute weight and the channel weight includes: Calculating the product of the time weight, the attribute weight and the channel weight of each of the latest tags; The product of each of the latest labels is determined as the second result probability.

4. The method according to claim 1, characterized in that Determining the first result probability of each type of the label according to the attribute weight and the channel weight includes: Calculate the product of the attribute weight of each type of the label and the corresponding channel weight; The product of each class of the labels is determined as the first result probability.

5. The method according to claim 1, characterized in that: Determining the attribute weight of each attribute result according to the attribute result of each tag includes: Calculate the frequency of occurrence of each attribute result in the first label group; The frequency of occurrence of each attribute result is normalized to determine the attribute weight of each attribute result.

6. The method according to claim 1, characterized in that The first tag group includes at least one new tag; The new label is obtained through the following process: Get text data; Determine a channel identifier and a time identifier of the text data, wherein the channel identifier is not limited to one of the following channel identifiers: WeChat, phone, QQ, and the time identifier is the time when the corresponding conversation of the text data is initiated; Execute the method 1, the method 2 and the method 3 on the text data, so as to obtain at least one new label; Method 1: Perform regular expression matching on the text data, and if the text data includes formatting attributes, obtain the formatting attributes; obtain at least one new tag according to the formatting attributes, the channel identifier of the text data, and the time identifier of the text data; wherein the formatting attributes include but are not limited to at least one of the following attributes: email address, mobile phone number, and QQ number; Method 2: Input the text data into a pre-trained binary classification model, and if the text data includes a classification attribute, obtain the classification attribute; obtain at least one new label according to the classification attribute, the channel identifier of the text data, and the time identifier of the text data; wherein the classification attribute includes but is not limited to at least one of the following attributes: whether there is medical insurance, whether there is commercial insurance, and whether there is a loan; Method three: the text data is input into a pre-trained entity recognition model, and if the text data includes entity attributes, the entity attributes are obtained; at least one new label is obtained according to the entity attributes, the channel identifier of the text data and the time identifier of the text data; wherein the entity attributes include but are not limited to at least one of the following attributes: age, disease and occupation.

7. The method according to claim 6, characterized in that The binary classification model is a fasttext model, and the training process of the fasttext model includes: Obtaining a labeled data set, wherein the labeled data set includes at least one of the following data sets: a data set with medical insurance, a data set without medical insurance, a data set with commercial insurance, a data set without commercial insurance, a data set with loans, and a data set without loans; The annotated data set is input into the fasttext classification algorithm, so as to perform binary classification training on the fasttext classification algorithm.

8. The method according to claim 6, characterized in that The entity recognition model includes: BERT algorithm, LSTM algorithm and CRF algorithm, and the training process of the entity recognition model includes: Inputting a pre-constructed structured training set into the BERT algorithm, wherein the format of each training corpus in the structured training set conforms to the format requirement of the BERT algorithm; The BERT algorithm outputs semantic vectors corresponding to each of the training corpora to the LSTM algorithm, wherein the semantic vectors incorporate the full-text semantic information of the corresponding training corpora; The LSTM algorithm performs sequence labeling on each of the semantic vectors input by the BERT algorithm, thereby obtaining a label sequence and outputting it to the CRF algorithm; The CRF algorithm outputs the probability of the label sequence as a test result of testing the structured training set. When the test result meets the preset result requirement, the BERT algorithm, the LSTM algorithm and the CRF algorithm are jointly constructed as the entity recognition model.

9. A data fusion device, characterized in that: include: A label group obtaining unit, a classification determining unit, an attribute weight determining unit, a channel weight obtaining unit, a first result unit and a fusion determining unit; The tag group obtaining unit is configured to obtain a first tag group, wherein the first tag group includes a plurality of tags, each of which includes at least: an attribute result and a channel, wherein basic information of the customer involved in the conversation is extracted to form the tag; The classification determination unit is configured to determine the classification of each of the tags according to the attribute results and channels of each of the tags, wherein the multiple tags of the first tag group cover different attribute results of the same attribute, tags with the same attribute results and the same channels are tags of the same category, and tags with different attribute results and / or different channels are tags of different categories; The attribute weight determination unit is configured to determine the attribute weight of each attribute result according to the attribute result of each tag; The channel weight obtaining unit is configured to obtain channel weights of different channels, wherein the channel weights are preset weights matching the channels; The first result unit is configured to determine the first result probability of each type of the label according to the attribute weight and the channel weight; The fusion determination unit is configured to determine the attribute results and channels of a type of the labels with the largest probability of the first result as the attribute results and channels of a fused label, where the fused label is a data fusion result of multiple labels of the first label group.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the data fusion method according to any one of claims 1 to 8 is implemented.

Images