A method and device for generating multidimensional key-value pair data
By screening and processing business data, multi-dimensional key-value pair data are generated, the problem of insufficient expression ability of business data is solved, deeper information is extracted and expressed, and the richness and analyticity of the data is improved.
Patent Information
- Application Number
- CN202111134047.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-27
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-09-27
AI Technical Summary
The existing business data has weak expressive capabilities, making it difficult to extract deeper information, and cannot effectively improve the expressive capabilities of the data.
By acquiring multiple initial business data, the first attribute group and its specified attribute values are determined, the data is filtered and data processed based on these attribute groups and values, and multi-dimensional key-value pair data is generated, thereby extracting and expressing deep features.
It improves the data expression ability, makes data information richer, is easy to store and retrieve, and enhances the analysis and decision-making support for the service level of the service platform.
Smart Images

Figure CN113849514B_ABST
Abstract
Description
Technical Field
[0001] One or more embodiments of the present specification relate to the field of data processing technology, and in particular, to a method and device for generating multi-dimensional key-value pair data. Background Art
[0002] With the development of technology, various service platforms can provide more and more services. The service platform can record a large amount of business data related to the business, and analyzing these business data can further improve the service level. The form of business data is similar to log data, and usually records data from multiple aspects. For example, in a bank loan scenario, a business data may include data such as the borrower's name, gender, age, loan city, loan term, loan amount, whether the loan has been repaid, and whether there is any overdue payment. Such business data can reflect certain information, but there is a problem of weak expression ability. At present, there is a need to extract deeper information from these business data.
[0003] Therefore, we hope to have an improved solution that can extract deeper features from business data and improve the expressiveness of the data. Summary of the invention
[0004] One or more embodiments of this specification describe a method and device for generating multi-dimensional key-value pair data to extract deeper features from business data and improve the expressiveness of data. The specific technical solution is as follows.
[0005] In a first aspect, an embodiment provides a method for generating multidimensional key-value pair data, comprising:
[0006] Acquire multiple initial business data, where one initial business data contains attribute values of several attributes;
[0007] Determine a first attribute group and designated attribute values of the attributes included therein, and determine a key in key-value pair data to be generated based on the first attribute group and the corresponding designated attribute values;
[0008] Based on the first attribute group and the corresponding designated attribute value, the plurality of initial business data are screened to obtain selected business data;
[0009] Based on a preset data processing method, data processing is performed on the attribute values in the selected business data, and the values in the key-value pair data are determined based on the data processing results.
[0010] In one implementation, the key includes a field portion and a field value portion, wherein a group of field values corresponds to a subkey; the key is determined in the following manner:
[0011] Determine the attributes in the first attribute group as fields included in the key;
[0012] Based on the specified attribute values of the attributes in the first attribute group, one or more groups of field values included in the key are determined and correspondingly used as one or more subkeys of the key.
[0013] In one implementation, the first attribute group includes multiple attributes; the specified attribute value of any attribute in the first attribute group is one or more;
[0014] The step of determining the attributes in the first attribute group as fields included in the key comprises:
[0015] Determine the multiple attributes in the first attribute group as multiple fields included in the key respectively;
[0016] The step of determining one or more groups of field values contained in the key based on the specified attribute values of the attributes in the first attribute group comprises:
[0017] For multiple attributes in the first attribute group, when the specified attribute value of each of the multiple attributes is one, the specified attribute values of the multiple attributes are respectively determined as corresponding field values of the fields included in the key to obtain a group of field values;
[0018] For multiple attributes in the first attribute group, when there are multiple specified attribute values for at least one of the multiple attributes, the specified attribute values of the multiple attributes are combined to obtain multiple attribute value combinations; the multiple attribute value combinations are respectively determined as corresponding field values of the fields contained in the key to obtain multiple groups of field values.
[0019] In one implementation, the step of screening the plurality of initial business data based on the first attribute group and the corresponding designated attribute value includes:
[0020] For any subkey contained in the key, the plurality of initial business data are screened using the attribute corresponding to the subkey and the specified attribute value to obtain a group of selected business data corresponding to the subkey.
[0021] In one embodiment, the step of performing data processing on the attribute value in the selected business data based on a preset data processing method, and determining the value in the key-value pair data based on the data processing result includes:
[0022] For any subkey contained in the key, based on a preset data processing method, data processing is performed on the attribute values in a set of selected business data corresponding to the subkey, and the obtained data processing result is determined as the value in the key-value pair data corresponding to the subkey.
[0023] In one embodiment, the step of performing data processing on the attribute value in the selected business data includes:
[0024] determining a second attribute group;
[0025] Based on a preset data processing method, data processing is performed on the attribute values of the second attribute group in the selected business data.
[0026] In one embodiment, the value includes a field portion and a field value portion; the data processing method includes one or more; any first data processing method corresponds to the first field of the value;
[0027] The step of performing data processing on the attribute value in the selected business data based on a preset data processing method, and determining the value in the key-value pair data based on the data processing result includes:
[0028] Based on the first data processing method, data processing is performed on the attribute value in the selected business data, and the obtained data processing result is determined as the field value of the first field.
[0029] In one embodiment, the data processing method includes at least one of the following methods:
[0030] Sum, find the average, perform statistics on the set attribute values, find the variance, find the standard deviation, and find the covariance.
[0031] In one implementation, there are multiple first attribute groups; and the method further includes:
[0032] Determine the association relationship between the attributes included in the plurality of first attribute groups; wherein the association relationship is used to identify the relationship between different key-value pair data.
[0033] In one embodiment, after obtaining the key-value pair data, the method further includes:
[0034] Storing the key-value pair data in a database;
[0035] When data retrieval is required, several data engines are used to retrieve data from the database.
[0036] In a second aspect, an embodiment provides a device for generating multidimensional key-value pair data, including:
[0037] The acquisition module is configured to acquire a plurality of initial business data, wherein one initial business data includes attribute values of a plurality of attributes;
[0038] A determination module configured to determine a first attribute group and designated attribute values of the attributes included therein, and a key in key-value pair data to be generated determined based on the first attribute group and the corresponding designated attribute values;
[0039] A screening module configured to screen the plurality of initial business data based on the first attribute group and the corresponding designated attribute value to obtain selected business data;
[0040] The processing module is configured to perform data processing on the attribute value in the selected business data based on a preset data processing method, and determine the value in the key-value pair data based on the data processing result.
[0041] In one implementation, the key includes a field portion and a field value portion, wherein a group of field values corresponds to a subkey; the determination module is further configured to determine the key by performing the following operations:
[0042] Determine the attributes in the first attribute group as fields included in the key;
[0043] Based on the specified attribute values of the attributes in the first attribute group, one or more groups of field values included in the key are determined and correspondingly used as one or more subkeys of the key.
[0044] In one implementation, the first attribute group includes multiple attributes; the specified attribute value of any attribute in the first attribute group is one or more;
[0045] The determining module, when determining the attributes in the first attribute group as the fields included in the key, includes:
[0046] Determine the multiple attributes in the first attribute group as multiple fields included in the key respectively;
[0047] The determining module, when determining one or more groups of field values included in the key based on the specified attribute value of the attribute in the first attribute group, includes:
[0048] For multiple attributes in the first attribute group, when the specified attribute value of each of the multiple attributes is one, the specified attribute values of the multiple attributes are respectively determined as corresponding field values of the fields included in the key to obtain a group of field values;
[0049] For multiple attributes in the first attribute group, when there are multiple specified attribute values for at least one of the multiple attributes, the specified attribute values of the multiple attributes are combined to obtain multiple attribute value combinations; the multiple attribute value combinations are respectively determined as corresponding field values of the fields contained in the key to obtain multiple groups of field values.
[0050] In one embodiment, the screening module is specifically configured as follows:
[0051] For any subkey contained in the key, the plurality of initial business data are screened using the attribute corresponding to the subkey and the specified attribute value to obtain a group of selected business data corresponding to the subkey.
[0052] In one implementation, the processing module is specifically configured as follows:
[0053] For any subkey contained in the key, based on a preset data processing method, data processing is performed on the attribute values in a set of selected business data corresponding to the subkey, and the obtained data processing result is determined as the value in the key-value pair data corresponding to the subkey.
[0054] In one implementation, the processing module is specifically configured as follows:
[0055] determining a second attribute group;
[0056] Based on a preset data processing method, data processing is performed on the attribute values of the second attribute group in the selected business data.
[0057] In one embodiment, the value includes a field portion and a field value portion; the data processing method includes one or more; any first data processing method corresponds to the first field of the value;
[0058] The processing module is specifically configured as follows:
[0059] Based on the first data processing method, data processing is performed on the attribute value in the selected business data, and the obtained data processing result is determined as the field value of the first field.
[0060] In a third aspect, an embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute any one of the methods described in the first aspect.
[0061] In a fourth aspect, an embodiment provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any one of the first aspects is implemented.
[0062] In the method and device provided in the embodiments of this specification, multi-dimensional key-value pair data can be determined based on multiple initial business data. Among them, multiple business data are screened based on the first attribute group and the specified attribute value, data processing is performed on the selected business data obtained by screening, and the value in the key-value pair data is determined based on the data processing result, and the key in the key-value pair data is determined based on the first attribute group and the specified attribute value, and the processed data is expressed in the form of key-value pair data. In the embodiments of this specification, deep features are extracted from selected business data through screening and data processing, and the extracted deep features are expressed in the form of multi-dimensional key-value pair data, which can improve the expressiveness of data. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0064] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;
[0065] Figure 2 A flowchart of a method for generating multidimensional key-value pair data provided by an embodiment;
[0066] Figure 3 A schematic diagram of a key in a key-value pair data;
[0067] Figure 4 A schematic block diagram of a device for generating multi-dimensional key-value pair data provided in an embodiment. DETAILED DESCRIPTION
[0068] The solution provided in this specification is described below in conjunction with the accompanying drawings.
[0069] Figure 1 This is a schematic diagram of an implementation scenario of an embodiment disclosed in this specification. Figure 1 In the service platform (such as the platform provided by banking institutions, the platform provided by telecommunications providers, the shopping platform and other institutional platforms), a large amount of business data can be recorded, and business data is continuously generated as time goes on. Business data contains the attribute values of several attributes. The processing platform can obtain a large amount of business data, and perform deep feature extraction on these business data, and represent the extracted features in the form of key-value pairs. The extracted features can reflect the deeper and richer information of multiple business data, and are also easy to store and retrieve. They can be used by relevant personnel for analysis and decision-making in the future, triggering solutions to improve the service level of the service platform.
[0070] The service platform can also realize the function of the processing platform. Business data can be obtained from log data. For example, in a bank lending scenario, business data can be in the form listed in Table 1.
[0071] Table 1
[0072]
[0073] Table 1 above lists only 6 business data as an example. The first row is each attribute, and the second to seventh rows are 6 business data, each of which contains attribute values of multiple attributes. Of course, in other cases, a business data may also contain only one attribute.
[0074] As can be seen from Table 1, the business data records the basic information related to the business, and the information that can be expressed by each business data is limited. In order to enhance the expressive power of the data and extract deeper features from the business data, an embodiment of this specification provides a method for generating multidimensional key-value pair data. It includes: step S210, obtaining multiple initial business data; step S220, determining the specified attribute values of the first attribute group and the attributes contained therein, and the keys in the key-value pair data to be generated based on the first attribute group and the corresponding specified attribute values; step S230, based on the first attribute group and the corresponding specified attribute values, screening multiple initial business data to obtain selected business data; step S240, based on a preset data processing method, performing data processing on the attribute values in the selected business data, and determining the values in the key-value pair data based on the data processing results. Among them, the initial business data can be but is not limited to the business data exemplified in Table 1. By screening the initial business data, the selected business data including the first attribute group and the specified attribute value can be obtained. The attribute value of the selected business data is processed, and the data within a specific range can be processed and counted in a targeted manner, and different data processing results can be obtained from different dimensions, making the data information richer. The use of multidimensional key-value pairs can make the expression of the data better. The following specific embodiments are used to illustrate this application.
[0075] Figure 2 A flowchart of a method for generating multi-dimensional key-value pair data provided in an embodiment. The execution subject of the method can be any device, equipment, platform, equipment cluster, etc. with computing and processing capabilities. The method includes the following steps S210 to S240.
[0076] Step S210, obtaining a plurality of initial service data, wherein one initial service data includes attribute values of a plurality of attributes. The attributes included in each initial service data may be the same. The initial service data may include attribute values of one or more attributes as described in Table 1.
[0077] The multiple initial business data may be obtained from the service platform by the processing platform as the execution subject, or may be obtained from the device where the database for storing data is located. The multiple initial business data may also be obtained from the log data. For example, the log data may be directly used as the initial business data, or may be obtained after initialization processing of the daily master data. The initialization processing may include, for example, cleaning, screening, etc.
[0078] The number of initial business data obtained can be a preset number or an indefinite number. When step S210 is executed, it can be executed multiple times according to a preset period. For example, in a monitoring scenario, log data is continuously generated, and the log data obtained multiple times can be used as initial business data; or multiple log data obtained in each preset period can be used as multiple initial business data.
[0079] Step S220 : determining the first attribute group and the designated attribute values of the attributes included therein, and determining the key key1 in the key-value pair data data1 to be generated based on the first attribute group and the corresponding designated attribute values.
[0080] The first attribute group includes one or more attributes. Each attribute has a corresponding designated attribute value, and the corresponding designated attribute value of each attribute can be one or more. When determining the first attribute group and the corresponding designated attribute value, the execution subject can directly obtain the set first attribute group and the corresponding designated attribute value, or determine the first attribute group and the corresponding designated attribute value according to the input operation of relevant personnel.
[0081] The attributes in the first attribute group and the corresponding designated attribute values may be selected from the multiple initial business data acquired in step S210. The attributes in the first attribute group may be selected from the multiple attributes of the initial business data as needed. The designated attribute value of the attribute (e.g., attribute 1) included in the first attribute group may be selected from the multiple attribute values corresponding to attribute 1 of the multiple initial business data. The attributes of the multiple initial business data may include the attributes in the first attribute group, and the attribute values of the attributes of the multiple initial business data may include the designated attribute value of the corresponding attribute in the first attribute group.
[0082] For example, the first attribute group includes the three attributes of "city", "loan purpose" and "loan amount" in Table 1, and the corresponding specified attribute values of each attribute are, "city" = Beijing, "loan purpose" = 1, "loan amount" = 5k, 4k. See Table 2 below for details.
[0083] Table 2
[0084] City Loan Purpose Loan amount Beijing 1 5k, 4k
[0085] The number of designated attribute values corresponding to the attribute "loan amount" is 2. The setting of the first attribute group and the corresponding designated attribute values can reflect the specific data group with important significance, as well as the data attributes and attribute values that are of concern.
[0086] Key-Value is a data representation format, where the key and the value are in a corresponding relationship. In step S220, based on the first attribute group and the corresponding specified attribute value, the key key1 in the key-value pair data data1 to be generated can be determined. The key key1 can be determined in advance based on the first attribute group and the corresponding specified attribute value, and the key key1 is directly obtained in step S220; or it can be determined based on the first attribute group and the corresponding specified attribute value when step S220 is executed.
[0087] In this embodiment, the key key1 may include a field part and a field value part, wherein a group of field values corresponds to a subkey. When determining the key key1 in the key-value pair data data1 based on the first attribute group and the corresponding specified attribute value, the following steps 1a and 2a may be used.
[0088] Step 1a: determine the attributes in the first attribute group as fields included in the key key1.
[0089] Step 2a: Based on the specified attribute values of the attributes in the first attribute group, one or more groups of field values included in the key key1 are determined, and correspondingly used as one or more subkeys of the key key1.
[0090] In step 1a, when the first attribute group includes one attribute, the attribute can be directly used as the field of key key1. When the first attribute group includes multiple attributes, the multiple attributes in the first attribute group can be respectively determined as multiple fields included in key key1.
[0091] In step 2a, the specified attribute value of any attribute in the first attribute group may be one or more.
[0092] For multiple attributes in the first attribute group, when each of the multiple attributes has one specified attribute value, the specified attribute values of the multiple attributes are respectively determined as corresponding field values of the fields included in the key key1 to obtain a group of field values.
[0093] For example, the first attribute group includes three attributes: "city", "loan purpose", and "loan amount". The corresponding specified attribute values of each attribute are: "city" = Beijing, "loan purpose" = 1, and "loan amount" = 5k. Then, "city", "loan purpose", and "loan amount" can be used as fields in key1 respectively; "Beijing", "1", and "5k" are determined as the corresponding field values of each field in key1 respectively, forming a group of field values. See Table 3 below for details.
[0094] Table 3
[0095] City Loan Purpose Loan amount Beijing 1 5k
[0096] The first row in the table is the field part, including the three fields in key1; the second row is the field value part, including the corresponding field values of the three fields, which is a group of field values. "City" = Beijing, "Purpose of loan" = 1, "Loan amount" = 5k, which is the content contained in a subkey in key1.
[0097] For multiple attributes in the first attribute group, when there are multiple specified attribute values for at least one of the multiple attributes, the specified attribute values of the multiple attributes are combined to obtain multiple attribute value combinations, and the multiple attribute value combinations are respectively determined as corresponding field values of the fields included in the key Key1 to obtain multiple groups of field values.
[0098] For example, taking the first attribute group and designated attribute value in Table 2 as an example, the designated attribute values of the attribute "loan amount" include 5k and 4k. When determining the subkey, the designated attribute values of the three attributes can be combined to obtain two groups of attribute value combinations, namely "Beijing, 1, 5k" and "Beijing, 1, 4k". These two attribute value combinations can be used as two groups of field values, corresponding to two subkeys respectively. Figure 3 This is a schematic diagram of a key in a key-value pair data. It shows the field part and field value part of the key, as well as two subkeys.
[0099] In step S220, the first attribute group and the corresponding specified attribute value, and the key key1 in the key-value pair data data1, can be one type of data, or two types of data obtained separately. In the former case, the first attribute group, the specified attribute value, and the key key1 are determined. In the latter case, the first attribute group and the specified attribute value, and the corresponding relationship between the first attribute group, the specified attribute value, and the key key1 are determined; or, the key key1, and the corresponding relationship between the first attribute group, the specified attribute value, and the key key1 are determined. From Table 2 and Figure 3 It can be seen that there is a correspondence between the first attribute group, the specified attribute value and the key key1.
[0100] Step S230, based on the first attribute group and the corresponding designated attribute value, multiple initial business data are screened to obtain selected business data. The multiple initial business data are screened, that is, the initial business data whose attribute value in the first attribute group is the designated attribute value is screened from the multiple initial business data as the selected business data.
[0101] For example, when the first attribute group and the designated attribute value are data in Table 2, the initial business data shown in the second row, the fifth row, and the sixth row may be screened out from the multiple initial business data in Table 1 as selected business data.
[0102] When the key key1 contains one or more subkeys (see Figure 3 As shown in the figure, when filtering multiple initial business data, the multiple initial business data can be filtered for any subkey contained in the key key1 by using the attribute corresponding to the subkey and the specified attribute value to obtain a set of selected business data corresponding to the subkey. For each subkey in the key key1, a corresponding set of selected business data is filtered out.
[0103] A subkey corresponds to a set of field values, which can be used as a set of filtering conditions to filter out the corresponding selected business data from multiple initial business data. Figure 3 For subkey 1 in , the attribute corresponding to subkey 1 and the specified attribute value can be used to filter out the second and fifth rows from the initial business data in Table 1 to obtain the first group of selected business data; Figure 3 The subkey 2 in Table 1 can be used to filter out the 6th row from the initial business data in Table 1 by using the attributes and specified attribute values corresponding to the subkey 2 to obtain the second set of selected business data. Among them, the attributes and specified attribute values corresponding to the subkey 1 include: city = Beijing, loan purpose = 1, loan amount = 5k. The attributes and specified attribute values corresponding to the subkey 1 include: city = Beijing, loan purpose = 1, loan amount = 4k.
[0104] Step S240: Based on a preset data processing method, data processing is performed on the attribute value in the selected business data, and the value value1 in the key-value pair data data1 is determined based on the data processing result. Here, the attribute value in the selected business data may be one or more.
[0105] When there are one or more attributes in the first attribute group, and the specified attribute values corresponding to the attributes contained in the first attribute group are all one, the key key1 contains a subkey, and the corresponding selected business data is a group. The subkey has a value corresponding to it. In this embodiment, the value value1 in the key-value pair data data1 can be determined directly according to the data processing result after the data processing is performed on the group of selected business data.
[0106] When there are one or more attributes in the first attribute group, and there are multiple specified attribute values corresponding to at least one attribute contained in the first attribute group, the key key1 contains multiple subkeys, and the corresponding selected business data is multiple groups, and each subkey corresponds to a group of selected business data. There are multiple subkeys, and there are values corresponding to each subkey. In this embodiment, for any subkey k1 contained in the key key1, based on the preset data processing method, data processing is performed on the attribute value in a group of selected business data corresponding to the subkey k1, and the obtained data processing result is determined as the value v1 in the key-value pair data data1 corresponding to the subkey k1. That is, when data processing is performed on the selected business data, it is executed separately according to different groups, and the obtained data processing results are also used to determine the values corresponding to the group (i.e., the subkey).
[0107] For example, with Figure 3 The subkey 1 in the table corresponds to the first group of selected business data. Figure 3 The subkey 2 in corresponds to the second group of selected business data. When performing data processing, for subkey 1, based on the preset data processing method, the attribute value in the first group of selected business data is processed to obtain the value of subkey 1; for subkey 2, based on the preset data processing method, the attribute value in the second group of selected business data is processed to obtain the value of subkey 2.
[0108] The above data processing method can be pre-set as needed, which can be one or more, for example, can include at least one of the following methods: summing, averaging, statistics of set attribute values, variance, standard deviation, and covariance. The set attribute value can be pre-set as needed, and the number can be one or more. The set attribute value can be selected from multiple attributes included in the initial business data.
[0109] In step S240, the attribute value in the selected business data may be the attribute value of one attribute or the attribute values of multiple attributes. For example, when the data processing method is summing, the attribute here may be one, that is, summing multiple attribute values of one attribute in the selected business data; when the data processing method is covariance finding, the attribute here may be two, that is, finding the covariance between two groups of attribute values corresponding to the two attributes.
[0110] In another embodiment of the present specification, when step S240 is executed, it may include the following steps 1b and 2b.
[0111] Step 1b: determine the second attribute group.
[0112] The second attribute group may include one or more attributes. The second attribute group may be selected from the attributes included in the initial business data. When determining the second attribute group, the execution subject may obtain a set second attribute group, or may determine the second attribute group according to input operations of relevant personnel.
[0113] The attributes included in the second attribute group may be different from the attributes included in the first attribute group. This step 1b may be performed simultaneously with step S220, and the execution order of the two may not be particular.
[0114] Step 2b: Based on a preset data processing method, data processing is performed on the attribute values of the second attribute group in the selected business data.
[0115] For example, for the initial business data shown in Table 1, the selected business data includes the 2nd and 5th rows of data. When processing the data, the attribute value corresponding to the second attribute group "whether there is overdue" in the selected business data can be processed to obtain the value value1 in the key-value pair data data1.
[0116] When the key key1 includes one or more sub-keys, for any sub-key k1 contained in the key key1, based on a preset data processing method, data processing is performed on the attribute value of the second attribute group in a group of selected business data corresponding to the sub-key k1, and the obtained data processing result is determined as the value v1 in the key-value pair data data1 corresponding to the sub-key k1.
[0117] In another embodiment of the present specification, in order to enrich the information in the value of the key-value pair data, the value value1 may include a field part and a field value part. The data processing method may include one or more. Each data processing method has a corresponding field in the value value1, for example, any first data processing method corresponds to the first field of the value value1. The data processing result of the first data processing method can be used as the field value corresponding to the first field.
[0118] When executing step S240, data processing may be performed on the attribute value in the selected business data based on the first data processing method, and the obtained data processing result may be determined as the field value of the first field.
[0119] When the key key1 includes one or more subkeys, for any subkey k1 contained in the key key1, based on the first data processing method, data processing is performed on the attribute value of the second attribute group in a group of selected business data corresponding to the subkey k1, and the obtained data processing result is determined as the field value of the first field in the value v1 corresponding to the subkey k1.
[0120] Specifically, when there are one or more attributes in the first attribute group and the specified attribute value of each attribute is one, that is, when the key key1 contains a subkey, the value corresponding to the subkey may also include multiple fields and corresponding field values.
[0121] For example, data processing method 1 is to count the proportion of the attribute value set to "yes" in the attribute "whether there is overdue", corresponding to the field "overdue rate" in value1; data processing method 2 is to count the proportion of the attribute value set to "no" in the attribute "whether there is overdue", corresponding to the field "non-overdue rate" in value1. See Table 4 below for details.
[0122] Table 4
[0123]
[0124] When there are multiple specified attribute values for at least one attribute in the first attribute group, that is, when the key key1 contains multiple subkeys, the value corresponding to each subkey may also include multiple fields and corresponding field values.
[0125] When the key key1 includes multiple sub-keys, data processing can be performed on a group of selected business data corresponding to each sub-key based on each data processing method, and the data processing result can be determined as the field value of the field corresponding to the data processing method in the value corresponding to the sub-key.
[0126] For example, taking Table 1 as part of the initial business data, Figure 3 As the subkey contained in the key key1, the two fields "overdue rate" and "non-overdue rate" in Table 4 are used as the fields of value1, and the two fields correspond to data processing methods 1 and 2. For subkey 1 and subkey 2, based on data processing method 1, the attribute values corresponding to the attribute "whether there is overdue" in the selected business data of the first group and the second group are processed respectively, and the overdue rates are 0.4% and 1.0% respectively; for subkey 1 and subkey 2, based on data processing method 2, the attribute values corresponding to the attribute "whether there is overdue" in the selected business data of the first group and the second group are processed respectively, and the overdue rates are 99.6% and 99.0% respectively. See Table 5 below for details.
[0127] Table 5
[0128]
[0129] Among them, the key1 part in Table 5 can be called a group, and each subkey can be called a group. Table 5 shows the situation of multiple groups. The first group (i.e., the third row of Table 5) indicates that among multiple initial business data (such as Table 1), the initial business data that meets the conditions of "the city is Beijing, the purpose of the loan is 1, and the loan amount is 5k" has the specific data situation of the attribute "whether there is overdue". It can be seen that the key-value pair data extracts specific deeper features from the initial business data, which has more important and far-reaching significance for the subsequent analysis and processing of the data, and improves the expressiveness of the data.
[0130] In this embodiment, the value of the key-value pair data includes a field part and a field value part, and the value may no longer contain only one data, but may include more abundant value values. In this way, the expression ability of the key-value pair data can be further improved, making the data cost lower.
[0131] The following reviews steps S210 to S240, wherein step S210 can obtain multiple initial business data according to a preset period, and for the multiple initial business data obtained in each preset period, steps S230 and S240 are executed to obtain key-value pair data corresponding to the preset period.
[0132] For different preset periods, the attribute groups (including the first attribute group and the second attribute group) in step S230 and step S240 may be different, and the specific forms of the key-value pair data obtained in this way are also different. The specific situation can also be flexibly set according to business needs. In this way, a large amount of "multi-dimensional" key-value pair data with multiple groups, multiple periods, multiple keys, and multiple values can be obtained.
[0133] After obtaining the key-value pair data, the key-value pair data may also be stored in a database.
[0134] In the case of the above-mentioned "multidimensional" key-value pair data, the data itself contains rich information. When data retrieval is needed, several data engines can be used to retrieve data from the database, and specifically one or more data engines can be used. When at least two data engines are used, the data retrieved by the at least two data engines can be different, which can make the retrieval results richer. For example, one or more of the data engines openTSDB, promQL, and SQL can be used for data retrieval. Among them, openTSDB can be used for retrieval in the form of black and white list regular expressions, promQL can provide flexible post-calculation query methods, and can directly perform post-calculation processing such as summation and difference on the initial retrieval results, and SQL realizes universal query between data. The above data engines also provide different programming languages for selection.
[0135] In each of the above embodiments, there may be multiple first attribute groups, and the association relationship between the attributes included in the multiple first attribute groups may be determined, and the association relationship is used to identify the relationship between different key-value pairs of data. For example, the first attribute group 1 includes the attribute "city", and the first attribute group 2 includes the attribute "province". An association relationship between "city" and "province" may be established, and the association relationship may be recorded in an association relationship table. By establishing association relationships between different attributes, an association relationship between key-value pairs of data is also established.
[0136] The above association relationship can also be stored in a database together with the key-value pair data. When searching for data in the database, the relevant key-value pair data can be retrieved based on the association relationship. This improves the scalability of the key-value pair data and also improves the retrieval efficiency.
[0137] In this specification, the “first” in the first attribute group and the first data processing method, and the corresponding “second” in the text are only for the convenience of distinction and description and do not have any limiting meaning.
[0138] The foregoing describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0139] Figure 4 The schematic block diagram of a device for generating multi-dimensional key-value pair data provided in the embodiment. The device 400 can be deployed in any device, equipment, platform, device cluster, etc. with computing and processing capabilities. Figure 2 The device 400 includes:
[0140] The acquisition module 410 is configured to acquire a plurality of initial business data, wherein one initial business data includes attribute values of a plurality of attributes;
[0141] The determination module 420 is configured to determine the first attribute group and the specified attribute values of the attributes included therein, and the key in the key-value pair data to be generated determined based on the first attribute group and the corresponding specified attribute values;
[0142] A screening module 430 is configured to screen the plurality of initial business data based on the first attribute group and the corresponding designated attribute value to obtain selected business data;
[0143] The processing module 440 is configured to perform data processing on the attribute value in the selected business data based on a preset data processing method, and determine the value in the key-value pair data based on the data processing result.
[0144] In one implementation, the key includes a field portion and a field value portion, wherein a group of field values corresponds to a subkey; the determination module 420 is further configured to determine the key by performing the following operations:
[0145] Determine the attributes in the first attribute group as fields included in the key;
[0146] Based on the specified attribute values of the attributes in the first attribute group, one or more groups of field values included in the key are determined and correspondingly used as one or more subkeys of the key.
[0147] In one implementation, the first attribute group includes multiple attributes; the specified attribute value of any attribute in the first attribute group is one or more;
[0148] The determining module 420, when determining the attributes in the first attribute group as fields included in the key, includes:
[0149] Determine the multiple attributes in the first attribute group as multiple fields included in the key respectively;
[0150] The determining module 420, when determining one or more groups of field values included in the key based on the specified attribute value of the attribute in the first attribute group, includes:
[0151] For multiple attributes in the first attribute group, when the specified attribute value of each of the multiple attributes is one, the specified attribute values of the multiple attributes are respectively determined as corresponding field values of the fields included in the key to obtain a group of field values;
[0152] For multiple attributes in the first attribute group, when there are multiple specified attribute values for at least one of the multiple attributes, the specified attribute values of the multiple attributes are combined to obtain multiple attribute value combinations; the multiple attribute value combinations are respectively determined as corresponding field values of the fields contained in the key to obtain multiple groups of field values.
[0153] In one implementation, the screening module 430 is specifically configured as follows:
[0154] For any subkey contained in the key, the plurality of initial business data are screened using the attribute corresponding to the subkey and the specified attribute value to obtain a group of selected business data corresponding to the subkey.
[0155] In one implementation, the processing module 440 is specifically configured as follows:
[0156] For any subkey contained in the key, based on a preset data processing method, data processing is performed on the attribute values in a set of selected business data corresponding to the subkey, and the obtained data processing result is determined as the value in the key-value pair data corresponding to the subkey.
[0157] In one implementation, the processing module 440 is specifically configured as follows:
[0158] determining a second attribute group;
[0159] Based on a preset data processing method, data processing is performed on the attribute values of the second attribute group in the selected business data.
[0160] In one embodiment, the value includes a field portion and a field value portion; the data processing method includes one or more; any first data processing method corresponds to the first field of the value;
[0161] The processing module 440 is specifically configured as follows:
[0162] Based on the first data processing method, data processing is performed on the attribute value in the selected business data, and the obtained data processing result is determined as the field value of the first field.
[0163] In one embodiment, the data processing method includes at least one of the following methods: summing, averaging, performing statistics on set attribute values, calculating variance, calculating standard deviation, and calculating covariance.
[0164] In one implementation, the first attribute group is multiple; the device 400 further includes:
[0165] The association module (not shown in the figure) is configured to determine the association relationship between the attributes included in the multiple first attribute groups; wherein the association relationship is used to identify the relationship between different key-value pair data.
[0166] In one embodiment, the device 400 further includes:
[0167] A storage module (not shown in the figure), configured to store the key-value pair data in a database after obtaining the key-value pair data;
[0168] The retrieval module (not shown in the figure) is configured to use several data engines to retrieve data from the database when data retrieval is required.
[0169] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.
[0170] The present specification also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute Figures 1 to 3 Any of the methods described above.
[0171] The embodiment of the present specification also provides a computing device, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, Figures 1 to 3 Any of the methods described above.
[0172] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0173] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.
[0174] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.
Claims
1. A method for generating multidimensional key-value pair data, comprising: Acquire multiple initial business data, where one initial business data contains attribute values of several attributes; Determine a first attribute group and designated attribute values of the attributes included therein, and determine a key in key-value pair data to be generated based on the first attribute group and the corresponding designated attribute values; The key includes the first attribute group and the corresponding specified attribute value; Based on the first attribute group and the corresponding designated attribute value, the plurality of initial business data are screened to obtain selected business data; Based on a preset data processing method, data processing is performed on the attribute values in the selected business data, and the values in the key-value pair data are determined based on the data processing results.
2. The method according to claim 1, wherein the key comprises a field portion and a field value portion, wherein: A set of field values corresponds to a subkey; the key is determined as follows: Determine the attributes in the first attribute group as fields included in the key; Based on the specified attribute values of the attributes in the first attribute group, one or more groups of field values included in the key are determined and correspondingly used as one or more subkeys of the key.
3. The method according to claim 2, wherein the first attribute group includes multiple attributes; and the specified attribute value of any attribute in the first attribute group is one or more; The step of determining the attributes in the first attribute group as fields included in the key comprises: Determine the multiple attributes in the first attribute group as multiple fields included in the key respectively; The step of determining one or more groups of field values contained in the key based on the specified attribute values of the attributes in the first attribute group comprises: For multiple attributes in the first attribute group, when the specified attribute value of each of the multiple attributes is one, the specified attribute values of the multiple attributes are respectively determined as corresponding field values of the fields included in the key to obtain a group of field values; For multiple attributes in the first attribute group, when there are multiple specified attribute values for at least one of the multiple attributes, the specified attribute values of the multiple attributes are combined to obtain multiple attribute value combinations; the multiple attribute value combinations are respectively determined as corresponding field values of the fields contained in the key to obtain multiple groups of field values.
4. The method according to claim 2, wherein the step of screening the plurality of initial business data based on the first attribute group and the corresponding specified attribute value comprises: For any subkey contained in the key, the plurality of initial business data are screened using the attribute corresponding to the subkey and the specified attribute value to obtain a group of selected business data corresponding to the subkey.
5. According to the method of claim 4, the step of performing data processing on the attribute value in the selected business data based on a preset data processing method, and determining the value in the key-value pair data based on the data processing result comprises: For any subkey contained in the key, based on a preset data processing method, data processing is performed on the attribute values in a set of selected business data corresponding to the subkey, and the obtained data processing result is determined as the value in the key-value pair data corresponding to the subkey.
6. The method according to claim 1, wherein the step of performing data processing on the attribute value in the selected business data comprises: determining a second attribute group; Based on a preset data processing method, data processing is performed on the attribute values of the second attribute group in the selected business data.
7. The method according to claim 1, wherein the value comprises a field portion and a field value portion; the data processing method comprises one or more; any first data processing method corresponds to a first field of the value; The step of performing data processing on the attribute value in the selected business data based on a preset data processing method, and determining the value in the key-value pair data based on the data processing result includes: Based on the first data processing method, data processing is performed on the attribute value in the selected business data, and the obtained data processing result is determined as the field value of the first field.
8. The method according to claim 1, wherein the data processing method comprises at least one of the following methods: Sum, find the average, perform statistics on the set attribute values, find the variance, find the standard deviation, and find the covariance.
9. The method according to claim 1, wherein the first attribute group is multiple; the method further comprises: Determine the association relationship between the attributes included in the plurality of first attribute groups; wherein the association relationship is used to identify the relationship between different key-value pair data.
10. The method according to claim 1, after obtaining the key-value pair data, further comprises: Storing the key-value pair data in a database; When data retrieval is required, several data engines are used to retrieve data from the database.
11. A device for generating multidimensional key-value pair data, comprising: The acquisition module is configured to acquire a plurality of initial business data, wherein one initial business data includes attribute values of a plurality of attributes; a determination module configured to determine a first attribute group and designated attribute values of the attributes included therein, and a key in key-value pair data to be generated determined based on the first attribute group and the corresponding designated attribute values; the key includes the first attribute group and the corresponding designated attribute value; A screening module configured to screen the plurality of initial business data based on the first attribute group and the corresponding designated attribute value to obtain selected business data; The processing module is configured to perform data processing on the attribute value in the selected business data based on a preset data processing method, and determine the value in the key-value pair data based on the data processing result.
12. The apparatus according to claim 11, wherein the key comprises a field portion and a field value portion, wherein: A set of field values corresponds to a subkey; the determination module is further configured to determine the key by the following operations: Determine the attributes in the first attribute group as fields included in the key; Based on the specified attribute values of the attributes in the first attribute group, one or more groups of field values included in the key are determined and correspondingly used as one or more subkeys of the key.
13. The device according to claim 12, wherein the first attribute group includes a plurality of attributes; and the specified attribute value of any one attribute in the first attribute group is one or more; The determining module, when determining the attributes in the first attribute group as the fields included in the key, includes: Determine the multiple attributes in the first attribute group as multiple fields included in the key respectively; The determining module, when determining one or more groups of field values included in the key based on the specified attribute value of the attribute in the first attribute group, includes: For multiple attributes in the first attribute group, when the specified attribute value of each of the multiple attributes is one, the specified attribute values of the multiple attributes are respectively determined as corresponding field values of the fields included in the key to obtain a group of field values; For multiple attributes in the first attribute group, when there are multiple specified attribute values for at least one of the multiple attributes, the specified attribute values of the multiple attributes are combined to obtain multiple attribute value combinations; the multiple attribute value combinations are respectively determined as corresponding field values of the fields contained in the key to obtain multiple groups of field values.
14. The device according to claim 12, wherein the screening module is specifically configured as follows: For any subkey contained in the key, the plurality of initial business data are screened using the attribute corresponding to the subkey and the specified attribute value to obtain a group of selected business data corresponding to the subkey.
15. The device according to claim 14, wherein the processing module is specifically configured as follows: For any subkey contained in the key, based on a preset data processing method, data processing is performed on the attribute values in a set of selected business data corresponding to the subkey, and the obtained data processing result is determined as the value in the key-value pair data corresponding to the subkey.
16. The device according to claim 11, wherein the processing module is specifically configured as follows: determining a second attribute group; Based on a preset data processing method, data processing is performed on the attribute values of the second attribute group in the selected business data.
17. The device according to claim 11, wherein the value comprises a field portion and a field value portion; the data processing mode comprises one or more; any first data processing mode corresponds to a first field of the value; The processing module is specifically configured as follows: Based on the first data processing method, data processing is performed on the attribute value in the selected business data, and the obtained data processing result is determined as the field value of the first field.
18. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 10.
19. A computing device, comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 10 is implemented.
Citation Information
Patent Citations
Data processing method and device, computer equipment and storage medium
CN111027295A
