Data processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a data processing method, device, electronic device and storage medium, which relate to the field of computer technology, and in particular to the field of data processing. The specific implementation scheme is: obtaining a continuous feature sequence to be discretized from a target memory, wherein the continuous feature sequence includes multiple feature values; determining the positive sample ratio corresponding to the feature value; using a target processor, according to the positive sample ratio, dividing the continuous feature sequence into multiple target intervals, wherein the positive sample ratio corresponding to the feature value in the target interval is monotonic; based on the multiple target intervals, determining the discrete coding of the continuous feature sequence. The present disclosure discretizes the data according to the monotonicity of the positive sample ratio, thereby improving the discrimination of the features while solving the lack of monotonicity, and effectively improving the accuracy of data discretization without increasing the consumption of hardware resources.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of data processing technology, and more particularly to a data processing method, device, electronic device, and storage medium. Background Art

[0002] In the field of data processing, the data that needs to be processed typically includes both continuous and discrete variables. In some scenarios, continuous variables such as age and amount of money require binning (i.e., discretization) to enable data mining and analysis using the discrete codes corresponding to the continuous variables. Common binning methods include equal frequency binning, equal interval binning, and distribution binning. Summary of the Invention

[0003] The present disclosure provides a data processing method, device, electronic device and storage medium, which improve the accuracy of data discretization.

[0004] According to one aspect of the present disclosure, a data processing method is provided, the method comprising: obtaining a continuous feature sequence to be discretized from a target memory, wherein the continuous feature sequence includes multiple feature values; determining the proportion of positive samples corresponding to the feature values; using a target processor, dividing the continuous feature sequence into multiple target intervals according to the positive sample proportions, wherein the proportion of positive samples corresponding to the feature values ​​in the target intervals is monotonic; and determining a discretized encoding of the continuous feature sequence based on the multiple target intervals.

[0005] According to another aspect of the present disclosure, a data processing method is provided, which includes: obtaining a product rating feature sequence to be discretized from a target memory, wherein the product rating feature sequence includes multiple feature rating values; determining the proportion of positive samples corresponding to the feature rating values, wherein the positive samples are determined based on the product brand; using a target processor, dividing the product rating feature sequence into multiple target intervals according to the positive sample proportions, wherein the proportion of positive samples corresponding to the feature rating values ​​in the target intervals is monotonic; and determining the discrete encoding of the product rating feature sequence based on the multiple target intervals.

[0006] According to another aspect of the present disclosure, a data processing device is provided, which includes: a continuous feature sequence acquisition module, used to obtain a continuous feature sequence to be discretized from a target memory, wherein the continuous feature sequence includes multiple feature values; a first positive sample ratio determination module, used to determine the positive sample ratio corresponding to the feature value; a first division module, used to use a target processor to divide the continuous feature sequence into multiple target intervals according to the positive sample ratio, wherein the positive sample ratio corresponding to the feature value in the target interval is monotonic; a first discretization code determination module, used to determine the discretization code of the continuous feature sequence based on the multiple target intervals.

[0007] According to another aspect of the present disclosure, a data processing device is provided, which includes: a product rating feature sequence acquisition module, used to obtain a product rating feature sequence to be discretized from a target memory, wherein the product rating feature sequence includes multiple feature rating values; a second positive sample ratio determination module, used to determine the positive sample ratio corresponding to the feature rating value, wherein the positive sample is determined based on the product brand; a second division module, used to use a target processor to divide the product rating feature sequence into multiple target intervals according to the positive sample ratio, wherein the positive sample ratio corresponding to the feature rating value in the target interval is monotonic; a second discretization code determination module, used to determine the discretization code of the product rating feature sequence based on the multiple target intervals.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the aforementioned data processing method.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the aforementioned data processing method.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the aforementioned data processing method when executed by a processor.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0014] Figure 2 is a flow chart of an embodiment of a data processing method according to the present disclosure;

[0015] Figure 3 is a flow chart of another embodiment of a data processing method according to the present disclosure;

[0016] Figure 4 is a flow chart of another embodiment of a data processing method according to the present disclosure;

[0017] Figure 5 is a flowchart of a merge determination operation according to the present disclosure;

[0018] Figure 6a -b is a schematic diagram of merging target intervals according to the present disclosure;

[0019] Figure 7 is a flow chart of another embodiment of a data processing method according to the present disclosure;

[0020] Figure 8 is a structural diagram of an embodiment of a data processing device according to the present disclosure;

[0021] Figure 9 is a structural diagram of another embodiment of a data processing device according to the present disclosure;

[0022] Figure 10 It is a block diagram of an electronic device used to implement the data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] Figure 1 FIG. 1 shows an exemplary system architecture 100 to which an embodiment of the data processing method of the present disclosure may be applied. Figure 1As shown, system architecture 100 may include terminal 101, network 102, and server 103. Network 102 is used to provide a medium for a communication link between terminal 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0025] The terminal 101 can interact with the server 103 through the network 102. The terminal 101 includes a target memory, wherein the target memory stores a continuous feature sequence to be discretized; the server 103 obtains the continuous feature sequence to be discretized from the target memory of the terminal 101, and processes the continuous feature sequence to be discretized to obtain a discrete code of the continuous feature sequence, further uses the preset logistic regression model in the server to process the discrete code to obtain prediction information, and sends the prediction information to the display interface of the terminal 101 for the user to review.

[0026] It should be noted that the server 103 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 103 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0027] It should be understood that Figure 1 The numbers of terminals, networks, and servers in the embodiment are merely illustrative. Any number of terminals, networks, and servers may be used as required.

[0028] It should be noted that the data processing method provided in the embodiments of the present disclosure is generally executed by the server 103 , and accordingly, the device for data processing is generally set in the server 103 .

[0029] Figure 2 A schematic diagram of a data processing method provided by an embodiment of the present disclosure is shown. Figure 2 As shown, the method includes the following steps:

[0030] S201. Obtain a continuous feature sequence to be discretized from a target memory, wherein the continuous feature sequence includes multiple feature values.

[0031] It should be noted that the target memory can be a memory within the terminal device or a memory external to the terminal device; a feature sequence is a variable that characterizes the characteristics of object data (such as user data or product data), wherein a feature sequence whose corresponding feature values ​​are discrete values ​​is a discrete feature sequence; a feature sequence whose corresponding feature values ​​are continuous values ​​is a continuous feature sequence. In this embodiment, the execution subject obtains the continuous feature sequence to be discretized from the target memory and performs binning processing on the continuous feature sequence to convert the continuous feature sequence into a discrete feature sequence.

[0032] For users, a continuous feature sequence can be test scores, age, height, etc., and the continuous feature sequence includes multiple feature values. For example, the feature value corresponding to age can be 0-99, and the feature value corresponding to height can be 50 cm to 200 cm, etc.

[0033] In this embodiment, if a factory has 130 products, the feature values ​​of the product score (continuous feature sequence) are 45, 80, 88, 77, 68, 99, 60, 92, and 85, among which 20 products have a product score of 45 points; 10 products have a product score of 60 points; 10 products have a product score of 68 points; 20 products have a product score of 77 points; 10 products have a product score of 80 points; 20 products have a product score of 85 points; 20 products have a product score of 88 points; 20 products have a product score of 85 points; 20 products have a product score of 85 points; and 20 products have a product score of 85 points. The relationship between the product score and the number of products is shown in Table 1:

[0034] Table 1

[0035] Product Ratings 45 80 88 77 68 99 60 92 85 Product quantity 20 10 10 20 10 20 20 10 10

[0036] S202: Determine the proportion of positive samples corresponding to the feature values.

[0037] It should be noted that positive samples can be pre-set based on the business scenario. Typically, users will call the sample corresponding to the value of interest in the sample data a positive sample, and the sample corresponding to the other value a negative sample. For example, for the sample data of "Is it Brand A?", there will be two values: "Yes" = 1 and "No" = 0. If the user is interested in Brand A, the sample with a value of 1 (Brand A) can be used as a positive sample; correspondingly, the sample with a value of 0 (other brands) can be used as a negative sample.

[0038] In this embodiment, the execution entity first counts the total number of samples corresponding to each feature value and the number of positive samples under the feature value, and then calculates the ratio of the number of positive samples under the feature value to the total number of samples corresponding to the feature value, thereby obtaining the positive sample ratio corresponding to each feature value.

[0039] Continuing with the content shown in Table 1 of the previous example, assuming that the positive sample is a sample of brand A, first count the number of products under different brands for each product score, as shown in Table 2; secondly, calculate the ratio of the number of brand A under each product score to the total number of products under that product score, thereby obtaining the positive sample ratio corresponding to each product score value.

[0040] Table 2

[0041] Product Ratings 45 80 88 77 68 99 60 92 85 Brand A 3 1 2 5 4 1 3 6 5 Other brands 17 9 8 15 6 19 17 4 5

[0042] For example, the positive sample ratio for a product with a score of 45 points is 3 (the number of positive samples) / 20 (the total number of samples) = 0.15; the positive sample ratio for a product with a score of 80 points is 1 (the number of positive samples) / 10 (the total number of samples) = 0.1; the positive sample ratio for a product with a score of 88 points is 2 (the number of positive samples) / 10 (the total number of samples) = 0.2, thus obtaining the positive sample ratio corresponding to each product score value shown in Table 3.

[0043] Table 3

[0044] Product Ratings 45 80 88 77 68 99 60 92 85 Positive sample ratio 0.15 0.4 0.15 0.25 0.2 0.5 0.1 0.6 0.5

[0045] S203 , using a target processor to divide the continuous feature sequence into multiple target intervals according to the positive sample ratio, wherein the positive sample ratio corresponding to the feature value in the target interval is monotonic.

[0046] In this embodiment, the server includes a target processor that can divide the continuous feature sequence into multiple target intervals based on the positive sample ratio of each feature value determined in step 202, and ensure that the positive sample ratio corresponding to the feature value within the target interval is monotonic. For example, using the aforementioned product rating example, based on the positive sample ratio corresponding to each product rating value determined in Table 3, the continuous feature sequence of product ratings from 45 to 99 can be divided into the following target intervals: 45-60, 68-80, 85-92, and 99, and the positive sample ratio corresponding to the product rating within each target interval is monotonic.

[0047] For example, in the target range of 68 to 80, the proportion of positive samples with a product score of 68 is 0.2, the proportion of positive samples with a product score of 77 is 0.25, and the proportion of positive samples with a product score of 80 is 0.4. That is, the proportion of positive samples corresponding to product scores in the target range of 68 to 80 increases monotonically. Based on similar reasons, the proportion of positive samples corresponding to product scores in the target range of 45 to 60 decreases monotonically. The proportion of positive samples corresponding to product scores in the target range of 85 to 92 increases monotonically.

[0048] S204: Determine the discretization code of the continuous feature sequence based on the multiple target intervals.

[0049] In this embodiment, the execution entity maps the feature values ​​of the continuous feature sequence to each target interval, thereby obtaining the discretized code of the specific feature value based on the discretized code corresponding to each target interval. Continuing with the aforementioned product rating example, if the target intervals of 45-60, 68-80, 85-92, and 99 correspond to the discretized codes 1, 2, 3, and 4, respectively, then the discretized code for a product rating of 45 is 1, and the discretized code for a product rating of 77 is 2.

[0050] In some optional implementations of this embodiment, after determining the discretization code of the continuous feature sequence, the execution entity may further perform the following steps: using a preset logistic regression model to process the discretization code to obtain prediction information corresponding to the continuous feature sequence.

[0051] In this embodiment, the above steps S201-S204 can be used in the application field of the algorithm model. The algorithm model is, for example, a logistic regression model. For example, the above continuous feature sequence can be user data or product data. In the scenario of predicting relevant information of a user or product based on the logistic regression model, multiple target intervals or discretized intervals of a continuous feature sequence of the user or product can be determined based on the above steps S201-S204, and then based on the value of the continuous feature sequence and the multiple target intervals, the discretized coding of the continuous feature sequence is determined, so that the discretized coding of the user or product is used as the input information of the logistic regression model to obtain the prediction information of the continuous feature sequence output by the logistic regression model. Among them, the continuous feature sequence is, for example, the user's age, income amount, consumption amount, etc., or the number of products sold, the number of returns, etc. The prediction information is, for example, the user's consumption level, the product's service life, etc.

[0052] In this embodiment, by discretizing the data based on the monotonicity of the positive sample ratio, we address the lack of monotonicity while improving the distinguishability of features. This effectively improves the accuracy of data discretization without increasing hardware resource consumption. Furthermore, using a logistic regression model to discretize the continuous feature sequence can improve the accuracy of the prediction information.

[0053] Continue to refer Figure 3 , which shows a schematic diagram of a data processing method provided by another embodiment of the present disclosure, the method comprising the following steps:

[0054] S301. Obtain a continuous feature sequence to be discretized from a target memory, wherein the continuous feature sequence includes multiple feature values.

[0055] In this embodiment, the description of S301 refers to S201 and will not be described in detail herein.

[0056] S302: Determine the proportion of positive samples corresponding to the feature values.

[0057] In this embodiment, the description of S302 refers to S202 and will not be described in detail herein.

[0058] S303 , sorting the corresponding positive sample proportions based on the arrangement order of the feature values ​​in the continuous feature sequence.

[0059] In this embodiment, multiple feature values ​​of a continuous feature sequence are sorted in ascending order or descending order, and the positive sample ratios corresponding to the feature values ​​are sorted according to the order in which the feature values ​​are arranged. For example, using the above-mentioned product rating example, multiple feature values ​​of the product rating are sorted in ascending order to obtain 45, 60, 68, 77, 80, 85, 88, 92, and 99. Then, the positive sample ratios are corresponding to the sorted feature values ​​one by one, thereby achieving the sorting of the positive sample ratios. Table 4 shows the sorted feature values ​​and the corresponding positive sample ratios.

[0060] Table 4

[0061] Product Ratings 45 60 68 77 80 85 88 92 99 Positive sample ratio 0.15 0.1 0.2 0.25 0.4 0.05 0.15 0.6 0.5

[0062] S304: Use the collision pointer to sequentially traverse the sorted positive sample ratios, and determine multiple feature values ​​corresponding to multiple positive sample ratios that change monotonically in value as a target interval.

[0063] In this embodiment, the collision pointer includes a left pointer and a right pointer. First, the traversal starts from the first feature value, that is, the left pointer and the right pointer point to the first feature value at the same time. At this time, the positive sample ratio corresponding to the current feature value is recorded as ration R1; secondly, the right pointer points to the second feature value, and the positive sample ratio corresponding to the second feature value is recorded as ration R2. If ration R1 is less than or equal to ration R2, the flag is recorded as 1; if ration R1 is greater than ration R2, the flag is recorded as 0; then, the right pointer points to the third value, and the positive sample ratio corresponding to the third feature value is recorded as ration R3. Similarly, if ration R2 is less than or equal to ration R3, the flag is recorded as 1; if ration R2 is greater than rationR3, the flag is recorded as 0, and the flag value is judged whether it occurs. If it changes, it indicates that the proportion of positive samples corresponding to the third feature value does not conform to monotonic change, then the feature value corresponding to the left pointer and the previous feature value corresponding to the right pointer are determined as a target interval, that is, the first feature value and the second feature value are determined as a target interval; if there is no change, the right pointer points to the fourth value, and the same judgment is performed until the collision pointer traverses to the positive sample ratio corresponding to the last feature value, and the judgment ends.

[0064] Taking the sorted feature values ​​and the corresponding positive sample ratios in Table 4 as an example, the left pointer and the right pointer both point to the first feature value 45, corresponding to 0.15, and the right pointer points to the second feature value 60, corresponding to a positive sample ratio of 0.1, and the flag is 0; the right pointer points to the third feature value 68, corresponding to a positive sample ratio of 0.2, and the flag is 1. The flag changes, and the first feature value and the second feature value are determined as a target interval, that is, the target interval 45 to 60.

[0065] S305 : In response to the traversal being completed, a plurality of target intervals are obtained.

[0066] In this embodiment, using the method of step 304, the execution entity begins judging the positive sample ratio of the first feature value and continues judging the positive sample ratio of the last feature value. Multiple feature values ​​corresponding to multiple positive sample ratios that change monotonically in value are then determined as a target interval, ultimately resulting in multiple target intervals. Using the aforementioned product rating example, the target intervals that can be obtained are 45-60, 68-80, 85-92, and 99.

[0067] In this embodiment, the collision pointer is used to traverse the sorted positive sample ratios, which can improve the efficiency of data discretization processing and reduce labor costs.

[0068] S306: Determine the discretization code of the continuous feature sequence based on the multiple target intervals.

[0069] In this embodiment, the description of S306 refers to S204 and will not be described in detail herein.

[0070] S307: Process the discretized code using a preset logistic regression model to obtain prediction information corresponding to the continuous feature sequence.

[0071] In this embodiment, the description of S307 refers to the above embodiment, and this application will not describe it in detail.

[0072] Continue to refer Figure 4 , which shows a schematic diagram of a data processing method provided by another embodiment of the present disclosure, the method comprising the following steps:

[0073] S401. Obtain a continuous feature sequence to be discretized from a target memory, wherein the continuous feature sequence includes multiple feature values.

[0074] S402: Determine the proportion of positive samples corresponding to the feature values.

[0075] In this embodiment, the description of S401-S402 refers to S201-S202, and this application will not describe them in detail here.

[0076] S403: Using a target processor, divide the continuous feature sequence into multiple target intervals according to the positive sample ratio, wherein the positive sample ratio corresponding to the feature value in the target interval is monotonic.

[0077] In this embodiment, the description of S403 may refer to S203, and in some optional implementations may refer to S303-S305, which will not be described in detail in this application.

[0078] S404: Calculate the information value of the target interval.

[0079] In this embodiment, information value is a numerical value that characterizes predictive power, which can also be called information quantity. In practical applications, information value can be used to measure the predictive power of each variable grouping obtained after variable binning, such as the target interval mentioned above.

[0080] For example, for a target interval, its information value can be calculated based on the WOE (Weight of Evidence) of the target interval, where the WOE represents the difference between the ratio of positive and negative samples in the target interval and the ratio of positive and negative samples in the total sample volume.

[0081] In this embodiment, k represents a target interval, and the information value of the target interval k can be determined by referring to the following formula:

[0082]

[0083] Among them, py k It is the ratio between the number of positive samples in the target interval and the number of positive samples in all sample data; pn k It is the ratio between the number of negative samples in this target interval and the number of all negative samples in all sample data; #y k is the number of positive samples in this target interval; #n k is the number of negative samples in this target interval; #y T is the number of positive samples in all sample data; #n T is the number of negative samples in all sample data.

[0084] Using the product rating example above, we obtain multiple target intervals: 45-60, 68-80, 85-92, and 99. The number of positive samples (products of Brand A) in the interval 45-60 is 4; the number of negative samples in the interval 45-60 is 26; the number of positive samples for all products is 30; and the number of negative samples for all products is 100. Based on the above formula, the information value of the target interval 45-60 is:

[0085]

[0086] Furthermore, referring to Table 5, based on a similar calculation method, the information value of each target interval obtained by discretizing the product ratings can be obtained.

[0087] Table 5

[0088] Product Ratings Number of positive samples Number of negative samples Information Value 45～60 4 26 0.08459 68～80 11 29 0.01798 85～92 10 40 0.01215 99 5 5 0.14046 Summary 30 100 —

[0089] S405 . Merge at least two target intervals that meet the merging conditions according to the information value of each target interval to obtain multiple binned intervals.

[0090] In this embodiment, the execution entity may treat each two adjacent target intervals as a group, thereby obtaining multiple groups of adjacent target intervals. Furthermore, based on the information value of each target interval, it is determined whether each group of adjacent target intervals meets the merging conditions. If so, the corresponding two adjacent target intervals are merged to obtain multiple binned intervals.

[0091] In some optional implementations of this embodiment, assuming that the number of target intervals is N, starting from the first target interval, the merging judgment operation is performed until the Nth target interval is judged, and multiple binning intervals are obtained, where N is a natural number greater than 1, such as Figure 5 As shown, the merge judgment operation includes the following steps:

[0092] S4051. Determine whether the i-th target interval and the i+1-th target interval meet the merging condition.

[0093] In this embodiment, the execution entity first calculates the information value of the interval after the i-th target interval and the i+1-th target interval are merged, and then determines whether the information value is greater than the preset information value threshold; if greater, it is determined that the i-th target interval and the i+1-th target interval meet the merging conditions.

[0094] In some optional implementations of this embodiment, the sum of the information value of the i-th target interval and the information value of the i+1-th target interval is determined as the information value before the merger; the information value of the interval obtained by merging the i-th target interval and the i+1-th target interval is determined as the information value after the merger; in response to the information value after the merger being greater than or equal to the information value before the merger, it is determined that the i-th target interval and the i+1-th target interval meet the merger condition.

[0095] Continuing with the product rating example above, when i = 1, taking the first target interval of 45-60 and the second target interval of 68-80 as an example, determine whether these two intervals meet the merging condition. The information value before merging is the sum of the information value of the first target interval of 45-60 and the information value of the second target interval of 68-80, that is, 0.08459 + 0.01798 = 0.12057. The information value after merging is the information value of the interval 45-80 after merging the first target interval 45-60 and the second target interval 68-80. According to the above method for calculating information value, the information value of the interval 45-80 is 0.00477. Obviously, the information value after merging 0.00477 is less than the information value before merging 0.12057. Therefore, the first target interval 45-60 and the second target interval 60-80 do not meet the merging condition.

[0096] S4052. In response to the i-th target interval and the i+1-th target interval satisfying the merging condition, the i-th target interval and the i+1-th target interval are merged to obtain a binning interval, and it is again determined whether the i+2-th target interval and the i+3-th target interval satisfy the merging condition.

[0097] In this embodiment, if Figure 6a As shown, the example of first judging whether target interval 1 and target interval 2 meet the merging condition is used for explanation. When target interval 1 and target interval 2 meet the merging condition, target interval 1 and target interval 2 are merged to obtain bin interval 1; and then judging whether target interval 3 and target interval 4 meet the merging condition again. Similarly, if they meet the condition, target interval 3 and target interval 4 are merged to obtain bin interval 2, and so on, until the last target interval, that is, target interval N, is judged.

[0098] S4053. In response to the i-th target interval and the i+1-th target interval not satisfying the merging condition, the i-th target interval is used as a binning interval, and it is again determined whether the i+1-th target interval and the i+2-th target interval satisfy the merging condition.

[0099] In this embodiment, if Figure 6b As shown, the example of first judging whether target interval 1 and target interval 2 meet the merging condition is used as an example to illustrate. When target interval 1 and target interval 2 do not meet the merging condition, target interval 1 is used as bin interval 1; then judge whether target interval 2 and target interval 3 meet the merging condition. If so, target interval 2 and target interval 3 are merged to obtain bin interval 2. If not, judge whether target interval 3 and target interval 4 meet the merging condition, and so on, until the last target interval, that is, target interval N, is judged.

[0100] It should be noted that in this embodiment, i takes values ​​in the range {1, 2, ..., N-3} from smallest to largest, and can take values ​​continuously or intermittently as needed. For example, when i takes a value of 1, in response to the first and second target intervals satisfying the merge condition, i takes a value of 3, and the execution entity determines whether the third and fourth target intervals satisfy the merge condition. In response to the first and second target intervals not satisfying the merge condition, i takes a value of 2, and the execution entity determines whether the second and third target intervals satisfy the merge condition.

[0101] In some optional implementations of this embodiment, the aforementioned merging and judging operation may be repeated on the obtained bin intervals. Figure 6a Or as shown in 6b, after obtaining M binning intervals such as binning interval 1 and binning interval 2 (M is a natural number greater than 1), the execution entity again performs the aforementioned merging judgment operation on these M binning intervals, for example, judging whether binning interval 1 and binning interval 2 meet the merging conditions, etc., until there are no intervals that can be merged, and the final multiple binning intervals are obtained. In other words, no two adjacent binning intervals meet the merging conditions.

[0102] In this embodiment, by merging intervals whose information value after merging is greater than that before merging, the information value is continuously improved. This maximizes the information value during the discretization of the continuous feature sequence, achieving optimal discretization. Furthermore, compared to complex analysis of continuous feature sequences based on manual experience, this significantly improves the efficiency of the discretization process and reduces labor costs.

[0103] S406: Calculate the weight of evidence for the binning intervals.

[0104] In this embodiment, the discretization code is determined by calculating the weight of evidence for each binning interval, i.e., the WOE value. It should be noted that the weight of evidence is obtained based on the number of target data in the multiple sample data, where the target data can be data that meets the preset conditions, i.e., positive samples. Let x represent a binning interval, and the weight of evidence for the binning interval x can be determined by referring to the following formula:

[0105]

[0106] Among them, py x It is the ratio between the number of positive samples in the bin interval and the number of positive samples in all sample data; pn x It is the ratio between the number of negative samples in the bin interval and the number of all negative samples in all sample data; #y x is the number of positive samples in this bin interval; #n x is the number of negative samples in this bin interval; #y T is the number of positive samples in all sample data; #n T is the number of negative samples in all sample data.

[0107] Continuing with the example of the aforementioned product rating, assuming that the final binning intervals are 45-60, 68-80, 85-92, and 99, and taking the binning interval of 45-60 as an example, using the above formula for calculating the weight of evidence, the corresponding weight of evidence is: ln[(4 / 30) / (26 / 100)]=-0.66782.

[0108] Furthermore, referring to Table 6, based on a similar calculation method, the weight of evidence for each bin interval can be obtained.

[0109] Table 6

[0110] Product Ratings Number of positive samples Number of negative samples Weight of evidence 45～60 4 26 -0.66782 68～80 11 29 0.23457 85～92 10 40 -0.18232 99 5 5 1.20397 Summary 30 100 —

[0111] S407. Based on the weight of evidence, obtain a discretized encoding of the continuous feature sequence.

[0112] In this embodiment, the evidence weight of the binning interval calculated in step S408 is used as the discretization code corresponding to the continuous feature sequence. Taking the aforementioned product rating as an example, referring to Table 6, the feature values ​​of the continuous feature sequence of the product rating can be mapped to each target interval, and based on the discretization code corresponding to each binning interval, the discretization code of the specific product is obtained. Among them, the discretization codes corresponding to the above binning intervals are -0.66782, 0.23457, -0.18232 and 1.20397 respectively. The discretization code of the product rating of 45 points is -0.66782, and the discretization code of the product rating of 77 points is 0.23457.

[0113] In this embodiment, since the weight of evidence is obtained based on the number of positive samples in the target interval, it can reflect the predictive ability of the interval. Therefore, using the weight of evidence as the discretized code corresponding to the interval can increase the amount of information carried by the discretized code when applying the discretized code to predict information, thereby improving the prediction accuracy.

[0114] S408: Process the discretized code using a preset logistic regression model to obtain prediction information corresponding to the continuous feature sequence.

[0115] In this example, the feature values ​​of the original continuous feature sequence are replaced by discretized codes using the evidence weights corresponding to the binning intervals. This discretized code is then fed into a logistic regression model to generate prediction information. Furthermore, because the discretization of the continuous feature sequence maximizes information value, using the logistic regression model to process the discretized code of the continuous feature sequence can improve the accuracy of the prediction information.

[0116] Continue to refer Figure 7 , which shows a process of another embodiment of the data processing method according to the present disclosure. The method includes the following steps:

[0117] S701. Obtain a product rating feature sequence to be discretized from a target memory, wherein the product rating feature sequence includes multiple feature rating values.

[0118] In this embodiment, Figure 1 The terminal shown in the figure may be a product evaluator, and the execution subject obtains a product rating feature sequence to be discretized from the product evaluator, wherein the product rating feature sequence includes multiple feature rating values. For example, the feature rating values ​​are 45, 80, 88, 77, 68, and 99.

[0119] S702: Determine the proportion of positive samples corresponding to the feature score value, wherein the positive samples are determined based on the product brand.

[0120] In this embodiment, it is assumed that the product of brand A is a positive sample, and vice versa, it is a negative sample. The description of S602 refers to S202, and this application will not elaborate on it here.

[0121] S703: Using a target processor, divide the product rating feature sequence into multiple target intervals according to the positive sample ratio, wherein the positive sample ratio corresponding to the feature score value within the target interval is monotonic.

[0122] In this embodiment, based on the positive sample ratio determined in step S602, the execution entity may divide the product rating feature sequence into multiple target intervals, and ensure that the positive sample ratio corresponding to the feature score values ​​within the target interval is monotonic. The description of S603 is similar to that of S203 or S303-S305, and will not be further described in this application.

[0123] S704: Determine the discretization coding of the product rating feature sequence based on multiple target intervals.

[0124] In this embodiment, the execution subject can determine the discretization coding of the product rating feature sequence based on multiple target intervals, for example, calculate the evidence weight of the target interval, and use the evidence weight of each target interval as the discretization coding of the product rating feature sequence. The description of S704 refers to S204 or S404-S407, and this application will not be described in detail here.

[0125] In this embodiment, after determining the discretization code of the product rating feature sequence, the execution entity may further perform the following steps:

[0126] The discretized coding is processed using a pre-trained product grade prediction model to obtain the product grade prediction value corresponding to the product rating feature sequence.

[0127] In this embodiment, the original feature score is replaced with the discretized code corresponding to the target interval. This discretized code is then used as input to the product grade prediction model, resulting in the output of the product grade prediction model, which is the product grade prediction value. The description of this step is referenced in the previous embodiment and will not be repeated in detail here.

[0128] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0129] As an implementation of the above methods, the embodiment of the present disclosure further provides a data processing device 800, such as Figure 8 As shown, the device includes:

[0130] The continuous feature sequence acquisition module 801 is used to acquire a continuous feature sequence to be discretized from a target memory, wherein the continuous feature sequence includes multiple feature values;

[0131] A first positive sample ratio determination module 802 is used to determine the positive sample ratio corresponding to the feature value;

[0132] The first partitioning module 803 is configured to use a target processor to partition the continuous feature sequence into a plurality of target intervals according to the positive sample ratio, wherein the positive sample ratio corresponding to the feature value in the target interval is monotonic;

[0133] The first discretization code determination module 804 is configured to determine the discretization code of the continuous feature sequence based on multiple target intervals.

[0134] In some optional implementations of this embodiment, the device further includes:

[0135] Information value calculation module, used to calculate the information value of the target interval;

[0136] A merging module is used to merge at least two target intervals that meet the merging conditions according to the information value of each target interval to obtain multiple bin intervals; wherein the discretization code determination module includes:

[0137] Evidence weight calculation unit, used to calculate the evidence weight of the binning interval;

[0138] The discretization coding determination unit is used to obtain the discretization coding of the continuous feature sequence based on the evidence weight.

[0139] In some optional implementations of this embodiment, the number of the multiple target intervals is N, and the merging module includes:

[0140] A merging and judging unit is configured to perform a merging and judging operation starting from the first target interval until the Nth target interval is judged, thereby obtaining multiple binning intervals, where N is a natural number greater than 1. The merging and judging unit includes:

[0141] The first subunit is used to determine whether the i-th target interval and the i+1-th target interval meet the merging condition;

[0142] a second subunit, configured to, in response to the i-th target interval and the i+1-th target interval satisfying a merging condition, merge the i-th target interval and the i+1-th target interval to obtain a single target interval, and again determine whether the i+2-th target interval and the i+3-th target interval satisfy the merging condition;

[0143] The third subunit is configured to, in response to the i-th target interval and the i+1-th target interval not satisfying the merging condition, take the i-th target interval as a target interval and again determine whether the i+1-th target interval and the i+2-th target interval satisfy the merging condition; wherein i is in the range {1, 2, ..., N-3} in ascending order.

[0144] In some optional implementations of this embodiment, the first subunit is configured as:

[0145] The sum of the information value of the i-th target interval and the information value of the i+1-th target interval is determined as the information value before the merger; the information value of the interval after the merger of the i-th target interval and the i+1-th target interval is determined as the information value after the merger; in response to the information value after the merger being greater than or equal to the information value before the merger, it is determined that the i-th target interval and the i+1-th target interval meet the merger condition.

[0146] In some optional implementations of this embodiment, the first division module includes:

[0147] The sorting unit is used to sort the corresponding positive sample proportions based on the arrangement order of the feature values ​​in the continuous feature sequence; the traversal unit is used to use the collision pointer to traverse the sorted positive sample proportions in sequence, and determine the multiple feature values ​​corresponding to the multiple positive sample proportions that change monotonically in value as a target interval; the target interval generation unit is used to obtain multiple target intervals in response to the end of traversal.

[0148] In some optional implementations of this embodiment, the device further includes:

[0149] The prediction module is used to process the discretized code using a preset logistic regression model to obtain prediction information corresponding to the continuous feature sequence.

[0150] Since a data processing device proposed in an embodiment of the present application corresponds to a data processing method provided in the above embodiment, the previous implementation manner and beneficial effects are also applicable to the data processing device provided in this embodiment and will not be described in detail in this embodiment.

[0151] Figure 9 A data processing device 900 provided in another embodiment of the present disclosure includes:

[0152] The product rating feature sequence acquisition module 901 is used to acquire the product rating feature sequence to be discretized from the target memory, wherein the product rating feature sequence includes multiple feature rating values;

[0153] A second positive sample ratio determination module 902 is used to determine the positive sample ratio corresponding to the feature score value, wherein the positive sample is determined based on the product brand;

[0154] The second partitioning module 903 is configured to use a target processor to partition the product rating feature sequence into a plurality of target intervals according to the positive sample ratio, wherein the positive sample ratio corresponding to the feature score value within the target interval is monotonic;

[0155] The second discretization code determination module 904 is used to determine the discretization code of the product rating feature sequence based on multiple target intervals.

[0156] In some optional implementations of this embodiment, the device further includes:

[0157] The product grade prediction module is used to process the discretized code using a pre-trained product grade prediction model to obtain the product grade prediction value corresponding to the product rating feature sequence.

[0158] Since a data processing device proposed in an embodiment of the present application corresponds to a data processing method provided in the above embodiment, the previous implementation manner and beneficial effects are also applicable to the data processing device provided in this embodiment and will not be described in detail in this embodiment.

[0159] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0160] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0161] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0162] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0163] The computing unit 1001 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a data processing method. For example, in some embodiments, the data processing method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the data processing method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 can be configured to perform the aforementioned data processing method in any other appropriate manner (e.g., by means of firmware).

[0164] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0166] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0168] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0169] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0170] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0171] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A data processing method, comprising: Obtaining a continuous feature sequence to be discretized from a target memory, wherein the continuous feature sequence is a variable representing a feature of object data, including multiple feature values; the object data is user data or product data; Determine the proportion of positive samples corresponding to the feature value; Using a target processor, sorting corresponding positive sample proportions based on an arrangement order of feature values ​​in the continuous feature sequence; using a collision pointer to sequentially traverse the sorted positive sample proportions, and determining multiple feature values ​​corresponding to multiple positive sample proportions that change monotonically in value as a target interval; in response to completion of the traversal, obtaining multiple target intervals, wherein the positive sample proportions corresponding to the feature values ​​within the target intervals are monotonic; Based on the multiple target intervals, a discretization code of the continuous feature sequence is determined.

2. The method according to claim 1, further comprising: calculating the information value of the target interval; According to the information value of each target interval, at least two target intervals that meet the merging conditions are merged to obtain multiple bin intervals; The step of determining the discretization code of the continuous feature sequence based on the multiple target intervals includes: Calculating the weight of evidence for the binning intervals; Based on the evidence weight, a discretized encoding of the continuous feature sequence is obtained.

3. The method according to claim 2, wherein: The number of the multiple target intervals is N, and the at least two target intervals that meet the merging condition are merged to obtain multiple binned intervals, including: Starting from the first target interval, a merging judgment operation is performed until the Nth target interval is judged, thereby obtaining multiple binning intervals, where N is a natural number greater than 1. The merging judgment operation includes: Determine whether the i-th target interval and the i+1-th target interval meet the merging conditions; In response to the i-th target interval and the i+1-th target interval meeting the merging condition, the i-th target interval and the i+1-th target interval are merged to obtain a binning interval, and it is again determined whether the i+2-th target interval and the i+3-th target interval meet the merging condition; In response to the i-th target interval and the i+1-th target interval not satisfying the merging condition, the i-th target interval is used as a binning interval, and it is again determined whether the i+1-th target interval and the i+2-th target interval satisfy the merging condition; Among them, i is in the order from small to large {1, 2, ..., N 3}.

4. The method according to claim 3, wherein: The determining whether the i-th target interval and the (i+1)-th target interval meet the merging condition includes: The sum of the information value of the i-th target interval and the information value of the i+1-th target interval is determined as the information value before merging; The information value of the interval obtained by merging the i-th target interval and the i+1-th target interval is determined as the merged information value; In response to the information value after merging being greater than or equal to the information value before merging, it is determined that the i-th target interval and the (i+1)-th target interval meet the merging condition.

5. The method according to any one of claims 1 to 4, further comprising: The discretized codes are processed using a preset logistic regression model to obtain prediction information corresponding to the continuous feature sequence.

6. The method according to claim 1, wherein The product data is product rating data, and the continuous feature sequence is a product rating feature sequence.

7. The method according to claim 6, further comprising: The discretized codes are processed using a pre-trained product grade prediction model to obtain a product grade prediction value corresponding to the product rating feature sequence.

8. A data processing device comprising: A continuous feature sequence acquisition module is used to acquire a continuous feature sequence to be discretized from a target memory, wherein the continuous feature sequence is a variable representing the characteristics of object data, including multiple feature values, and the object data is user data or product data; A first positive sample ratio determination module is used to determine the positive sample ratio corresponding to the feature value; A first partitioning module is configured to use a target processor to sort the corresponding positive sample ratios based on the arrangement order of the feature values ​​in the continuous feature sequence; use a collision pointer to sequentially traverse the sorted positive sample ratios, and determine multiple feature values ​​corresponding to multiple positive sample ratios that change monotonically in value as a target interval; and obtain multiple target intervals in response to the completion of the traversal, wherein the positive sample ratios corresponding to the feature values ​​in the target intervals are monotonic; The first discretization code determination module is configured to determine the discretization code of the continuous feature sequence based on the multiple target intervals.

9. The apparatus according to claim 8, further comprising: An information value calculation module, used to calculate the information value of the target interval; A merging module is used to merge at least two target intervals that meet the merging conditions according to the information value of each target interval to obtain multiple bin intervals; The discretization code determination module includes: An evidence weight calculation unit, used to calculate the evidence weight of the binning interval; A discretization code determination unit is used to obtain the discretization code of the continuous feature sequence based on the evidence weight.

10. The device according to claim 9, wherein The number of the multiple target intervals is N, and the merging module includes: A merging and judging unit is configured to perform a merging and judging operation starting from the first target interval until the Nth target interval is judged, thereby obtaining multiple binning intervals, where N is a natural number greater than 1. The merging and judging unit includes: The first subunit is used to determine whether the i-th target interval and the i+1-th target interval meet the merging condition; The second subunit is configured to, in response to the i-th target interval and the i+1-th target interval satisfying the merging condition, merge the i-th target interval and the i+1-th target interval to obtain a binning interval, and again determine whether the i+2-th target interval and the i+3-th target interval satisfy the merging condition; a third subunit, configured to, in response to the i-th target interval and the (i+1)-th target interval not satisfying the merging condition, use the i-th target interval as a binning interval and again determine whether the (i+1)-th target interval and the (i+2)-th target interval satisfy the merging condition; Among them, i is in the order from small to large {1, 2, ..., N 3}.

11. The device according to claim 10, wherein The first subunit is configured as follows: The sum of the information value of the i-th target interval and the information value of the i+1-th target interval is determined as the information value before merging; The information value of the interval obtained by merging the i-th target interval and the i+1-th target interval is determined as the merged information value; In response to the information value after merging being greater than or equal to the information value before merging, it is determined that the i-th target interval and the (i+1)-th target interval meet the merging condition.

12. The device according to any one of claims 8 to 11, further comprising: The prediction module is used to process the discretized code using a preset logistic regression model to obtain prediction information corresponding to the continuous feature sequence.

13. The device according to claim 8, wherein The product data is product rating data, and the continuous feature sequence is a product rating feature sequence.

14. The apparatus according to claim 13, further comprising: The product grade prediction module is used to process the discretized code using a pre-trained product grade prediction model to obtain a product grade prediction value corresponding to the product rating feature sequence.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the data processing method according to any one of claims 1 to 7.

17. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 7.

Images