Data distribution balance optimization regrouping method, electronic device and storage medium

Abstract

The present invention discloses a data distribution balance optimization regrouping method, an electronic device and a storage medium, which groups and sorts the databases to be optimized according to their common indicators, and regroups the databases to be optimized according to the sorting results to balance the distribution of each group of data. The evaluation score difference distribution set by the present invention makes the score difference in the concentrated area of ​​the data set larger and the score difference in the dispersed area of ​​the data smaller, so that the score can more accurately reflect the position of the sample.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a data distribution balance optimization regrouping method, an electronic device and a storage medium. Background Art

[0002] Nowadays, data information is developing rapidly. Since the circulation of electronic file data, it has become more convenient for researchers to conduct in-depth research on a certain type of technology, the development of a certain industry, and other aspects. When evaluating the development of various technologies or industries, it is often necessary to collect statistics on certain indicators and evaluate them based on the statistical results. Evaluation based on specific indicator values ​​can objectively and quantitatively reflect the actual situation and advantages and disadvantages of different objects, help improve evaluation efficiency, promote fair competition, and guide resource allocation. It is an important means to promote scientific and technological progress, enhance innovation capabilities and social management levels.

[0003] In the field of document evaluation technology, especially the evaluation of patent documents, it is helpful to select high-quality resources and improve academic standards. However, in the existing evaluation system, evaluation factors are mostly statistical values. Due to the uncertainty of the sample, it often appears in the evaluation system that the evaluation factors of the reviewed object are converted to obtain the final evaluation results. However, in the process of macro data processing, it is not possible to obtain accurate evaluation scores by simply calculating according to the curve relationship obtained by fitting and using a specific conversion formula. Especially for patent documents, the gap between the number of citations and the number of citations is very obvious, and the total amount of patent documents is also large. Since there are citation relationships among many evaluated objects, especially documents that have been cited many times and have made major technological breakthroughs, there are only a few documents after all, and the document base is large, and the number of citations of a few documents is extremely large; but the number of citations of most documents is extremely small. According to the form of direct conversion of ordinary statistical results, the statistical values ​​cannot be evenly distributed. Therefore, the converted score cannot accurately represent the influence of the evaluated object. Summary of the invention

[0004] In view of the above-mentioned literature with a huge number of samples and large differences in the degree of dispersion of indicators, the purpose of the present invention is to provide a data distribution balanced optimization regrouping method, electronic device and storage medium, so that the data can be distributed relatively more evenly during the evaluation process.

[0005] The method and technical solution include: a data distribution balanced optimization regrouping method, including:

[0006] Obtain the database to be optimized, and determine the initial grouping basis index of the database to be optimized;

[0007] Classify the database data to be optimized according to the initial grouping basis index, and obtain the data set within the cluster group and the data set outside the cluster group;

[0008] Get the index value of the index for the initial grouping of each data in the data set within the cluster group;

[0009] Performing initial grouping of the data set within the cluster group according to the index value, and determining the amount of data in the data group corresponding to each index value;

[0010] Sort each data group in ascending order according to the amount of data, and fit the sorting results into a first curve;

[0011] Obtaining the maximum slope value and the minimum slope value of the first curve, and obtaining grouping nodes that meet the expected number of regroupings according to the maximum slope value and the minimum slope value of the first curve;

[0012] Obtain the index value at each grouping node, and sort each grouping node from large to small / small to large according to the index value, and obtain the index value interval of adjacent grouping nodes;

[0013] The data sets within the clustering group are regrouped according to the index value interval, and the data sets outside the clustering group are combined as the first group or the last group to complete the regrouping of the database to be optimized.

[0014] Furthermore, the data to be optimized are patent documents, and the initial grouping is based on an indicator selected from the group consisting of the number of citations, the number of citations, the number of claims, and the number of families.

[0015] Furthermore, the data to be optimized are paper documents, and the initial grouping basis indicator is one of the number of authors and the impact value of the document.

[0016] Furthermore, the process of clustering the database to be optimized according to the indicator characteristics includes: when the indicator value of the indicator according to which the data in the database to be optimized is initially grouped is 0, it is recorded as a data set outside the cluster group, otherwise it is recorded as a data set within the cluster group.

[0017] Furthermore, when the total amount of data in the data set outside the cluster group is 0, the data set inside the cluster group is regrouped according to the index value interval and directly output as the regrouping result of the database to be optimized.

[0018] Furthermore, the process of obtaining grouping nodes that meet the expected number of regroupings according to the maximum slope value and the minimum slope value of the first curve includes:

[0019] Obtaining a reference tangent line: calculating a slope intermediate value according to a maximum slope value and a slope minimum value of the first curve, and fitting a reference tangent line of a slope corresponding to the first curve according to the slope intermediate value;

[0020] Get endpoint tangent: Get the straight line that is parallel to the reference tangent and intersects with the maximum value point of the first curve and is farthest from the reference tangent in the two-dimensional coordinate system of the first curve as the endpoint tangent;

[0021] Obtain a grouping node; insert a straight line parallel to the reference tangent at equal distances between the reference tangent and the endpoint tangent until the intersection of the inserted straight line and the first curve meets the expected number of regroupings.

[0022] An electronic device comprises: a memory storing computer program instructions; and a processor, which implements the steps of any one of the above methods when the computer program instructions are executed by the processor.

[0023] A computer-readable storage medium is used to store instructions. When the stored instructions are executed on a computer, the computer is caused to execute the steps of any one of the methods described above.

[0024] The evaluation score difference distribution set by the present invention makes the score difference in the concentrated area of ​​the data set larger and the score difference in the dispersed area of ​​the data smaller, so that the score can more accurately reflect the position of the sample. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of a method flow chart of an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of the sorted sample quantities under various indicator values ​​in one embodiment of the present invention;

[0027] Figure 3 A schematic diagram of the slope of a curve after fitting in one embodiment of the present invention;

[0028] Figure 4 A schematic diagram of a reference tangent line in one embodiment of the present invention;

[0029] Figure 5 A schematic diagram of obtaining grouped nodes through reference tangent lines and parallel lines in one embodiment of the present invention;

[0030] Figure 6 The figure is a schematic diagram of grouping nodes in one implementation of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0032] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0033] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0034] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0035] Reference Figure 1-6 The present invention discloses a data distribution balancing optimization regrouping method, and the disclosed scheme of the method includes: obtaining a database to be optimized, determining a primary grouping basis index of the database to be optimized; classifying the data of the database to be optimized according to the primary grouping basis index, obtaining a clustering group data set and a clustering group data set; obtaining an index value of the primary grouping basis index of each data in the clustering group data set; performing primary grouping of the data in the clustering group data set according to the index value, and determining the data volume of the data group corresponding to each index value; combining Figure 2 After obtaining the data volume of each data group corresponding to the indicator value, sort the data groups from small to large according to the data volume, referring to Figure 3 , fit the sorting results into the first curve; combine Figure 4 , obtain the maximum and minimum slope values ​​of the first curve, where the maximum slope is l 1 At the tangent line between the straight line and the curve, the minimum slope is l 2 The tangent line to the curve. According to the maximum slope and the minimum slope of the first curve, the grouping node that meets the expected number of regroupings is obtained; in this embodiment, combined with Figure 4-6 The process of obtaining the grouping nodes that meet the expected number of regroupings according to the maximum slope and the minimum slope of the first curve includes: obtaining the reference tangent: calculating the middle slope value according to the maximum slope and the minimum slope of the first curve, and fitting the reference tangent l of the slope corresponding to the first curve according to the middle slope value. 3，The reference tangent line satisfies the tangent to the first curve; obtaining the endpoint tangent line: the endpoint curve obtaining process includes obtaining a straight line which is parallel to the reference tangent line and intersects with the maximum value point of the first curve and is farthest from the reference tangent line in the two-dimensional coordinate system of the first curve as the endpoint tangent line l 4 ; Obtain grouping nodes; Insert straight lines parallel to the reference tangent at equal distances between the reference tangent and the endpoint tangent until the intersection of the inserted straight line and the first curve meets the expected number of regroupings. In this implementation step, the grouping node is the intersection of the endpoint tangent, the reference tangent, and the straight line parallel to the endpoint tangent on the first curve. Since there may be inconsistency between the number of intersections and the number of grouping nodes, it may happen that the expected number of regroupings is equal to the number of intersections or one less than the number of intersections. When the expected number of regroupings is less than the number of intersections, exclude the leftmost intersection and use the remaining intersections as grouping nodes. Reference Figure 6 , after obtaining the index value at each grouping node, sort each grouping node again according to the index value from large to small / small to large, and obtain the index value interval of the adjacent grouping node. Then regroup the data set in the cluster group according to the index value interval, and collect the data outside the cluster group as the first group or the last group to complete the regrouping of the database to be optimized. The present invention regroups and optimizes databases with extremely uneven data distribution and more index values, such as ranking each patent according to the number of citations and then estimating the distribution interval of the patent, because the distribution of patent citations is extremely uneven, and the number of citations ranges from 0 to several thousand. Because the difference in data volume distribution under each number of citations is too obvious, the data ratios with the most, more, general, and fewer citations and zero are very different. For example, the total number of samples selected as the database to be optimized this time is 97048593, the number of patent citations is distributed in 0-8706, the index value is the number of citations, and the data group is a collection of samples with the same number of citations. The relationship between the index value and each data group is as shown in the following table.

[0036] Index value (number of citations / times) Data volume 8706 1 8689 1 8506 2 8430 4 …… …… 403 403 402 449 401 514 …… …… 5 2398684 4 3029911 3 4067249 2 6040683 1 10809023 0 46371958

[0037] The data with citations of 0-5 times account for 75% of the total data volume, of which samples with citations of 0 account for 48%, and the total amount of other data accounts for only 25%. According to the ordinary evaluation method of sorting by citation count, due to the large base of patent samples with few citations and zero citations, patents with average citations and patents with high citations cannot be intuitively displayed on the chart, and the approximate distribution and quality of patents at each citation count cannot be effectively evaluated. Therefore, the present invention adopts a re-balanced grouping method for the database to be optimized with a total sample size of 97,048,593 and patent citation counts distributed between 0-8706. Samples with citations of 0 are merged and grouped as data outside the cluster group, and data with non-0 citations are grouped as reference. Figure 5By grouping by the slope of the curve, the side with more data is grouped more (the grouping points are denser relative to the X-axis), and the side with less data is grouped less (the grouping points are sparser relative to the X-axis), and the samples are re-grouped according to the grouping points to obtain the index value at each grouping node. In the present invention, after obtaining the index value at each grouping node, it is sorted from large to small / from small to large to obtain the index value interval of adjacent grouping nodes; then the data set of each index value interval is divided into a group, so that the grouping of the sample database is balanced as much as possible and can reflect the distribution position of each sample in the total sample under the evaluation index, and finally the out-of-cluster data with a reference number of 0 is inserted to achieve the optimized grouping of the entire sample database.

[0038] Based on one or more of the above embodiments, the data to be optimized in this application is patent documents, and the index for the initial grouping is one of the number of citations, the number of citations, the number of claims, and the number of families. In addition to being applicable to the case where the number of patent citations is used as an index value, this application is also applicable to data optimization in cases where the number of claims and the number of families have large differences in the number of various index values.

[0039] Based on one or more of the above embodiments, the data to be optimized is paper documents, and the initial grouping basis indicator is one of the number of authors and the impact value of the document.

[0040] On the basis of one or more of the above embodiments, for the case where the number of patent citations / citations is 0, the present application generally groups the patents with a number of citations / citations of 0 into a group for separate consideration, divides the indicator data into data within the cluster group and data outside the cluster group, and clusters the database to be optimized according to the indicator characteristics. The process includes: when the data in the database to be optimized is initially grouped based on an indicator value of 0, it is recorded as a data set outside the cluster group; otherwise, it is recorded as a data set within the cluster group.

[0041] Based on one or more of the above embodiments, when the total amount of data in the data set outside the cluster group is 0, the data set inside the cluster group is regrouped according to the index value interval and directly output as the regrouping result of the database to be optimized.

[0042] Based on one or more of the above embodiments, Figure 4-Figure 6The process of obtaining grouping nodes that meet the expected number of regroupings according to the maximum slope and the minimum slope of the first curve includes: obtaining a reference tangent: calculating the middle slope value according to the maximum slope and the minimum slope of the first curve, and fitting a reference tangent of the corresponding slope of the first curve according to the middle slope value; obtaining an endpoint tangent: obtaining a straight line that is parallel to the reference tangent, intersects with the first curve and is farthest from the first curve in the two-dimensional coordinate system as the endpoint tangent; obtaining a grouping node; inserting a straight line parallel to the reference tangent at equal distances between the reference tangent and the endpoint tangent until the intersection of the inserted straight line and the first curve meets the expected number of regroupings.

[0043] An electronic device comprises: a memory storing computer program instructions; and a processor, which implements the steps of any one of the above-mentioned preferred embodiment methods when the computer program instructions are executed by the processor.

[0044] A computer-readable storage medium is used to store instructions. When the stored instructions are executed on a computer, the computer is caused to execute the steps of any one of the methods described above.

[0045] The protection scope of the present invention is not limited thereto, and any change or replacement of the technical solution that can be thought of without creative work should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope defined in the claims.

Claims

1. A data distribution balanced optimization regrouping method, characterized in that: include: Obtain the database to be optimized, and determine the initial grouping basis index of the database to be optimized; Classify the database data to be optimized according to the initial grouping basis index, and obtain the data set within the cluster group and the data set outside the cluster group; Get the index value of the index for the initial grouping of each data in the data set within the cluster group; Performing initial grouping of the data set within the cluster group according to the index value, and determining the amount of data in the data group corresponding to each index value; Sort each data group in ascending order according to the amount of data, and fit the sorting results into a first curve; Obtaining the maximum slope value and the minimum slope value of the first curve, and obtaining grouping nodes that meet the expected number of regroupings according to the maximum slope value and the minimum slope value of the first curve; Obtain the index value at each grouping node, and sort each grouping node from large to small / small to large according to the index value, and obtain the index value interval of adjacent grouping nodes; The data sets within the clustering group are regrouped according to the index value interval, and the data sets outside the clustering group are combined as the first group or the last group to complete the regrouping of the database to be optimized.

2. The method according to claim 1, characterized in that: The data to be optimized are patent documents, and the initial grouping is based on an indicator selected from the group consisting of the number of citations, the number of citations, the number of claims, and the number of families.

3. The method according to claim 1, characterized in that The data to be optimized are paper documents, and the initial grouping basis indicator is one of the number of authors and the impact value of the document.

4. The method according to claim 2 or 3, characterized in that: The process of clustering the database to be optimized according to the indicator characteristics includes: when the indicator value of the indicator according to which the data in the database to be optimized is initially grouped is 0, it is recorded as a data set outside the cluster group, otherwise it is recorded as a data set within the cluster group.

5. The method according to claim 4, characterized in that When the total amount of data in the data set outside the cluster group is 0, the data set inside the cluster group is regrouped according to the index value interval and directly output as the regrouping result of the database to be optimized.

6. The method according to claim 1, characterized in that The process of obtaining grouping nodes that meet the expected number of regroupings according to the maximum slope value and the minimum slope value of the first curve includes: Obtaining a reference tangent line: calculating a slope intermediate value according to a maximum slope value and a slope minimum value of the first curve, and fitting a reference tangent line of a slope corresponding to the first curve according to the slope intermediate value; Get endpoint tangent: Get the straight line that is parallel to the reference tangent, intersects with the maximum value point of the first curve, and is farthest from the reference tangent in the two-dimensional coordinate system of the first curve as the endpoint tangent; Obtain a grouping node; insert a straight line parallel to the reference tangent at equal distances between the reference tangent and the endpoint tangent until the intersection of the inserted straight line and the first curve meets the expected number of regroupings.

7. An electronic device comprising: a memory storing computer program instructions; A processor, when the computer program instructions are executed by the processor, implements the steps of the data distribution balancing optimization regrouping method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store instructions. When the stored instructions are executed on a computer, the computer is caused to execute the steps of the data distribution balancing optimization regrouping method as described in any one of claims 1 to 6.

Images