Method, device and computer equipment for screening information

Through the method of filtering information in Hadoop, avoid full permutation when filtering and sorting data, reduce resource consumption, and improve data processing efficiency, the problem of large resource consumption in traditional methods is solved, and the application needs of top value of different data types is adapted.

CN111737263BActive Publication Date: 2024-11-05CHINA PING AN PROPERTY INSURANCE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010621789.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-06-30
Publication Date
2024-11-05
Estimated Expiration
2040-06-30

AI Technical Summary

Technical Problem

When Hadoop uses hive for offline data processing in a data warehouse, the traditional grouping and sorting method consumes a lot of resources and only saves one filtered result data, which is not conducive to the expansion of data applications.

Method used

A method of filtering information is proposed. By filtering data that meets the size requirements from the source data, sorting it into the linked list according to preset order, comparing and extracting data that meets the conditions, the sorting and filtering process is optimized, and the ROW_NUMBER() function pair is avoided. All data are fully arranged.

Benefits of technology

It reduces the calculation amount and sorting complexity, improves data processing efficiency, simplifies the comparison and value acquisition process, and allows the intermediate storage of data to store multiple tuple numbers, making it convenient for the application of top values ​​of different data types.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111737263B_ABST
    Figure CN111737263B_ABST
Patent Text Reader

Abstract

The present application discloses a method for filtering information, including: filtering first data that meets size requirements from source data, and storing them in a preset linked list in sequence according to a preset sort to form a first linked list; selecting two first linked lists with the same data attributes from multiple first linked lists and performing a head data size comparison, storing the larger head data in a second linked list, and deleting the larger head data from the original first linked list, so that the data sorted after the larger head data becomes the new head data, and then performing a head data comparison until all the data that meets the requirements in the two first linked lists with the same data attributes are extracted into the second linked list, and according to the data comparison and extraction process of the two first linked lists, all the data with the same data attributes are extracted into the second linked list; obtaining the TOP value corresponding to the specified task from the second linked list; locking and displaying the TOP value corresponding to the specified task, as well as the associated information corresponding to the TOP value corresponding to the specified task, to simplify comparison and improve efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method, device and computer equipment for screening information. Background Art

[0002] When Hadoop is used as a data warehouse, Hive is often used for offline data processing. When Hive is used for offline data processing, grouping and sorting are needed to obtain the first piece of data. The traditional way is to use "row_number" plus "distributeby" to specify the corresponding grouping field, and then "sortby" to specify the field to be sorted and define it as the field "rn". Finally, the calculated data is used as a subquery, and then the filter condition "rn=1" is added to select the first record of the grouping and sorting. However, this requires all the data in a group to be aggregated and then sorted, which consumes a lot of resources. In addition, only one result data that is filtered out is saved in the data architecture, which is not conducive to the expansion of data applications. Summary of the invention

[0003] The main purpose of this application is to provide a method for filtering information, aiming to solve the technical problem of high resource consumption in existing data grouping and sorting.

[0004] This application proposes a method for screening information, comprising:

[0005] Filtering first data that meets the size requirement from the source data, and storing them in a preset linked list in sequence according to a preset sort order to form a first linked list, wherein the first data is any data in the source data;

[0006] Select two first linked lists with the same data attribute from the plurality of first linked lists and compare the size of the header data, store the larger header data into the second linked list, delete the larger header data from the original first linked list, make the data sorted after the larger header data become the new header data, and then compare the header data until all the data meeting the requirements in the two first linked lists with the same data attribute are extracted into the second linked list, and according to the data comparison and extraction process of the two first linked lists, all the data with the same data attribute are extracted into the second linked list;

[0007] Acquire a TOP value corresponding to a specified task from the second linked list, wherein the specified task belongs to any task in the task list, and the TOP value corresponding to the specified task is associated with corresponding associated information;

[0008] The TOP value corresponding to the specified task and the associated information corresponding to the TOP value corresponding to the specified task are locked and displayed.

[0009] Preferably, the first linked list includes a linked list structure and the number of tuples filled in the linked list according to a preset sorting, the preset sorting is in descending order, and the step of selecting the first data that meets the size requirement from the source data and storing them in the preset linked list in sequence according to the preset sorting to form the first linked list includes:

[0010] Get the first tuple number located at the start table of the first linked list;

[0011] Determine whether the first data is greater than or equal to the first number of tuples;

[0012] If yes, then adding a first new table before the descending sort of the starting table, so that the first new table becomes the starting table;

[0013] The first data is filled into the first new table to make the first data a new first tuple number, and the entry of all data in the first linked list is completed according to the adding process of the first new table and the data filling process in the first new table.

[0014] Preferably, after the step of determining whether the first data is greater than or equal to the first number of tuples, the method further comprises:

[0015] If not, obtaining the number of the second tuple adjacent to the start table and after the start table is sorted;

[0016] Determine whether the first data is greater than or equal to the second tuple number;

[0017] If yes, then add a second new table before the second tuple number;

[0018] The first data is filled into the second new table, and according to the adding process of the first new table and the second new table, and the data filling process in the first new table and the second new table, the entry of all data in the first linked table is completed.

[0019] Preferably, after the step of determining whether the first data is greater than or equal to the second tuple number, the method further comprises:

[0020] If not, sequentially obtain the third element group arranged after the second tuple number in the descending order until the last number in the descending order;

[0021] Determine whether the first data is greater than or equal to the last digit;

[0022] If not, then add a third new table after the last number in the descending order;

[0023] The third tuple number is filled into the third new table, and according to the adding process of the first new table, the second new table and the third new table, and the data filling process in the first new table, the second new table and the third new table, the entry of all data in the first linked table is completed.

[0024] Preferably, after completing the step of entering all the data in the first linked list, the following steps are included:

[0025] Determine whether the length of the first linked list after all data are added is greater than a first preset value;

[0026] If so, a specified number of tail tables arranged in the first linked list are removed, and the number of tuples filled in the specified number of tail tables is deleted, so that the linked list length of the first linked list is maintained less than or equal to the first preset amount.

[0027] Preferably, before the step of selecting the first data that meets the size requirement from the source data and sequentially storing the first data in a preset linked list according to a preset sort order to form the first linked list, the method includes:

[0028] Determine whether the number of tuples in the first linked list is greater than a second preset amount;

[0029] If so, stop filtering data from the source data and stop filling data into the first linked list.

[0030] Preferably, the step of selecting two first linked lists with the same data attributes from the plurality of first linked lists and comparing the sizes of the header data, storing the larger header data in the second linked list, and deleting the larger header data from the original first linked list, so that the data sorted after the larger header data becomes the new header data, and then comparing the header data until all the data meeting the requirements in the two first linked lists with the same data attributes are extracted into the second linked list, and extracting all the data with the same data attributes into the second linked list according to the data comparison and extraction process of the two first linked lists, includes:

[0031] Compare the number of head tuples of the first specified linked list with the number of head tuples of the second specified linked list, wherein the first specified linked list and the second specified linked list are two linked lists with the same data attributes and sorting rules among all the first linked lists, and the arrangement rule includes descending order;

[0032] If the number of head tuples of the first specified linked list is greater than or equal to the number of head tuples of the second specified linked list, the number of head tuples of the first specified linked list is extracted to the second linked list, and the number of head tuples of the first specified linked list is removed, and the number of tuples adjacent to the number of head tuples of the first specified linked list and located after the number of head tuples of the first specified linked list in descending order is changed to the new number of head tuples;

[0033] Compare the number of new head tuples of the first specified linked list with the number of head tuples of the second specified linked list;

[0034] If the number of new head tuples of the first designated linked list is less than the number of head tuples of the second designated linked list, the number of head tuples of the second designated linked list is extracted to the second designated linked list and arranged after the number of head tuples of the first designated linked list to form a descending arrangement, and the number of head tuples of the second designated linked list is removed, and the number of tuples adjacent to the number of head tuples of the second designated linked list and located after the number of head tuples of the second designated linked list in descending order is changed to the number of new head tuples;

[0035] According to the comparison process of the number of head tuples of the first designated linked list and the number of head tuples of the second designated linked list, and the comparison process of the new number of head tuples of the first designated linked list and the number of head tuples of the second designated linked list, continue to compare the remaining number of tuples in the first designated linked list and the second designated linked list until the number of data in the second linked list reaches the preset condition, and complete the entry of all data in the second linked list.

[0036] The present application also provides a device for screening information, comprising:

[0037] A screening module, used for screening first data that meets the size requirement from the source data, and storing the first data in a preset linked list in sequence according to a preset sorting order to form a first linked list, wherein the first data is any data in the source data;

[0038] a comparison module, configured to select two first linked lists with the same data attribute from the plurality of first linked lists and compare the size of the header data, store the larger header data in the second linked list, delete the larger header data from the original first linked list, make the data sorted after the larger header data become the new header data, and then compare the header data until all the data meeting the requirements in the two first linked lists with the same data attribute are extracted into the second linked list, and according to the data comparison and extraction process of the two first linked lists, all the data with the same data attribute are extracted into the second linked list;

[0039] an acquisition module, configured to acquire a TOP value corresponding to a specified task from the second linked list, wherein the specified task belongs to any task in the task list, and the TOP value corresponding to the specified task is associated with corresponding associated information;

[0040] The locking module is used to lock and display the TOP value corresponding to the specified task and the associated information corresponding to the TOP value corresponding to the specified task.

[0041] The present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0042] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0043] The present application changes the data storage structure so that the intermediate storage data can store multiple tuples, which is convenient for data applications that require top values ​​of different data types. Based on the improved data storage structure, the screening method for sorting and obtaining top values ​​is optimized, which simplifies the comparison and value-taking process and improves data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a method for screening information according to an embodiment of the present application;

[0045] Figure 2 A schematic diagram of the structure of a device for filtering information according to an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the internal structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] Reference Figure 1 , a method for screening information according to an embodiment of the present application includes:

[0049] S1: Filtering first data that meets the size requirement from the source data, and storing them in a preset linked list in sequence according to a preset sorting to form a first linked list, wherein the first data is any data in the source data;

[0050] S2: Select two first linked lists with the same data attributes from the plurality of first linked lists and compare the sizes of the header data, store the larger header data in the second linked list, delete the larger header data from the original first linked list, make the data sorted after the larger header data become the new header data, and then compare the header data again until all the data meeting the requirements in the two first linked lists with the same data attributes are extracted into the second linked list, and according to the data comparison and extraction process of the two first linked lists, all the data with the same data attributes are extracted into the second linked list;

[0051] S3: acquiring a TOP value corresponding to a specified task from the second linked list, wherein the specified task belongs to any task in the task list, and the TOP value corresponding to the specified task is associated with corresponding associated information;

[0052] S4: Locking and displaying the TOP value corresponding to the designated task and the associated information corresponding to the TOP value corresponding to the designated task.

[0053] In the embodiment of the present application, the above-mentioned source data includes historical data formed in the process of using the system by collecting browsing logs, etc., or conclusion data formed by statistically analyzing the historical data in the above-mentioned use process. For example, the source data includes: historical data in the process of using the system such as the total number of browsing times, browsing frequency, purchase data, etc., and also includes conclusion data such as the degree of liking and demand formed by analyzing the total number of browsing times, browsing frequency, purchase data, etc. This application changes the data structure of the above-mentioned first linked list and changes the filter function so that there can be multiple tuples arranged in order in the first linked list at the same time, and the above order includes descending order or ascending order, for example. The ROW_NUMBER() function sorting is abandoned, and the execution of the ROW_NUMBER() function needs to be fully arranged for all data and then the Top N that meets the task requirements is taken. In this embodiment, through a custom function, in the process of filtering data from the source data, comparison, screening and sorting are started, and there is no need for the ROW_NUMBER() function to fully arrange all data and then filter, which reduces the amount of calculation and sorting complexity. The above-mentioned custom function is an aggregation function designed using the udaf feature of Hive, including maxList and minList. The number of the second linked list mentioned above is the same as the number of maps designed in the above aggregation function, that is, the number of categories of data attributes. The method of filtering information in the present application can change the number of required TOP values ​​at any time as needed to meet the information needs of different data analyses. By changing the data storage structure, the intermediate storage data can store multiple tuples, which is convenient for data applications that require top values ​​of different data types, and based on the improved data storage structure, the sorting and filtering method for obtaining top values ​​is optimized, which simplifies the comparison and value-taking process and improves data processing efficiency.

[0054] Furthermore, the first linked list includes a linked list structure and the number of tuples filled in the linked list according to a preset sorting, the preset sorting is in descending order, and the step S1 of selecting first data that meets the size requirement from the source data and storing them in the preset linked list in sequence according to the preset sorting to form the first linked list includes:

[0055] S11: Obtain the first tuple number located at the start table of the first linked list;

[0056] S12: Determine whether the first data is greater than or equal to the first tuple number;

[0057] S13: If yes, then add a first new table before the descending sort of the starting table, so that the first new table becomes the starting table;

[0058] S14: Fill the first data into the first new table to make the first data a new first tuple number, and complete the entry of all data in the first linked list according to the adding process of the first new table and the data filling process in the first new table.

[0059] In this embodiment, the first element of the tuple number stores the field to be compared, and the other elements store other fields. The above-mentioned preset linked list is a custom linked list for storing element groups. The preset linked list contains a blank linked list structure and the number of tuples currently filled in the linked list. Every time a first data is obtained from the source data, the comparison starts from the starting table of the preset linked list. If it is greater than or equal to the first tuple number, the first data is inserted in front of the original first tuple number, so that the original first tuple number will be the second tuple number. The above-mentioned first tuple number is the data that is ranked first and filled in the starting table, and the second tuple number is the data filled in the table after the starting table. The directional expressions "before sorting, after sorting" in this embodiment are all based on descending sorting as an example, that is, the data before sorting is greater than the data after sorting. In ascending sorting, the process of filtering information is opposite to the descending process, and the principle is similar, so it will not be repeated. For example: the latest record of policy modification information, fields: policy number, update time, policy status; Select policy number, rs[0] as update time, rs[1] as policy status; From{Select policy number maxList[array(update time, policy status)]rs from policy modification table group by policy number}tmp. Using the characteristics of udaf, you can make a comparison in the map stage, that is, a group may have 100 records, but if you want to get the max, you don’t need to get all the records. You only need to get two data to compare the largest one first. After the comparison, the largest data in the remaining data can be compared with other data. There is no need to aggregate all the data and then sort them as a whole, that is, there is no need to call the ROW_NUMBER() function to fully arrange all the data and then calculate the sort.

[0060] Further, after step S12 of determining whether the first data is greater than or equal to the first number of tuples, the following steps are included:

[0061] S15: If not, obtaining the number of second tuples adjacent to the start table and located after the start table is sorted;

[0062] S16: Determine whether the first data is greater than or equal to the second tuple number;

[0063] S13: If yes, add a second new table before the second tuple number;

[0064] S14: Fill the first data into the second new table, and complete the entry of all data in the first linked table according to the adding process of the first new table and the second new table, and the data filling process in the first new table and the second new table.

[0065] In this embodiment, a suitable insertion position can be selected according to the size of the first data, so that after all the data in the first linked list are entered, all the data in the first linked list are arranged in descending order from the start table. The above-mentioned terms such as "first", "second", etc., except when used with tuple numbers with sorting restrictions, such as the first tuple number and the second tuple number represent the first and second sorting, the "first", "second", etc. and similar terms in other places are only used for distinction and are not used for limitation.

[0066] Further, after the step of determining whether the first data is greater than or equal to the second tuple number S16, the following steps are included:

[0067] S161: If not, sequentially obtain the third element groups arranged after the second tuple number in the descending order until the last number in the descending order;

[0068] S162: Determine whether the first data is greater than or equal to the last digit;

[0069] S163: If not, then add a third new table after the last number in the descending order;

[0070] S164: Fill the third tuple number into the third new table, and complete the entry of all data in the first linked table according to the adding process of the first new table, the second new table and the third new table, and the data filling process in the first new table, the second new table and the third new table.

[0071] In this embodiment, data is compared starting from the start table. If the comparison is continued to the end of the first linked list and the insertion requirement of the first data is still not met, that is, the first data is smaller than all the existing data in the first linked list, the first data is inserted to the end of the linked list so that the first linked list always maintains the original arrangement order during the data filling process. This embodiment takes descending arrangement as an example. The data filling process in ascending arrangement in other embodiments is the same in principle, but the data comparison process is opposite, which is not repeated.

[0072] Further, after completing the step of entering all the data in the first linked list, the following steps are included:

[0073] S111: Determine whether the length of the first linked list after all data are added is greater than a first preset value;

[0074] S166: If yes, remove the specified number of tail tables arranged in the first linked list, and delete the number of tuples filled in the specified number of tail tables, so that the linked list length of the first linked list is maintained less than or equal to the first preset amount.

[0075] In this embodiment, after completing the filling of all data in the first linked list, it is determined whether the length of the linked list is greater than the preset maximum value. If it is greater, the specified number of tail tuples in the descending order and their filling table are removed to reduce the amount of calculation while meeting the need to filter the information corresponding to the top value.

[0076] Furthermore, before the step S1 of selecting the first data that meets the size requirement from the source data and sequentially storing the first data in a preset linked list according to a preset sort order to form the first linked list, the step includes:

[0077] S101: Determine whether the number of tuples in the first linked list is greater than a second preset amount;

[0078] S102: If yes, stop filtering data from the source data and stop filling data into the first linked list.

[0079] In this embodiment, during the process of filling data in the first linked list in descending order, it is determined in real time whether the number of tuples in the first linked list reaches the preset number requirement. When the number reaches the requirement, the process of obtaining data from the source data and filling it into the first linked list is stopped, so as to save the amount of calculation and improve the efficiency. For example, when information screening needs to screen the data corresponding to the top 5, it is sufficient to obtain the data in the top 5 in descending order.

[0080] Further, the step S2 of selecting two first linked lists with the same data attributes from the plurality of first linked lists and comparing the sizes of the header data, storing the larger header data in the second linked list, and deleting the larger header data from the original first linked list, so that the data sorted after the larger header data becomes the new header data, and then comparing the header data until all the data meeting the requirements in the two first linked lists with the same data attributes are extracted into the second linked list, and extracting all the data with the same data attributes into the second linked list according to the data comparison and extraction process of the two first linked lists, includes:

[0081] S21: Compare the number of head tuples of the first specified linked list with the number of head tuples of the second specified linked list, wherein the first specified linked list and the second specified linked list are two linked lists with the same data attributes and sorting rules among all the first linked lists, and the sorting rules include descending order;

[0082] S22: If the number of head tuples of the first designated linked list is greater than or equal to the number of head tuples of the second designated linked list, the number of head tuples of the first designated linked list is extracted to the second linked list, and the number of head tuples of the first designated linked list is removed, and the number of tuples adjacent to the number of head tuples of the first designated linked list and located after the number of head tuples of the first designated linked list in descending order is changed to the new number of head tuples;

[0083] S23: comparing the number of new head tuples of the first specified linked list with the number of head tuples of the second specified linked list;

[0084] S24: If the number of new head tuples of the first designated linked list is less than the number of head tuples of the second designated linked list, the number of head tuples of the second designated linked list is extracted to the second linked list and arranged after the number of head tuples of the first designated linked list to form a descending arrangement, and the number of head tuples of the second designated linked list is removed, and the number of tuples adjacent to the number of head tuples of the second designated linked list and located after the number of head tuples of the second designated linked list in descending order is changed to the number of new head tuples;

[0085] S25: According to the comparison process of the number of head tuples of the first designated linked list and the number of head tuples of the second designated linked list, and the comparison process of the number of new head tuples of the first designated linked list and the number of head tuples of the second designated linked list, continue to compare the remaining number of tuples in the first designated linked list and the second designated linked list until the number of data in the second linked list reaches the preset condition, and complete the entry of all data in the second linked list.

[0086] In this embodiment, there are multiple first linked lists, and the number is the same as the number of maps. When filtering data, it is necessary to comprehensively consider the first linked lists corresponding to all related maps, and filter out the number of tuples that meet the topN requirements from the above first linked lists. By comparing the heads of the two first linked lists in sequence, that is, the largest data in the first linked lists sorted in descending order, the larger of the largest data in the two first linked lists is extracted to the second linked list, and deleted from the original first linked list, so that the second tuple number in the original first linked list becomes the head tuple number, and the head tuples of the two linked lists are continued to be compared until the two linked lists are compared or the upper limit of the amount of data in the second linked list is reached. For example, one first linked list is {7, 3, 1}, and the other first linked list is {8, 6, 2, 2}. In the comparison, 7 and 8 are compared first. If 7 is less than 8, 8 is extracted to the second linked list, and {8, 6, 2, 2} is changed to {6, 2, 2}. Then 7 and 6 are compared again. If 7 is greater than 6, 7 is extracted to the second linked list, and {7, 3, 1} is updated to {3, 1}. This process is repeated until all data are compared and the second linked list becomes {8, 7, 6, 3, 2, 2, 1}. Or if the amount of data in the second linked list reaches the topN requirement, for example, if the top5 needs to be obtained, the calculation ends when the data in the second linked list becomes {8, 7, 6, 3, 2}.

[0087] The sorting of all the above embodiments has an average time complexity of O(n*N), indicating that the sorting scheme is simple and efficient. The above sorting uses the udaf feature of Hive to set the aggregation function, which can be defined as maxList and minList. The above UDAF stage includes the following data processing stages: PARTIAL1: Aggregation of source data to partial data, that is, the process of forming all first linked lists, calling iterate and terminatePartial, that is, the map stage; FINAL: Aggregation of partial data to complete data, that is, the process of aggregating the first linked list to form the second linked list, calling merge and terminate, that is, the reduce stage; COMPLETE: Directly from source data to the map stage of complete data aggregation, and there is no reduce. In addition to iterate, merge, and terminatePartial mentioned above, there are also init, that is, initialization and return, with the type of return value, getNewAggregationBuffer, that is, obtaining a new buffer, that is, the object of passing parameters between methods, reset means resetting the buffer object and other data processing stages. getNewAggregationBuffer: used to define an object, wrap the passed list into an object for transmission; reset the buffer object, that is, update the record with the largest record; iterate is the arrangement aggregation of a data record and partial sorting results, that is, merge a data record into the partial sorting results to maintain the original sorting rules; merge is the sorting of partial sorting results and partial sorting results, respectively get the two first linked lists and compare their first elements, and store the first linked lists in the intermediate variable; terminatePartial means data verification, and returns null when the passed content is not an array; terminate means end.

[0088] Reference Figure 2 , a device for screening information according to an embodiment of the present application includes:

[0089] A screening module 1 is used to screen first data that meets the size requirement from the source data, and store the first data in a preset linked list in sequence according to a preset sort order to form a first linked list, wherein the first data is any data in the source data;

[0090] Comparison module 2, used for selecting two first linked lists with the same data attributes from the plurality of first linked lists and performing head data size comparison, storing the larger head data into the second linked list, and deleting the larger head data from the original first linked list, so that the data sorted after the larger head data becomes the new head data, and then performing head data comparison until all the data meeting the requirements in the two first linked lists with the same data attributes are extracted into the second linked list, and according to the data comparison and extraction process of the two first linked lists, all the data with the same data attributes are extracted into the second linked list;

[0091] An acquisition module 3 is used to acquire a TOP value corresponding to a specified task from the second linked list, wherein the specified task belongs to any task in the task list, and the TOP value corresponding to the specified task is associated with corresponding associated information;

[0092] The locking module 4 is used to lock and display the TOP value corresponding to the designated task and the associated information corresponding to the TOP value corresponding to the designated task.

[0093] In the embodiment of the present application, the above-mentioned source data includes historical data formed in the process of using the system by collecting browsing logs, etc., or conclusion data formed by statistically analyzing the historical data in the above-mentioned use process. For example, the source data includes: historical data in the process of using the system such as the total number of browsing times, browsing frequency, purchase data, etc., and also includes conclusion data such as the degree of liking and demand formed by analyzing the total number of browsing times, browsing frequency, purchase data, etc. This application changes the data structure of the above-mentioned first linked list and changes the filter function so that there can be multiple tuples arranged in order in the first linked list at the same time, and the above order includes descending order or ascending order, for example. The ROW_NUMBER() function sorting is abandoned, and the execution of the ROW_NUMBER() function needs to be fully arranged for all data and then the Top N that meets the task requirements is taken. In this embodiment, through a custom function, in the process of filtering data from the source data, comparison, screening and sorting are started, and there is no need for the ROW_NUMBER() function to fully arrange all data and then filter, which reduces the amount of calculation and sorting complexity. The above-mentioned custom function is an aggregation function designed using the udaf feature of Hive, including maxList and minList. The number of the second linked list mentioned above is the same as the number of maps designed in the above aggregation function, that is, the number of categories of data attributes. The method of filtering information in the present application can change the number of required TOP values ​​at any time as needed to meet the information needs of different data analyses. By changing the data storage structure, the intermediate storage data can store multiple tuples, which is convenient for data applications that require top values ​​of different data types, and based on the improved data storage structure, the sorting and filtering method for obtaining top values ​​is optimized, which simplifies the comparison and value-taking process and improves data processing efficiency.

[0094] Furthermore, the first linked list includes a linked list structure and the number of tuples filled in the linked list according to a preset sorting, the preset sorting is in descending order, and the screening module 1 includes:

[0095] A first acquisition unit, used for acquiring a first tuple number located at the start table of the first linked list;

[0096] A first judging unit, used for judging whether the first data is greater than or equal to the first number of tuples;

[0097] a first adding unit, configured to add a first new table before the descending sorting of the starting table if the first data is greater than or equal to the first number of tuples, so that the first new table becomes the starting table;

[0098] A first filling unit is used to fill the first data into the first new table so that the first data becomes a new first tuple number, and complete the entry of all data in the first linked list according to the adding process of the first new table and the data filling process in the first new table.

[0099] In this embodiment, the first element of the tuple number stores the field to be compared, and the other elements store other fields. The above-mentioned preset linked list is a custom linked list for storing element groups. The preset linked list contains a blank linked list structure and the number of tuples currently filled in the linked list. Every time a first data is obtained from the source data, the comparison starts from the starting table of the preset linked list. If it is greater than or equal to the first tuple number, the first data is inserted in front of the original first tuple number, so that the original first tuple number will be the second tuple number. The above-mentioned first tuple number is the data that is ranked first and filled in the starting table, and the second tuple number is the data filled in the table after the starting table. The directional expressions "before sorting, after sorting" in this embodiment are all based on descending sorting as an example, that is, the data before sorting is greater than the data after sorting. In ascending sorting, the process of filtering information is opposite to the descending process, and the principle is similar, so it will not be repeated. For example: the latest record of policy modification information, fields: policy number, update time, policy status; Select policy number, rs[0] as update time, rs[1] as policy status; From{Select policy number maxList[array(update time, policy status)]rs from policy modification table group by policy number}tmp. Using the characteristics of udaf, you can make a comparison in the map stage, that is, a group may have 100 records, but if you want to get the max, you don’t need to get all the records. You only need to get two data to compare the largest one first. After the comparison, the largest data in the remaining data can be compared with other data. There is no need to aggregate all the data and then sort them as a whole, that is, there is no need to call the ROW_NUMBER() function to fully arrange all the data and then calculate the sort.

[0100] Furthermore, the screening module 1 comprises:

[0101] A second acquisition unit, configured to acquire a second tuple number adjacent to the start table and located after the start table is sorted, if the first data is smaller than the first tuple number;

[0102] A second judging unit, used for judging whether the first data is greater than or equal to the second tuple number;

[0103] a second adding unit, configured to add a second new table before the second tuple number if the first data is greater than or equal to the second tuple number;

[0104] The second filling unit is used to fill the first data into the second new table, and complete the entry of all data in the first linked table according to the adding process of the first new table and the second new table, and the data filling process in the first new table and the second new table.

[0105] In this embodiment, a suitable insertion position can be selected according to the size of the first data, so that after all the data in the first linked list are entered, all the data in the first linked list are arranged in descending order from the start table. The above-mentioned terms such as "first", "second", etc., except when used with tuple numbers with sorting restrictions, such as the first tuple number and the second tuple number represent the first and second sorting, the "first", "second", etc. and similar terms in other places are only used for distinction and are not used for limitation.

[0106] Furthermore, the screening module 1 comprises:

[0107] A third acquisition unit is used for, if the first data is smaller than the second tuple number, sequentially acquiring the third element group arranged after the second tuple number in the descending order until the last number in the descending order;

[0108] A third judging unit, used for judging whether the first data is greater than or equal to the last digit;

[0109] A third adding unit, configured to add a third new table after the last number in descending order if the first data is smaller than the last number;

[0110] A third filling unit is used to fill the third tuple number into the third new table, and complete the entry of all data in the first linked table according to the adding process of the first new table, the second new table and the third new table, and the data filling process in the first new table, the second new table and the third new table.

[0111] In this embodiment, data is compared starting from the start table. If the comparison is continued to the end of the first linked list and the insertion requirement of the first data is still not met, that is, the first data is smaller than all the existing data in the first linked list, the first data is inserted to the end of the linked list so that the first linked list always maintains the original arrangement order during the data filling process. This embodiment takes descending arrangement as an example. The data filling process in ascending arrangement in other embodiments is the same in principle, but the data comparison process is opposite, which is not repeated.

[0112] Furthermore, the screening module 1 comprises:

[0113] A fourth judging unit, used to judge whether the length of the first linked list after all data are added is greater than a first preset value;

[0114] A removal unit is used to remove a specified number of tail tables arranged in the first linked list if the length of the linked list is greater than a first preset value, and at the same time delete the number of tuples filled in the specified number of tail tables, so that the length of the first linked list is maintained at less than or equal to the first preset value.

[0115] In this embodiment, after completing the filling of all data in the first linked list, it is determined whether the length of the linked list is greater than the preset maximum value. If it is greater, the specified number of tail tuples in the descending order and their filling table are removed to reduce the amount of calculation while meeting the need to filter the information corresponding to the top value.

[0116] Furthermore, the device for screening information includes:

[0117] A judging module, used for judging whether the number of tuples in the first linked list is greater than a second preset number;

[0118] A termination module is used to terminate the filtering of data from the source data and the filling of data into the first linked list if the number of tuples in the first linked list is greater than a second preset amount.

[0119] In this embodiment, during the process of filling data in the first linked list in descending order, it is determined in real time whether the number of tuples in the first linked list reaches the preset number requirement. When the number reaches the requirement, the process of obtaining data from the source data and filling it into the first linked list is stopped, so as to save the amount of calculation and improve the efficiency. For example, when information screening needs to screen the data corresponding to the top 5, it is sufficient to obtain the data in the top 5 in descending order.

[0120] Furthermore, the comparison module 2 comprises:

[0121] A first comparison unit is used to compare the number of head tuples of the first specified linked list with the number of head tuples of the second specified linked list, wherein the first specified linked list and the second specified linked list are two linked lists with the same data attributes and sorting rules among all the first linked lists, and the sorting rules include descending order;

[0122] A first extraction unit is used for extracting the number of head tuples of the first designated linked list to the second designated linked list if the number of head tuples of the first designated linked list is greater than or equal to the number of head tuples of the second designated linked list, and removing the number of head tuples of the first designated linked list, and changing the number of tuples that are adjacent to the number of head tuples of the first designated linked list and are located after the number of head tuples of the first designated linked list in descending order to a new number of head tuples;

[0123] A second comparing unit, used for comparing the number of new head tuples of the first specified linked list with the number of head tuples of the second specified linked list;

[0124] A second extraction unit is used for extracting the number of head tuples of the second designated linked list to the second designated linked list if the number of new head tuples of the first designated linked list is less than the number of head tuples of the second designated linked list, and arranging the number of head tuples after the number of head tuples of the first designated linked list to form a descending arrangement, and removing the number of head tuples of the second designated linked list, and changing the number of tuples adjacent to the number of head tuples of the second designated linked list and located after the number of head tuples of the second designated linked list in descending order to the number of new head tuples;

[0125] The third comparison unit is used to continue comparing the remaining tuple numbers in the first designated linked list and the second designated linked list according to the comparison process of the head tuple number of the first designated linked list and the head tuple number of the second designated linked list, and the comparison process of the new head tuple number of the first designated linked list and the head tuple number of the second designated linked list, until the number of data in the second linked list reaches a preset condition, thereby completing the entry of all data in the second linked list.

[0126] In this embodiment, there are multiple first linked lists, and the number is the same as the number of maps. When filtering data, it is necessary to comprehensively consider the first linked lists corresponding to all related maps, and filter out the number of tuples that meet the topN requirements from the above first linked lists. By comparing the heads of the two first linked lists in sequence, that is, the largest data in the first linked lists sorted in descending order, the larger of the largest data in the two first linked lists is extracted to the second linked list, and deleted from the original first linked list, so that the second tuple number in the original first linked list becomes the head tuple number, and the head tuples of the two linked lists are continued to be compared until the two linked lists are compared or the upper limit of the amount of data in the second linked list is reached. For example, one first linked list is {7, 3, 1}, and the other first linked list is {8, 6, 2, 2}. In the comparison, 7 and 8 are compared first. If 7 is less than 8, 8 is extracted to the second linked list, and {8, 6, 2, 2} is changed to {6, 2, 2}. Then 7 and 6 are compared again. If 7 is greater than 6, 7 is extracted to the second linked list, and {7, 3, 1} is updated to {3, 1}. This process is repeated until all data are compared and the second linked list becomes {8, 7, 6, 3, 2, 2, 1}. Or if the amount of data in the second linked list reaches the topN requirement, for example, if the top5 needs to be obtained, the calculation ends when the data in the second linked list becomes {8, 7, 6, 3, 2}.

[0127] The sorting of all the above embodiments has an average time complexity of O(n*N), indicating that the sorting scheme is simple and efficient. The above sorting uses the udaf feature of Hive to set the aggregation function, which can be defined as maxList and minList. The above UDAF stage includes the following data processing stages: PARTIAL1: Aggregation of source data to partial data, that is, the process of forming all first linked lists, calling iterate and terminatePartial, that is, the map stage; FINAL: Aggregation of partial data to complete data, that is, the process of aggregating the first linked list to form the second linked list, calling merge and terminate, that is, the reduce stage; COMPLETE: Directly from source data to the map stage of complete data aggregation, and there is no reduce. In addition to iterate, merge, and terminatePartial mentioned above, there are also init, that is, initialization and return, with the type of return value, getNewAggregationBuffer, that is, obtaining a new buffer, that is, the object of passing parameters between methods, reset means resetting the buffer object and other data processing stages. getNewAggregationBuffer: used to define an object, wrap the passed list into an object for transmission; reset the buffer object, that is, update the record with the largest record; iterate is the arrangement aggregation of a data record and partial sorting results, that is, merge a data record into the partial sorting results to maintain the original sorting rules; merge is the sorting of partial sorting results and partial sorting results, respectively get the two first linked lists and compare their first elements, and store the first linked lists in the intermediate variable; terminatePartial means data verification, and returns null when the passed content is not an array; terminate means end.

[0128] Reference Figure 3 In the embodiment of the present application, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store all data required for the process of filtering information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for filtering information is implemented.

[0129] The above-mentioned processor executes the above-mentioned method for filtering information, including: filtering first data that meets the size requirements from source data, and storing them in a preset linked list in sequence according to a preset sort to form a first linked list, wherein the first data is any data in the source data; selecting two first linked lists with the same data attributes from multiple first linked lists and performing a header data size comparison, storing the larger header data in a second linked list, and deleting the larger header data from the original first linked list, so that the data sorted after the larger header data becomes the new header data, and then performing a header data comparison until all the data that meets the requirements in the two first linked lists with the same data attributes are extracted to the second linked list, and according to the data comparison and extraction process of the two first linked lists, all the data with the same data attributes are extracted to the second linked list; obtaining the TOP value corresponding to the specified task from the second linked list, wherein the specified task belongs to any task in the task list, and the TOP value corresponding to the specified task is associated with the corresponding associated information; locking and displaying the TOP value corresponding to the specified task, and the associated information corresponding to the TOP value corresponding to the specified task.

[0130] The above-mentioned computer device, by changing the data storage structure, enables the intermediate storage data to store multiple tuple numbers, which is convenient for data applications that require top values ​​of different data types, and optimizes the screening method of sorting to obtain top values ​​based on the improved data storage structure, simplifies the comparison and value-taking process, and improves data processing efficiency.

[0131] In one embodiment, the first linked list includes a linked list structure and a number of tuples filled in the linked list according to a preset sort, the preset sort is descending order, and the processor screens the first data that meets the size requirement from the source data, and stores them in the preset linked list in sequence according to the preset sort to form the first linked list, including: obtaining the first tuple number located in the starting table of the first linked list; determining whether the first data is greater than or equal to the first tuple number; if so, adding a first new table before the descending sort of the starting table to make the first new table the starting table; filling the first data into the first new table to make the first data a new first tuple number, and completing the entry of all data in the first linked list according to the adding process of the first new table and the data filling process in the first new table.

[0132] In one embodiment, after the step of the processor determining whether the first data is greater than or equal to the first tuple number, the processor further comprises: if not, obtaining a second tuple number adjacent to the starting table and located after the starting table is sorted; determining whether the first data is greater than or equal to the second tuple number; if so, adding a second new table before the second tuple number; filling the first data into the second new table, and completing the entry of all data in the first linked list according to the process of adding the first new table and the second new table, and the process of filling the data in the first new table and the second new table.

[0133] In one embodiment, after the processor determines whether the first data is greater than or equal to the second tuple number, the processor further comprises: if not, sequentially obtaining the third element group arranged after the second tuple number in the descending order until the last number in the descending order; determining whether the first data is greater than or equal to the last number; if not, adding a third new table after the last number in the descending order; filling the third tuple number into the third new table, and completing the entry of all data in the first linked list according to the adding process of the first new table, the second new table and the third new table, and the data filling process in the first new table, the second new table and the third new table.

[0134] In one embodiment, after the processor completes the step of entering all data in the first linked list, it includes: determining whether the linked list length of the first linked list after adding all data is greater than a first preset value; if so, removing a specified number of end tables arranged in the first linked list, and deleting the number of tuples filled in the specified number of end tables, so that the linked list length of the first linked list remains less than or equal to the first preset value.

[0135] In one embodiment, before the above-mentioned processor filters the first data that meets the size requirement from the source data and stores them in a preset linked list in sequence according to a preset order to form a first linked list, it includes: judging whether the number of tuples in the first linked list is greater than a second preset amount; if so, terminating the filtering of data from the source data and terminating the filling of data into the first linked list.

[0136] In one embodiment, the processor selects two first linked lists with the same data attributes from a plurality of first linked lists and compares the size of the header data, stores the larger header data in the second linked list, and deletes the larger header data from the original first linked list, so that the data sorted after the larger header data becomes the new header data, and then compares the header data until all the data meeting the requirements in the two first linked lists with the same data attributes are extracted to the second linked list, and according to the data comparison and extraction process of the two first linked lists, the step of extracting all the data with the same data attributes to the second linked list includes: comparing the number of header tuples of the first designated linked list with the number of header tuples of the second designated linked list, wherein the first designated linked list and the second designated linked list are two linked lists with the same data attributes and sorting rules among all the first linked lists, and the arrangement rule includes descending order; if the number of header tuples of the first designated linked list is greater than or equal to the number of header tuples of the second designated linked list, the number of header tuples of the first designated linked list is extracted to the second linked list, and the number of header tuples of the first designated linked list is removed, and the first designated linked list with the first designated linked list is extracted to the second linked list. The number of tuples that are adjacent to the head tuple number of the first specified linked list and are located after the head tuple number of the first specified linked list in descending order is changed to the new head tuple number; the new head tuple number of the first specified linked list is compared with the head tuple number of the second specified linked list; if the new head tuple number of the first specified linked list is less than the head tuple number of the second specified linked list, the head tuple number of the second specified linked list is extracted to the second specified linked list and arranged after the head tuple number of the first specified linked list to form a descending arrangement, and the head tuple number of the second specified linked list is removed, and the head tuple number of the second specified linked list is compared with the head tuple number of the second specified linked list. The number of tuples whose head tuple numbers of the second designated linked list are adjacent and located after the number of head tuples of the second designated linked list in descending order are changed to the new number of head tuples; according to the comparison process of the number of head tuples of the first designated linked list and the number of head tuples of the second designated linked list, and the comparison process of the new number of head tuples of the first designated linked list and the number of head tuples of the second designated linked list, the remaining number of tuples in the first designated linked list and the second designated linked list continue to be compared until the number of data in the second designated linked list reaches a preset condition, and the entry of all data in the second linked list is completed.

[0137] Those skilled in the art will understand that Figure 3 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0138] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a method for filtering information is implemented, including: filtering first data that meets the size requirement from source data, and storing them in a preset linked list in sequence according to a preset sort to form a first linked list, wherein the first data is any data in the source data; selecting two first linked lists with the same data attribute from a plurality of first linked lists and performing a size comparison of the header data, storing the larger header data in a second linked list, and deleting the larger header data from the original first linked list, so that the data sorted after the larger header data becomes the new header data, and then performing a header data comparison until all the data that meets the requirement in the two first linked lists with the same data attribute are extracted into the second linked list, and according to the data comparison and extraction process of the two first linked lists, all the data with the same data attribute are extracted into the second linked list; obtaining a TOP value corresponding to a specified task from the second linked list, wherein the specified task belongs to any task in the task list, and the TOP value corresponding to the specified task is associated with the corresponding associated information; locking and displaying the TOP value corresponding to the specified task, and the associated information corresponding to the TOP value corresponding to the specified task.

[0139] The above-mentioned computer-readable storage medium, by changing the data storage structure, enables the intermediate storage data to store multiple tuple numbers, which is convenient for data applications that require top values ​​of different data types, and optimizes the screening method of sorting to obtain top values ​​based on the improved data storage structure, simplifies the comparison and value-taking process, and improves data processing efficiency.

[0140] In one embodiment, the first linked list includes a linked list structure and a number of tuples filled in the linked list according to a preset sort, the preset sort is descending order, and the processor screens the first data that meets the size requirement from the source data, and stores them in the preset linked list in sequence according to the preset sort to form the first linked list, including: obtaining the first tuple number located in the starting table of the first linked list; determining whether the first data is greater than or equal to the first tuple number; if so, adding a first new table before the descending sort of the starting table to make the first new table the starting table; filling the first data into the first new table to make the first data a new first tuple number, and completing the entry of all data in the first linked list according to the adding process of the first new table and the data filling process in the first new table.

[0141] In one embodiment, after the step of the processor determining whether the first data is greater than or equal to the first tuple number, the processor further comprises: if not, obtaining a second tuple number adjacent to the starting table and located after the starting table is sorted; determining whether the first data is greater than or equal to the second tuple number; if so, adding a second new table before the second tuple number; filling the first data into the second new table, and completing the entry of all data in the first linked list according to the process of adding the first new table and the second new table, and the process of filling the data in the first new table and the second new table.

[0142] In one embodiment, after the processor determines whether the first data is greater than or equal to the second tuple number, the processor further comprises: if not, sequentially obtaining the third element group arranged after the second tuple number in the descending order until the last number in the descending order; determining whether the first data is greater than or equal to the last number; if not, adding a third new table after the last number in the descending order; filling the third tuple number into the third new table, and completing the entry of all data in the first linked list according to the adding process of the first new table, the second new table and the third new table, and the data filling process in the first new table, the second new table and the third new table.

[0143] In one embodiment, after the processor completes the step of entering all data in the first linked list, it includes: determining whether the linked list length of the first linked list after adding all data is greater than a first preset value; if so, removing a specified number of end tables arranged in the first linked list, and deleting the number of tuples filled in the specified number of end tables, so that the linked list length of the first linked list remains less than or equal to the first preset value.

[0144] In one embodiment, before the above-mentioned processor filters the first data that meets the size requirement from the source data and stores them in a preset linked list in sequence according to a preset order to form a first linked list, it includes: judging whether the number of tuples in the first linked list is greater than a second preset amount; if so, terminating the filtering of data from the source data and terminating the filling of data into the first linked list.

[0145] In one embodiment, the processor selects two first linked lists with the same data attributes from a plurality of first linked lists and compares the size of the header data, stores the larger header data in the second linked list, and deletes the larger header data from the original first linked list, so that the data sorted after the larger header data becomes the new header data, and then compares the header data until all the data meeting the requirements in the two first linked lists with the same data attributes are extracted to the second linked list, and according to the data comparison and extraction process of the two first linked lists, the step of extracting all the data with the same data attributes to the second linked list includes: comparing the number of header tuples of the first designated linked list with the number of header tuples of the second designated linked list, wherein the first designated linked list and the second designated linked list are two linked lists with the same data attributes and sorting rules among all the first linked lists, and the arrangement rule includes descending order; if the number of header tuples of the first designated linked list is greater than or equal to the number of header tuples of the second designated linked list, the number of header tuples of the first designated linked list is extracted to the second linked list, and the number of header tuples of the first designated linked list is removed, and the first designated linked list with the first designated linked list is extracted to the second linked list. The number of tuples that are adjacent to the head tuple number of the first specified linked list and are located after the head tuple number of the first specified linked list in descending order is changed to the new head tuple number; the new head tuple number of the first specified linked list is compared with the head tuple number of the second specified linked list; if the new head tuple number of the first specified linked list is less than the head tuple number of the second specified linked list, the head tuple number of the second specified linked list is extracted to the second specified linked list and arranged after the head tuple number of the first specified linked list to form a descending arrangement, and the head tuple number of the second specified linked list is removed, and the head tuple number of the second specified linked list is compared with the head tuple number of the second specified linked list. The number of tuples whose head tuple numbers of the second designated linked list are adjacent and located after the number of head tuples of the second designated linked list in descending order are changed to the new number of head tuples; according to the comparison process of the number of head tuples of the first designated linked list and the number of head tuples of the second designated linked list, and the comparison process of the new number of head tuples of the first designated linked list and the number of head tuples of the second designated linked list, the remaining number of tuples in the first designated linked list and the second designated linked list continue to be compared until the number of data in the second designated linked list reaches a preset condition, and the entry of all data in the second linked list is completed.

[0146] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0147] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.

[0148] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for screening information, characterized in that: include: Filtering first data that meets the size requirement from the source data, and storing them in a preset linked list in sequence according to a preset sort order to form a first linked list, wherein the first data is any data in the source data; Select two first linked lists with the same data attribute from the plurality of first linked lists and compare the size of the header data, store the larger header data into the second linked list, delete the larger header data from the original first linked list, make the data sorted after the larger header data become the new header data, and then compare the header data until all the data meeting the requirements in the two first linked lists with the same data attribute are extracted into the second linked list, and according to the data comparison and extraction process of the two first linked lists, all the data with the same data attribute are extracted into the second linked list; Acquire a TOP value corresponding to a specified task from the second linked list, wherein the specified task belongs to any task in the task list, and the TOP value corresponding to the specified task is associated with corresponding associated information; Lock and display the TOP value corresponding to the specified task, and the associated information corresponding to the TOP value corresponding to the specified task; The first linked list includes a linked list structure and the number of tuples filled in the linked list according to a preset sorting, the preset sorting is in descending order, and the step of selecting first data that meets the size requirement from the source data and storing them in the preset linked list in sequence according to the preset sorting to form the first linked list includes: Get the first tuple number located at the start table of the first linked list; Determine whether the first data is greater than or equal to the first number of tuples; If yes, then adding a first new table before the descending sort of the starting table, so that the first new table becomes the starting table; The first data is filled into the first new table to make the first data a new first tuple number, and the entry of all data in the first linked list is completed according to the adding process of the first new table and the data filling process in the first new table.

2. The method for filtering information according to claim 1, characterized in that: After the step of determining whether the first data is greater than or equal to the first number of tuples, the method further comprises: If not, obtaining the number of the second tuple adjacent to the start table and after the start table is sorted; Determine whether the first data is greater than or equal to the second tuple number; If yes, then add a second new table before the second tuple number; The first data is filled into the second new table, and according to the adding process of the first new table and the second new table, and the data filling process in the first new table and the second new table, the entry of all data in the first linked table is completed.

3. The method for filtering information according to claim 2, characterized in that: After the step of determining whether the first data is greater than or equal to the second tuple number, the method further comprises: If not, sequentially obtain the third tuple number arranged after the second tuple number in the descending order until the last number in the descending order; Determine whether the first data is greater than or equal to the last digit; If not, then add a third new table after the last number in the descending order; The third tuple number is filled into the third new table, and according to the adding process of the first new table, the second new table and the third new table, and the data filling process in the first new table, the second new table and the third new table, the entry of all data in the first linked table is completed.

4. The method for filtering information according to any one of claims 2 to 3, characterized in that: After completing the step of entering all the data in the first linked list, the method further comprises: Determine whether the length of the first linked list after all data are added is greater than a first preset value; If so, a specified number of tail tables arranged in the first linked list are removed, and the number of tuples filled in the specified number of tail tables is deleted, so that the linked list length of the first linked list is maintained less than or equal to the first preset amount.

5. The method for filtering information according to claim 1, characterized in that: Before the step of selecting first data that meets the size requirement from the source data and sequentially storing the data in a preset linked list according to a preset sort order to form a first linked list, the method includes: Determine whether the number of tuples in the first linked list is greater than a second preset amount; If so, stop filtering data from the source data and stop filling data into the first linked list.

6. The method for filtering information according to claim 1, characterized in that: The step of selecting two first linked lists with the same data attributes from the plurality of first linked lists and comparing the sizes of the header data, storing the larger header data in the second linked list, deleting the larger header data from the original first linked list, making the data sorted after the larger header data become the new header data, and then comparing the header data until all the data meeting the requirements in the two first linked lists with the same data attributes are extracted into the second linked list, and extracting all the data with the same data attributes into the second linked list according to the data comparison and extraction process of the two first linked lists, includes: Compare the number of head tuples of the first specified linked list with the number of head tuples of the second specified linked list, wherein the first specified linked list and the second specified linked list are two linked lists with the same data attributes and sorting rules among all the first linked lists, and the sorting rules include descending order; If the number of head tuples of the first specified linked list is greater than or equal to the number of head tuples of the second specified linked list, the number of head tuples of the first specified linked list is extracted to the second linked list, and the number of head tuples of the first specified linked list is removed, and the number of tuples adjacent to the number of head tuples of the first specified linked list and located after the number of head tuples of the first specified linked list in descending order is changed to the new number of head tuples; Compare the number of new head tuples of the first specified linked list with the number of head tuples of the second specified linked list; If the number of new head tuples of the first designated linked list is less than the number of head tuples of the second designated linked list, the number of head tuples of the second designated linked list is extracted to the second designated linked list and arranged after the number of head tuples of the first designated linked list to form a descending arrangement, and the number of head tuples of the second designated linked list is removed, and the number of tuples adjacent to the number of head tuples of the second designated linked list and located after the number of head tuples of the second designated linked list in descending order is changed to the number of new head tuples; According to the comparison process of the number of head tuples of the first designated linked list and the number of head tuples of the second designated linked list, and the comparison process of the new number of head tuples of the first designated linked list and the number of head tuples of the second designated linked list, continue to compare the remaining number of tuples in the first designated linked list and the second designated linked list until the number of data in the second linked list reaches the preset condition, and complete the entry of all data in the second linked list.

7. A device for screening information, used to execute the method according to any one of claims 1 to 6, characterized in that: include: A screening module, used for screening first data that meets the size requirement from the source data, and storing the first data in a preset linked list in sequence according to a preset sorting order to form a first linked list, wherein the first data is any data in the source data; a comparison module, configured to select two first linked lists with the same data attribute from the plurality of first linked lists and compare the size of the header data, store the larger header data in the second linked list, delete the larger header data from the original first linked list, make the data sorted after the larger header data become the new header data, and then compare the header data until all the data meeting the requirements in the two first linked lists with the same data attribute are extracted into the second linked list, and according to the data comparison and extraction process of the two first linked lists, all the data with the same data attribute are extracted into the second linked list; an acquisition module, configured to acquire a TOP value corresponding to a specified task from the second linked list, wherein the specified task belongs to any task in the task list, and the TOP value corresponding to the specified task is associated with corresponding associated information; The locking module is used to lock and display the TOP value corresponding to the specified task and the associated information corresponding to the TOP value corresponding to the specified task.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.