Data sorting method, data sorting device, medium and electronic equipment

By comparing the URL domain name grouping, hash value and unified resource locator, the problem of low URL comparison efficiency is solved, resource sorting efficiency is improved, and recommendation information can be determined based on URL ranking.

CN110362763BActive Publication Date: 2024-11-22TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN201910642579.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-07-16
Publication Date
2024-11-22
Estimated Expiration
2039-07-16

AI Technical Summary

Technical Problem

In the prior art, the comparison efficiency of URLs is low, resulting in low resource sorting efficiency.

Method used

By grouping the domain names of the data to be grouped, data with the same hash value and the same unified resource locator are aggregated, and the data is sorted according to the number of visits or traffic according to the aggregation results, using the hash value last character and the unified resource locator last character. Compare to improve comparison efficiency.

Benefits of technology

It improves the efficiency of URL comparison, improves the efficiency of resource sorting, and can determine recommended information based on the URL rankings under different domain names.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110362763B_ABST
    Figure CN110362763B_ABST
Patent Text Reader

Abstract

The present disclosure provides a data sorting method, a data sorting device, a computer-readable storage medium and an electronic device, and relates to the field of computer technology. The data sorting method comprises: grouping the data to be grouped according to the domain name of each data to be grouped, and obtaining multiple groups of data to be sorted; aggregating the data to be sorted with the same hash value and the same uniform resource locator in each group; and sorting the data to be sorted in each group according to the aggregation result. The data sorting method in the present disclosure can overcome the problem of low URL comparison efficiency to a certain extent, improve the URL comparison efficiency, and thus improve the resource sorting efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data sorting method, a data sorting device, a computer-readable storage medium, and an electronic device. Background Art

[0002] Uniform Resource Locator (URL) is a concise representation of the location and access method of resources available on the Internet. It is the address of standard resources on the Internet. Every file on the Internet has a unique URL, which contains information indicating the location of the file and how the browser handles the file.

[0003] At present, the number of visits to different resources can be determined by analyzing the access logs, so as to recommend popular resources to users according to the ranking of the number of visits. Generally, the way to rank different resources is to compare all URLs in the access logs, and then determine the number of visits to the URLs according to the number of identical URLs, and then determine the number of visits to different resources, so as to rank different resources according to the number of visits.

[0004] However, URLs are generally long and contain many characters, and it usually takes a long time to compare URLs, which in turn leads to low URL comparison efficiency.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a data sorting method, a data sorting device, a computer-readable storage medium and an electronic device, thereby overcoming the problem of low URL matching efficiency to a certain extent, improving URL matching efficiency, and further improving resource sorting efficiency.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to a first aspect of the present disclosure, a data sorting method is provided, comprising:

[0009] The data to be grouped are grouped according to the domain names of the data to be grouped, so as to obtain multiple groups of data to be sorted;

[0010] Aggregate the data to be sorted that have the same hash value and the same uniform resource locator in each group;

[0011] Sort the data to be sorted in each group according to the aggregation results.

[0012] In an exemplary embodiment of the present disclosure, the data sorting method further includes:

[0013] Get data access logs;

[0014] Convert data access logs into data to be grouped.

[0015] In an exemplary embodiment of the present disclosure, the data sorting method further includes:

[0016] Starting from the last character of the hash value, the hash values ​​of the data to be sorted in each group are compared by a preset number of bits to determine the data to be sorted with the same hash value in each group.

[0017] In an exemplary embodiment of the present disclosure, the data to be sorted with the same hash value and the same uniform resource locator in each group are aggregated, including:

[0018] Starting from the last character of the uniform resource locator, the uniform resource locators of the data to be sorted with the same hash value are compared;

[0019] Aggregate the data to be sorted with the same hash value and the same uniform resource locator in each group.

[0020] In an exemplary embodiment of the present disclosure, sorting the data to be sorted in each group according to the aggregation result includes:

[0021] Determine the flow rate corresponding to the data to be sorted in each group based on the aggregation results;

[0022] Sort the data to be sorted in each group in descending order of traffic volume.

[0023] In an exemplary embodiment of the present disclosure, sorting the data to be sorted in each group according to the aggregation result includes:

[0024] Determine the number of accesses corresponding to the data to be sorted in each group according to the aggregation results;

[0025] Sort the data to be sorted in each group in descending order of the number of accesses.

[0026] In an exemplary embodiment of the present disclosure, the data sorting method further includes:

[0027] The pre-set data to be sorted in each group of sorting results are stored, so as to determine the recommended information according to the pre-set data to be sorted.

[0028] According to a second aspect of the present disclosure, a data sorting device is provided, comprising a data grouping unit, a data aggregation unit and a data sorting unit, wherein:

[0029] A data grouping unit, used for grouping the data to be grouped according to the domain names of the data to be grouped, to obtain multiple groups of data to be sorted;

[0030] A data aggregation unit, used for aggregating the to-be-sorted data with the same hash value and the same uniform resource locator in the group;

[0031] The data sorting unit is used to sort the data to be sorted in each group according to the aggregation result.

[0032] In an exemplary embodiment of the present disclosure, the data sorting device further includes a log acquisition unit and a data conversion unit, wherein:

[0033] A log acquisition unit, used for acquiring data access logs;

[0034] The data conversion unit is used to convert the data access log into data to be grouped.

[0035] In an exemplary embodiment of the present disclosure, the data sorting device further includes a hash value comparison unit, wherein:

[0036] The hash value comparison unit is used to compare the hash values ​​of the data to be sorted in each group by a preset number of digits starting from the last character of the hash value, so as to determine the data to be sorted with the same hash value in each group.

[0037] In an exemplary embodiment of the present disclosure, the data aggregation unit aggregates the to-be-sorted data with the same hash value and the same uniform resource locator in each group in a specific manner as follows:

[0038] The data aggregation unit starts from the last character of the uniform resource locator, and performs uniform resource locator comparison on the data to be sorted with the same hash value;

[0039] The data aggregation unit aggregates the data to be sorted with the same hash value and the same uniform resource locator in each group.

[0040] In an exemplary embodiment of the present disclosure, the data sorting unit sorts the data to be sorted in each group according to the aggregation result in the following manner:

[0041] The data sorting unit determines the flow corresponding to the data to be sorted in each group according to the aggregation result;

[0042] The data sorting unit sorts the data to be sorted in each group in the order of flow from large to small.

[0043] In an exemplary embodiment of the present disclosure, the data sorting unit sorts the data to be sorted in each group according to the aggregation result in the following manner:

[0044] The data sorting unit determines the number of accesses corresponding to the data to be sorted in each group according to the aggregation result;

[0045] The data sorting unit sorts the data to be sorted in each group in descending order of the number of accesses.

[0046] In an exemplary embodiment of the present disclosure, the data sorting device further includes a data storage unit, wherein:

[0047] The data storage unit is used to store the pre-set data to be sorted in each group of sorting results, so as to determine the recommended information according to the pre-set data to be sorted.

[0048] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned methods by executing the executable instructions.

[0049] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, any one of the methods described above is implemented.

[0050] The exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

[0051] In a data sorting method provided in an example implementation of the present disclosure, the data to be grouped can be grouped according to the domain name of each data to be grouped, and multiple groups of data to be sorted can be obtained to improve the comparison efficiency of URLs; then, the data to be sorted with the same hash value and the same uniform resource locator in each group are aggregated, and the data to be sorted in each group is sorted according to the number of visits or traffic determined in the aggregation result. According to the above scheme description, on the one hand, the present disclosure can overcome the problem of low URL comparison efficiency to a certain extent, and first classifying URLs according to domain names can improve the comparison efficiency of URLs, thereby improving the sorting efficiency of resources; on the other hand, it can determine the URL sorting under different domain names, so as to determine the recommended information according to the URL rankings corresponding to different domain names.

[0052] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0054] Figure 1 A schematic diagram showing an exemplary system architecture of a data sorting method and a data sorting device to which the embodiments of the present disclosure can be applied;

[0055] Figure 2 A schematic diagram showing the structure of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure is shown;

[0056] Figure 3 A flowchart of a data sorting method according to an embodiment of the present disclosure is schematically shown;

[0057] Figure 4 A flowchart of a process for determining data to be grouped according to an embodiment of the present disclosure is schematically shown;

[0058] Figure 5 The following schematically shows an architecture diagram of a data sorting method according to an embodiment of the present disclosure;

[0059] Figure 6 A flowchart schematically shows another data sorting method according to an embodiment of the present disclosure;

[0060] Figure 7 The structure block diagram of a data sorting device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0061] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0062] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0063] Figure 1 A schematic diagram of a system architecture of an exemplary application environment in which a data sorting method and a data sorting device according to an embodiment of the present disclosure can be applied is shown.

[0064] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with display screens, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0065] The data sorting method provided in the embodiment of the present disclosure is generally executed by the server 105, and accordingly, the data sorting device is generally arranged in the server 105. However, it is easy for those skilled in the art to understand that the data sorting method provided in the embodiment of the present disclosure can also be executed by the terminal devices 101, 102, 103, and accordingly, the data sorting device can also be arranged in the terminal devices 101, 102, 103, and this is not particularly limited in the present exemplary embodiment. For example, in an exemplary embodiment, the server 105 can group the data to be grouped according to the domain name of each data to be grouped, obtain multiple groups of data to be sorted, and aggregate the data to be sorted with the same hash value and the same uniform resource locator in each group, and then sort the data to be sorted in each group according to the aggregation result.

[0066] Figure 2 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure is shown.

[0067] It should be noted that Figure 2The computer system 200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0068] like Figure 2 As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage part 208 into a random access memory (RAM) 203. Various programs and data required for system operation are also stored in the RAM 203. The CPU 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0069] The following components are connected to the I / O interface 205: an input section 206 including a keyboard, a mouse, etc.; an output section 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that a computer program read therefrom is installed into the storage section 208 as needed.

[0070] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 209, and / or installed from the removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, the various functions defined in the method and apparatus of the present application are executed. In some embodiments, the computer system 200 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0071] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0072] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0073] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0074] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being installed in the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiment. For example, the electronic device may implement the following method. Figure 3 , Figure 4 as well as Figure 6 The various steps shown, etc.

[0075] The technical solution of the embodiment of the present disclosure is described in detail below:

[0076] In the prior art, for the URL sorting method in the SPARK big data computing engine, the SPARK native method is usually used to compare and sort URLs. For longer URLs, the comparison operation consumes more CPU performance. In the comparison method of the existing solution, the hash value of the URL needs to be compared. If the hash values ​​of the two URLs are equal, the characters in the URL are further compared. If the characters in the URLs are equal, the two URLs are aggregated. It can be understood that the two URLs correspond to the same resource, that is, the resource has been accessed twice.

[0077] However, URLs are generally long and contain many characters, and it usually takes a long time to compare URLs, which in turn leads to the problem of low efficiency in sorting resources corresponding to the URLs.

[0078] Based on one or more of the above problems, this exemplary embodiment provides a data sorting method. The data sorting method can be applied to the above server 105, and can also be applied to one or more of the above terminal devices 101, 102, 103, which is not particularly limited in this exemplary embodiment. Figure 3 As shown, the data sorting method may include the following steps S310 to S330:

[0079] Step S310: grouping the data to be grouped according to the domain names of the data to be grouped to obtain multiple groups of data to be sorted.

[0080] Step S320: Aggregate the data to be sorted with the same hash value and the same uniform resource locator in each group.

[0081] Step S330: Sort the data to be sorted in each group according to the aggregation result.

[0082] In a data sorting method provided in an example implementation of the present disclosure, the data to be grouped can be grouped according to the domain name of each data to be grouped, and multiple groups of data to be sorted can be obtained to improve the comparison efficiency of URLs; then, the data to be sorted with the same hash value and the same uniform resource locator in each group are aggregated, and the data to be sorted in each group is sorted according to the number of visits or traffic determined in the aggregation result. According to the above scheme description, on the one hand, the present disclosure can overcome the problem of low URL comparison efficiency to a certain extent, and first classifying URLs according to domain names can improve the comparison efficiency of URLs, thereby improving the sorting efficiency of resources; on the other hand, it can determine the URL sorting under different domain names, so as to determine the recommended information according to the URL rankings corresponding to different domain names.

[0083] Next, the above steps of this exemplary embodiment are described in more detail.

[0084] In step S310, the data to be grouped are grouped according to the domain names of the data to be grouped, so as to obtain multiple groups of data to be sorted.

[0085] In this example implementation, the data to be grouped may be Resilient Distributed Datasets (RDDs), which is an abstract concept of distributed memory and provides a restricted shared memory model, i.e., RDD is a set of read-only record partitions. RDD can be regarded as a Spark object that runs in memory; Spark is an open source cluster computing environment similar to Hadoop.

[0086] In this example implementation, a domain name is the name of a computer or computer group on the Internet consisting of a string of names separated by dots, and is used to identify the electronic location or geographical location of the computer during data transmission.

[0087] In this example implementation, the data to be sorted and the data to be grouped are the same data. The data to be sorted is used to indicate that the current state of the data is the state to be sorted; the data to be grouped is used to indicate that the current state of the data is the state to be grouped. In multiple groups of data to be sorted, the domain name of the data to be sorted in each group is the same.

[0088] In this example implementation, optionally, see Figure 4 , Figure 4 A flowchart of a process for determining data to be grouped according to an embodiment of the present disclosure is schematically shown. Figure 4 As shown, the data sorting method may further include step S410 and step S420, wherein:

[0089] Step S410: Obtain data access log;

[0090] Step S420: Convert the data access log into data to be grouped.

[0091] In this example implementation, the data access log is a log collected from multiple CDN servers, and the log includes the URL of the resources that the user has visited. CDN (Content Delivery Network) is a content distribution network. CDN is a content distribution network built on the Internet. It relies on edge servers deployed in various places, and through the load balancing, content distribution and scheduling functional modules of the central platform, it enables users to obtain the required content nearby, thereby reducing network congestion and improving user access response speed and hit rate. The data format of the data access log and the data to be grouped are different, but the data content is the same.

[0092] visible, Figure 4 The flowchart of the process of determining the data to be grouped shown can convert the data access logs into the data to be grouped, so that the data access logs are suitable for the Spark computing cluster so as to group and aggregate them.

[0093] In step S320, the data to be sorted with the same hash value and the same uniform resource locator in each group are aggregated.

[0094] In this example implementation, each data to be sorted corresponds to a hashcode function and an equals function; wherein, the hashcode function is a hash function, and the hashcode function makes each data to be sorted correspond to a hash value. If the hash values ​​of two data to be sorted are the same, then it can be determined that the two data to be sorted are the same data, that is, the data has been accessed twice; the equals function is used to compare the characters of the URL of the data to be sorted.

[0095] In this example implementation, aggregating the data to be sorted with the same hash value and the same uniform resource locator in each group can be understood as aggregating the data with the same hash value and the same uniform resource locator, so that the number of times the same data has been accessed can be determined based on the amount of aggregated data.

[0096] In this example implementation, optionally, the data sorting method may further include:

[0097] Starting from the last character of the hash value, the hash values ​​of the data to be sorted in each group are compared by a preset number of bits to determine the data to be sorted with the same hash value in each group.

[0098] Further optionally, the data to be sorted with the same hash value and the same uniform resource locator in each group are aggregated, including:

[0099] Starting from the last character of the uniform resource locator, the uniform resource locators of the data to be sorted with the same hash value are compared;

[0100] Aggregate the data to be sorted with the same hash value and the same uniform resource locator in each group.

[0101] In this example implementation, the hash values ​​of a preset number of digits (e.g., 10 digits) are compared starting from the last character. As long as the hash values ​​of the preset number of digits starting from the last character are the same, the hash values ​​of the data to be sorted are deemed to be the same. Compared with the traditional method of starting from the first character and comparing each character of the hash value, this optional implementation can reduce the occupancy of CPU resources, reduce power consumption, and improve comparison efficiency. Similarly, by comparing the uniform resource locator starting from the last character of the uniform resource locator, it is possible to stop detecting and determine that the uniform resource locators of the data to be sorted are different when different characters are detected. Since the probability that the first n digits of the uniform resource locator are similar is higher than the probability that the last n digits are similar, comparing the uniform resource locator starting from the last character of the uniform resource locator can more quickly determine whether the uniform resource locators are the same, thereby improving the comparison efficiency of the URL.

[0102] Please refer to the code in the table, which is used to implement the method for comparing the hash values ​​and uniform resource locators of the data to be sorted in each group in this optional embodiment. It includes an algorithm process of comparing the hash values ​​of the data to be sorted in each group by a preset number of digits starting from the last character of the hash value to determine the data to be sorted with the same hash value in each group, and an algorithm process of comparing the uniform resource locators of the data to be sorted with the same hash value starting from the last character of the uniform resource locator.

[0103]

[0104]

[0105]

[0106]

[0107] In actual applications, the above code can be used to analyze 4 billion lines of logs in 170 seconds. Compared with the traditional comparison method that takes 200 seconds, the implementation method of the present disclosure can reduce the overall time by 15%, thereby improving the efficiency of resource sorting.

[0108] In this example implementation, optionally, a method of aggregating the to-be-sorted data with the same hash value and the same uniform resource locator in each group may specifically be:

[0109] The hash value is mapped to the hash value to be compared according to the mapping relationship; wherein the number of character strings of the hash value to be compared is less than the number of character strings of the hash value. This can reduce the amount of calculation in the data comparison process and improve the comparison efficiency.

[0110] It can be seen that the implementation of this optional implementation can improve the comparison efficiency by first comparing the hash value and then comparing the URL characters, and by combining the comparison starting from the last character, thereby improving the resource sorting efficiency.

[0111] In step S330, the data to be sorted in each group is sorted according to the aggregation result.

[0112] In this example implementation, the aggregation result includes a request field and a flux field; wherein the request field is used to indicate the number of times a resource is accessed, the flux field is used to indicate the traffic corresponding to the resource, and the traffic is used to describe the number of users accessing the resource and the number of web pages accessed by the users.

[0113] visible, Figure 3 The data sorting method shown can improve the comparison efficiency by first comparing the hash value and then comparing the URL characters, combined with the method of starting the comparison from the last character, thereby improving the resource sorting efficiency; and, based on the pre-set data to be sorted, determining the recommended information to recommend popular resources to users, so that users can understand the latest information and improve the user experience.

[0114] In this example implementation, optionally, sorting the data to be sorted in each group according to the aggregation result includes:

[0115] Determine the flow rate corresponding to the data to be sorted in each group based on the aggregation results;

[0116] Sort the data to be sorted in each group in descending order of traffic volume.

[0117] In this example implementation, optionally, sorting the data to be sorted in each group in order of traffic from large to small can be understood as: comparing the flux field of the data to be sorted in each group used to represent the traffic corresponding to the resource, and then sorting the data to be sorted in each group in order of traffic from large to small.

[0118] It can be seen that by implementing this optional implementation, the data to be sorted in each group can be sorted according to the flux field to determine the resources that can attract more traffic, and then determine the corresponding resource promotion strategy based on these resources.

[0119] In this example implementation, optionally, sorting the data to be sorted in each group according to the aggregation result includes:

[0120] Determine the number of accesses corresponding to the data to be sorted in each group according to the aggregation results;

[0121] Sort the data to be sorted in each group in descending order of the number of accesses.

[0122] In this example implementation, optionally, sorting the data to be sorted in each group in order of traffic from large to small can be understood as: comparing the request field of the data to be sorted in each group for indicating the number of times a resource is accessed, and then sorting the data to be sorted in each group in order of the number of times the access is made from large to small.

[0123] It can be seen that by implementing this optional implementation, the data to be sorted in each group can be sorted according to the request field to determine the resources with more access times, and the resources with fewer access times can be improved to increase the access times of these resources.

[0124] In this example implementation, optionally, the data sorting method may further include:

[0125] The pre-set data to be sorted in each group of sorting results are stored, so as to determine the recommended information according to the pre-set data to be sorted.

[0126] In this example implementation, optionally, storing the pre-positioned data to be sorted in each group of sorting results includes:

[0127] Determine the data to be sorted that are preset in front of each group of sorting results (for example, determine the first 10 data to be sorted in each group); and store the data to be sorted.

[0128] It can be seen that by implementing this optional implementation, it is possible to determine recommendation information based on the pre-set data to be sorted, so as to recommend popular resources to users, so that users can learn the latest information and improve their usage experience.

[0129] See also Figure 5 , Figure 5 The following schematically shows an architecture diagram of a data sorting method according to an embodiment of the present disclosure. Figure 5 As shown, the architecture diagram includes a CDN server 501, a computing cluster 502, and a CDN user 503; wherein the number of CDN servers 501 can be multiple, and Figure 5Four are shown, but this does not mean that only four CDN servers 501 are involved in this application; in addition, the computing cluster 502 can be a Spark computing cluster, which can be composed of 12 servers, each server is configured with 192G memory, 80-core CPU, 10G network card, 2TB SSD hard disk; and, all 12 servers are installed with hadoop program and Spark program, forming a storage cluster based on HDFS and a computing cluster of Spark. Specifically, the data access log can be obtained in the CDN501 server so that the computing cluster 502 analyzes the data access log (including executing steps S310 to S330), and then, the pre-set data to be sorted is determined, so that the CDN user 503 can determine the recommended information based on the pre-set data to be sorted.

[0130] It can be seen that based on Figure 5 The architecture diagram shown implements the data sorting method disclosed in the present invention, which can improve the comparison efficiency by first comparing the hash value and then comparing the URL characters in combination with a method of starting the comparison from the last character, thereby improving the resource sorting efficiency; and determining the recommended information based on the pre-set data to be sorted to recommend popular resources to users, so that users can understand the latest information and improve the user experience.

[0131] See also Figure 6 , Figure 6 A flowchart of another data sorting method according to an embodiment of the present disclosure is schematically shown. Figure 6 As shown, another data sorting method includes steps S610, S620, S630, S640, S650 and S660, wherein:

[0132] Step S610: Obtain data access log.

[0133] Step S620: Convert the data access log into data to be grouped.

[0134] Step S630: grouping the data to be grouped according to the domain names of the data to be grouped to obtain multiple groups of data to be sorted.

[0135] Step S640: Aggregate the data to be sorted with the same hash value and the same uniform resource locator in each group.

[0136] Step S650: Sort the data to be sorted in each group according to the number of visits or traffic determined in the aggregation result.

[0137] Step S660: storing the pre-positioned data to be sorted in each group of sorting results.

[0138] Specifically, the data access logs can be obtained and converted into data to be grouped, and the data access logs can be converted into a format suitable for a computing cluster; then, the data to be grouped are grouped according to the domain names of each data to be grouped to obtain multiple groups of data to be sorted, so as to improve the comparison efficiency of URLs; then, the data to be sorted with the same hash value and the same uniform resource locator in each group are aggregated, and the data to be sorted in each group is sorted according to the number of visits or traffic determined in the aggregation result; then, the data to be sorted that are pre-set in each group of sorting results are stored to determine the recommended information based on the pre-set data to be sorted.

[0139] It can be seen that implementation Figure 6 The data sorting method shown can improve the comparison efficiency by first comparing the hash value and then comparing the URL characters, combined with the method of starting the comparison from the last character, thereby improving the resource sorting efficiency; and, based on the pre-set data to be sorted, determining the recommended information to recommend popular resources to users, so that users can understand the latest information and improve the user experience.

[0140] Furthermore, in this exemplary embodiment, a data sorting device is also provided. The data sorting device can be applied to a server or a terminal device. Figure 7 As shown, the data sorting device may include: a data grouping unit 701, a data aggregation unit 702 and a data sorting unit 703, wherein:

[0141] A data grouping unit 701 is used to group the data to be grouped according to the domain name of each data to be grouped, so as to obtain multiple groups of data to be sorted;

[0142] A data aggregation unit 702, configured to aggregate the to-be-sorted data with the same hash value and the same uniform resource locator within a group;

[0143] The data sorting unit 703 is used to sort the data to be sorted in each group according to the aggregation result.

[0144] It can be seen that implementation Figure 7 The data sorting device shown can improve the comparison efficiency by first comparing the hash value and then comparing the URL characters, combined with a method of starting the comparison from the last character, thereby improving the resource sorting efficiency; and, based on the pre-set data to be sorted, determining the recommended information to recommend popular resources to users, so that users can understand the latest information and improve the user experience.

[0145] In an exemplary embodiment of the present disclosure, the data sorting device further includes a log acquisition unit (not shown) and a data conversion unit (not shown), wherein:

[0146] A log acquisition unit, used for acquiring data access logs;

[0147] The data conversion unit is used to convert the data access log into data to be grouped.

[0148] It can be seen that by implementing this exemplary embodiment, the data access logs can be converted into data to be grouped, so that the data access logs are suitable for the Spark computing cluster, so as to group and aggregate them.

[0149] In an exemplary embodiment of the present disclosure, the data sorting device further includes a hash value comparison unit (not shown), wherein:

[0150] The hash value comparison unit is used to compare the hash values ​​of the data to be sorted in each group by a preset number of digits starting from the last character of the hash value, so as to determine the data to be sorted with the same hash value in each group.

[0151] Furthermore, the data aggregation unit 702 aggregates the to-be-sorted data with the same hash value and the same uniform resource locator in each group in a specific manner as follows:

[0152] The data aggregation unit 702 starts from the last character of the uniform resource locator and compares the uniform resource locators of the data to be sorted with the same hash value;

[0153] The data aggregation unit 702 aggregates the to-be-sorted data with the same hash value and the same uniform resource locator in each group.

[0154] It can be seen that the implementation of this exemplary embodiment can improve the comparison efficiency and thus improve the resource sorting efficiency by first comparing the hash value and then comparing the URL characters, and combining the method of starting the comparison from the last character.

[0155] In an exemplary embodiment of the present disclosure, the data sorting unit 703 sorts the data to be sorted in each group according to the aggregation result in the following manner:

[0156] The data sorting unit 703 determines the flow corresponding to the data to be sorted in each group according to the aggregation result;

[0157] The data sorting unit 703 sorts the data to be sorted in each group in the order of flow rate from large to small.

[0158] It can be seen that by implementing this exemplary embodiment, the data to be sorted in each group can be sorted according to the flux field to determine the resources that can attract more traffic, and then determine the corresponding resource promotion strategy based on these resources.

[0159] In an exemplary embodiment of the present disclosure, the data sorting unit 703 sorts the data to be sorted in each group according to the aggregation result in the following manner:

[0160] The data sorting unit 703 determines the number of accesses corresponding to the data to be sorted in each group according to the aggregation result;

[0161] The data sorting unit 703 sorts the data to be sorted in each group in descending order of the number of accesses.

[0162] It can be seen that by implementing this exemplary embodiment, the data to be sorted in each group can be sorted according to the request field to determine the resources with more access times, and then improve the resources with less access times to increase the access times of these resources.

[0163] In an exemplary embodiment of the present disclosure, the data sorting device further includes a data storage unit (not shown), wherein:

[0164] The data storage unit is used to store the pre-set data to be sorted in each group of sorting results, so as to determine the recommended information according to the pre-set data to be sorted.

[0165] It can be seen that by implementing this exemplary embodiment, it is possible to determine recommendation information based on the pre-set data to be sorted, so as to recommend popular resources to users, so that users can learn the latest information and improve their usage experience.

[0166] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0167] Since the functional modules of the data sorting device of the example embodiment of the present disclosure correspond to the steps of the example embodiment of the data sorting method described above, please refer to the embodiment of the data sorting method described above for details not disclosed in the device embodiment of the present disclosure.

[0168] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0169] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data sorting method, characterized in that: include: Acquire data access logs from multiple content distribution network servers, and convert the data access logs into data to be grouped; Grouping the data to be grouped according to the domain names of the data to be grouped to obtain multiple groups of data to be sorted; Aggregate the data to be sorted that have the same hash value and the same uniform resource locator in each group; Sorting the data to be sorted in each group according to the aggregation result; the aggregation result includes the flow and access times of the data to be sorted in each group; The method of aggregating the data to be sorted in each group with the same hash value and the same uniform resource locator includes: if the character length of any uniform resource locator is greater than a set length threshold, starting from the last character of any uniform resource locator, extracting characters whose length matches the set length threshold, and calculating the hash value of any uniform resource locator based on the extracted characters; starting from the last character of the hash value, performing a preset bit comparison on the hash values ​​of the data to be sorted in each group, and if the characters of the preset bit starting from the last character of any two hash values ​​are the same, then it is determined that the any two hash values ​​are the same; starting from the last character of the uniform resource locator, performing a uniform resource locator comparison on the data to be sorted with the same hash value of the uniform resource locator, and if it is detected that the characters at the same position of any two uniform resource locators are different, then it is determined that the any two uniform resource locators are different; the data to be sorted in each group with the same hash value and the same uniform resource locator are aggregated.

2. The method according to claim 1, characterized in that: Sort the data to be sorted in each group according to the aggregation results, including: Determine the flow rate corresponding to the data to be sorted in each group according to the aggregation result; The data to be sorted in each group is sorted in descending order of flow rate.

3. The method according to claim 2, characterized in that Sort the data to be sorted in each group according to the aggregation results, including: Determine the number of accesses corresponding to the data to be sorted in each group according to the aggregation result; The data to be sorted in each group is sorted in descending order of the number of accesses.

4. The method according to claim 1, characterized in that: Also includes: The pre-set data to be sorted in each group of sorting results are stored, so as to determine the recommended information according to the pre-set data to be sorted.

5. A data sorting device, characterized in that: include: A log acquisition unit, used to acquire data access logs from a plurality of content distribution network servers; A data conversion unit, used for converting the data access log into data to be grouped; A data grouping unit, used for grouping the data to be grouped according to the domain names of the data to be grouped, to obtain multiple groups of data to be sorted; A data aggregation unit, used for aggregating the data to be sorted having the same hash value and the same uniform resource locator in each group; A data sorting unit, used to sort the data to be sorted in each group according to the aggregation result; the aggregation result includes the flow rate and access times of the data to be sorted in each group; The method of aggregating the data to be sorted in each group with the same hash value and the same uniform resource locator includes: if the character length of any uniform resource locator is greater than a set length threshold, starting from the last character of any uniform resource locator, extracting characters whose length matches the set length threshold, and calculating the hash value of any uniform resource locator based on the extracted characters; starting from the last character of the hash value, performing a preset bit comparison on the hash values ​​of the data to be sorted in each group, and if the characters of the preset bit starting from the last character of any two hash values ​​are the same, then it is determined that the any two hash values ​​are the same; starting from the last character of the uniform resource locator, performing a uniform resource locator comparison on the data to be sorted with the same hash value of the uniform resource locator, and if it is detected that the characters at the same position of any two uniform resource locators are different, then it is determined that the any two uniform resource locators are different; the data to be sorted in each group with the same hash value and the same uniform resource locator are aggregated.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

7. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method according to any one of claims 1 to 4 by executing the executable instructions.

8. A computer program product, characterized in that The invention comprises a computer program carried on a computer-readable medium, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Citation Information

Patent Citations

  • Method for searching matched character string

    CN102063510A

  • Website access statistical method and website access statistical system

    CN102571404A

  • Preferred resource selector

    US20110225447A1