Data pushing method and system, electronic device and storage medium

Abstract

Embodiments of the present application provide a data pushing method and system, electronic equipment and storage medium. The method comprises: acquiring real-time log information, performing pattern recognition on the real-time log information to obtain data file features; determining hot data based on the data file features, and storing the hot data to an edge cache; in response to a user resource access request, generating a download link based on the hot data stored in the edge cache, and returning the download link. The real-time log information is recognized by a pattern recognition method to obtain data file features, and the data file features are judged to estimate the hot data, so as to intelligently judge the hot data stored in the edge cache from the massive access data, facilitate subsequent generation of a download link based on the hot data stored in the edge cache and return of the download link, improve the hit rate of user access content, and enable the user to improve the access quality by using edge capability as much as possible.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud services, in particular to a hot data pushing method, a corresponding data pushing system, an electronic device and a computer readable storage medium. BACKGROUND

[0002] A CDN (Content Delivery Network) is an intelligent virtual network built on the basis of an existing network. It usually relies on edge servers deployed in various locations to enable users to access the required content nearby through the load balancing, content distribution, scheduling and other function modules of the central platform, thereby reducing network congestion and improving user access response speed and hit rate. The CDN is usually used for content storage and content distribution in Internet services.

[0003] In order to improve the data access speed experience, the CDN can be used to cache pictures, videos, files and other related data in edge nodes through the CDN back-to-source station mode, that is, the edge capability of the CDN is used to enable users to access data nearby to improve the service quality. However, the CDN back-to-source of a large amount of data can easily cause the phenomenon that users cannot effectively use the edge capability of the CDN to improve the access quality. SUMMARY

[0004] In view of the above problems, the present application is proposed to provide a data pushing method, a corresponding data pushing system, an electronic device and a computer readable storage medium which can overcome the above problems or at least partially solve the above problems.

[0005] The present application discloses a data pushing method, which comprises:

[0006] Obtaining real-time log information, and performing pattern recognition on the real-time log information to obtain data file features;

[0007] Determining hot data based on the data file features, and storing the hot data in an edge cache;

[0008] In response to a user resource access request, generating a download link based on the hot data stored in the edge cache, and returning the download link.

[0009] Optionally, the step of performing pattern recognition on the real-time log information to obtain data file features comprises:

[0010] Obtaining a preset multi-dimensional data model, wherein the preset multi-dimensional model is used to measure the access popularity of data with log information in combination with multiple dimensions;

[0011] The hotspot real-time calculation is performed on the real-time log information through the multi-dimensional data model, and data file features are obtained.

[0012] Optionally, the real-time log information at least includes object storage log information and content distribution log information; before the hotspot real-time calculation is performed on the real-time log information, the method further includes:

[0013] The object storage log information and the content distribution log information are respectively preprocessed.

[0014] The object storage log information and the content distribution log information are respectively preprocessed, including:

[0015] The object storage log information is subjected to an intranet request filtering operation, and a request resource deduplication processing is performed based on a preset local time;

[0016] And / or, the content distribution log is subjected to a parameter format unification operation, and a request resource deduplication processing is performed based on the preset local time.

[0017] Optionally, the data file features correspond to multiple dimensions, at least including data attribution number and data same-time download number; the hotspot real-time calculation is performed on the real-time log information through the multi-dimensional data model, and data file features are obtained, including:

[0018] The hotspot real-time calculation is performed on the object storage log information and the content distribution log information of a preset local time through the multi-dimensional data model, and data having log information within the preset local time, corresponding data attribution number and data same-time download number are obtained.

[0019] Optionally, the hotspot data is determined based on the data file features, including:

[0020] The data attribution number and the data same-time download number are taken as threshold conditions, and data having log information within a preset local time is filtered;

[0021] In response to the data attribution number and / or the data same-time download number reaching a preset threshold, hotspot data of a preset local time is determined.

[0022] Optionally, after the hotspot data is determined based on the data file features, the method further includes:

[0023] The hotspot data is configured with a heat effective period; the heat effective period is used to measure a storage time of the hotspot data in the edge cache.

[0024] Optionally, the user resource access request comprises a request for accessing data; the response to the user resource access request comprises generating a download link for the hot data stored in the edge cache and returning the download link, comprising:

[0025] obtaining a cache set of the edge cache, the cache set comprising hot data and a hotness validity period of the hot data;

[0026] in response to the cache set containing the requested access data and the hotness validity period of the requested access data not being invalid, determining that the requested access data is hot data;

[0027] generating a download link for the requested access data based on the edge cache and returning the download link.

[0028] The embodiments of the present application also disclose a data pushing system, which is configured with a log service module, a data stream processing module, a message system, a database and a computer program for execution in the data pushing system, and the computer program implements the data pushing method when executed by the data pushing system.

[0029] The embodiments of the present application also disclose an electronic device, which comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, and the computer program implements any of the data pushing methods when executed by the processor.

[0030] The embodiments of the present application also disclose a computer readable storage medium, which stores a computer program, and the computer program implements any of the data pushing methods when executed by a processor.

[0031] The embodiments of the present application have the following advantages:

[0032] In the embodiments of the present application, the obtained real-time log information can be identified by using a pattern recognition method to obtain data file features, and the data file features are judged to estimate hot data, so as to intelligently judge the hot data stored in the edge cache from the massive access data, facilitate generating a download link based on the hot data stored in the edge cache and returning the download link when a user accesses a resource, avoid non-hot data occupying edge nodes in the case that the data service downlink is large and the user behavior is relatively random, improve the hit rate of user access content, and make the user use edge capability to improve the access quality as much as possible. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a step flowchart of a data pushing method embodiment of the present application;

[0034] Figure 2 is a step flow chart of another data push method embodiment of the present application;

[0035] Figure 3 is a framework schematic diagram of a data push system provided by an embodiment of the present application;

[0036] Figure 4 is a schematic diagram of an application scenario of data push provided by an embodiment of the present application;

[0037] Figure 5 is a structural block diagram of a data push device embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0039] In the Internet service, the data downlink service is widely used. In order to improve the data access speed experience, the related data such as pictures, videos and files can be cached in the edge node through the CDN back source source station, that is, the edge capability of the CDN is used to enable the user to access the data nearby to improve the service quality. However, there will be a delay when the CDN accesses the source station for the first time, and the data pushed by the conventional CDN is based on the service model itself and focuses on the static resources commonly used, such as the static pictures loaded on the homepage. In the case of large data service downlink and random user behavior, the data related to the random behavior of the user occupies the edge node of the CDN after being back sourced by the CDN, which easily leads to the phenomenon that the user cannot effectively use the edge capability of the CDN to improve the access quality. Among them, back source refers to when the user requests to access the resource through the client, if the CDN node (i.e. edge node) does not cache the resource, or the user deploys a preheating task to the CDN node, the CDN node will source the resource from the source station.

[0040] In order to avoid the occupation of a large amount of data CDN back source, in the related technology, the calculation of the download request count is used to distinguish the data hot and cold, but the counting of the user's request behavior to obtain the download link cannot accurately reflect the actual data of the user, for example, in some scenarios, the download link can be downloaded repeatedly after being obtained, and the counting method based on the request access cannot cover this situation.

[0041] The core idea of the embodiments of the present application is to propose a scheme of estimating popular accessed data files for CDN pushing by analyzing the data access model of the user, i.e., using a multi-dimensional data model to determine the characteristics of the data files, and then reaching the intelligent judgment of the hot files that need to be pushed to the CDN. Specifically, the real-time log information obtained can be recognized to obtain the data file characteristics, and the characteristics of the data file characteristics are determined to estimate the hot data, and then the purpose of intelligently judging the hot data stored in the edge cache is achieved, so as to facilitate the subsequent user to generate a download link based on the hot data stored in the edge cache when accessing the resource, and return the download link. In the case of large data service downlink and random user behavior, non-hot data is avoided from occupying the edge node, the hit rate of user access content is improved, and the user can use the edge capability to improve the access quality as much as possible.

[0042] Referring to Figure 1 , a step flowchart of an embodiment of a data pushing method of the present application is shown, which can specifically include the following steps:

[0043] Step 101, obtaining real-time log information, and recognizing the real-time log information to obtain data file characteristics;

[0044] In the embodiments of the present application, in order to avoid the case that in the case of large data service downlink and random user behavior, the data related to the random behavior of the user occupies the edge node of the CDN after being returned to the source by the CDN, the data access log can be collected at this time, so as to perform subsequent pattern recognition based on the collected data access log.

[0045] In order to ensure the real-time of the subsequent hot data determination, the collected data access log is usually real-time log information, and in order to ensure the completeness of the access log, the log information involved in the data access can be obtained when the real-time log information is obtained for pattern recognition. Since the user download behavior will be recorded in the object storage log information and the content distribution CDN log information after the CDN hot spot is online. Therefore, the log information involved in the data access at least includes the object storage log information and the content distribution log information, which is not limited in the embodiments of the present application.

[0046] In a preferred embodiment of the present application, before the real-time log information is subjected to pattern recognition, the massive access log data can be subjected to a cleaning operation, which can specifically be manifested as pre-processing the object storage log information and the content distribution log information respectively to clean away interference data that is not needed in subsequent real-time hotspot computing. It should be noted that the specific data cleaning operation can include data deduplication operation and other operations for removing interference data, and the embodiments of the present application do not limit this.

[0047] In addition, the embodiments of the present application can involve the use of user data, and in actual application, user-specific personal data can be used in the schemes described herein in a manner that complies with the applicable laws and regulations of the country (for example, the user has given explicit consent, the user has been effectively notified, etc.) and within the scope permitted by the applicable laws and regulations.

[0048] Step 102, determining hotspot data based on the data file features, and storing the hotspot data to the edge cache;

[0049] In general, pattern recognition mainly refers to the automatic processing and interpretation of research patterns by computers using mathematical techniques. With the development of computer technology, for the research of complex information processing, an important form of information processing can be manifested as the recognition of the environment and the object by living beings, that is, for humans, focusing on the recognition of optical information (obtained through visual organs) and acoustic information (obtained through auditory organs), and for the development of computer information, it can be manifested as optical character recognition and speech recognition systems, that is, the environment and the object can be collectively referred to as patterns.

[0050] Therefore, in the embodiments of the present application, the pattern recognition performed on the collected data access log is mainly manifested as hotspot pattern recognition, and the judgment of the hotspot pattern can be determined based on the proposed data file features. At this time, the hotspot data can be determined based on the data file features, thereby achieving the purpose of intelligently judging the hotspot data stored to the edge cache from the massive access data.

[0051] Among them, the hotspot pattern refers to the phenomenon that the hotness of the access data exceeds a certain degree, and the degree required to measure the hotness can be determined based on multiple dimensions, such as dimension information related to user download behavior, including data belonging number, data same time download number, etc., and the embodiments of the present application do not limit this.

[0052] In actual application, the hotspot analysis model can be acquired, since the required degree for measuring the hotness can be determined based on multiple dimensions, the acquired hotspot analysis model can be a preset multi-dimensional data model, and the acquired preset multi-dimensional model can be used to measure the access hotness of data with log information in combination with multiple dimensions. Specifically, real-time hotspot calculation can be performed on real-time log information through the multi-dimensional data model to obtain data file features, so that hotspot analysis is performed based on the data file features to determine hotspot data.

[0053] Specifically, the data file features are extracted based on the multi-dimensional data model, and the data file features can correspond to multiple dimensions. After obtaining the data file features, the hotspot analysis model can identify hotspot data in a manner of filtering access data based on threshold conditions. For example, when certain access data exceeds a certain threshold, it can be determined that the hotness of the access data exceeds a certain degree, and the access data is hotspot data.

[0054] Step 103, in response to a user resource access request, generating a download link based on the hotspot data stored in the edge cache and returning the download link.

[0055] In an embodiment of the present application, the determined hotspot data can also be stored in the edge cache, so that when a user accesses a resource, a download link can be generated based on the hotspot data stored in the edge cache and returned. In the case of large data service downlink and random user behavior, non-hotspot data can be avoided from occupying edge nodes, recall rate can be improved as much as possible under the premise of meeting a certain CDN hit rate / accuracy, and users can use edge capabilities to improve access quality as much as possible.

[0056] In the embodiment of the present application, the acquired real-time log information can be identified to obtain data file features by using pattern recognition, and the data file features can be judged to estimate hotspot data, so as to intelligently judge the hotspot data stored in the edge cache from massive access data, facilitate subsequent users to generate a download link based on the hotspot data stored in the edge cache and return the download link when accessing a resource, avoid non-hotspot data from occupying edge nodes in the case of large data service downlink and random user behavior, improve the hit rate of user access content, and enable users to use edge capabilities to improve access quality as much as possible.

[0057] Referring to Figure 2 FIG. 6 shows a step flowchart of another data pushing method embodiment of the present application, which can specifically include the following steps:

[0058] Step 201, acquiring real-time log information, and preprocessing object storage log information and content distribution log information respectively;

[0059] In the embodiment of the present application, in order to avoid the case that the data related to the random behavior of the user occupies the edge node of the CDN after the CDN is sourced back under the condition that the data service downlink is large and the user behavior is relatively random, the log of the data access can be collected, so as to perform subsequent mode recognition based on the collected data access log.

[0060] In order to ensure the real-time of the subsequent hot data determination, the collected data access log is usually real-time log information, and in order to ensure the completeness of the access log, the log information related to the data access can be acquired when the real-time log information is acquired for mode recognition, and since the user download behavior will be recorded in the object storage log information after the CDN hot spot is online, that is, the object storage log information and the content distribution CDN log information. Therefore, the log information related to the data access can at least include the object storage log information and the content distribution, which is not limited in the embodiment of the present application.

[0061] After acquiring the real-time log information, in order to clean the interference data in the mass access log data, the object storage log information and the content distribution log information can be preprocessed respectively. In the embodiment of the present application, the interference data can refer to the interference data in the log information which is not needed for subsequent real-time hot spot calculation.

[0062] The data cleaning operation can include data deduplication operation and other operations for removing interference data.

[0063] Exemplarily, since the intranet request is not triggered by the user, and the hot data to be determined in the embodiment of the present application refers to the popular file generated by the real user, the preprocessing operation performed on the object storage log information can be an intranet request filtering operation performed on the object storage log information, and a deduplication processing of the requested resource based on a preset local time. Assuming that the preset local time is t0, the resource can be determined based on the URL (Universal Resource Locator, Universal Resource Locator), and the deduplication operation can be performed based on the URL within t0 time. Since the log formats of the content distribution log information CDN log and the object storage log information are different, in order to facilitate subsequent hot real-time calculation, the preprocessing operation performed on the content distribution log can be a parameter format unification operation performed on the content distribution log, and a deduplication processing of the requested resource based on a preset local time. The parameter format unification operation refers to unifying the parameter format of the CDN log with the parameter format of the object storage log information. The embodiment of the present application is not limited in this regard.

[0064] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0065] In step 202, a preset multi-dimensional data model is obtained, and a hot spot real-time calculation is performed on the real-time log information through the multi-dimensional data model to obtain data file features.

[0066] The collected data access log is subjected to pattern recognition, mainly hot spot pattern recognition. The hot spot pattern can be determined based on the data file features. At this time, the hot spot data can be determined based on the data file features, and the purpose of intelligently determining the hot spot data stored in the edge cache from the massive access data is achieved.

[0067] The hot spot pattern refers to a phenomenon that the hotness of the access data exceeds a certain degree. The degree required to measure the hotness can be determined based on multiple dimensions, such as dimension information related to user download behavior, including data attribution number, data download number at the same time, etc. The embodiments of the present application do not limit this.

[0068] In actual application, a hot spot analysis model can be obtained. Since the degree required to measure the hotness can be determined based on multiple dimensions, the obtained hot spot analysis model can be a preset multi-dimensional data model. The obtained preset multi-dimensional model can be used to measure the access hotness of data with log information in combination with multiple dimensions. Specifically, at this time, a hot spot real-time calculation can be performed on the real-time log information through the multi-dimensional data model to obtain data file features, so as to perform hot spot analysis based on the data file features and determine hot spot data.

[0069] Specifically, the data file features are extracted based on the multi-dimensional data model. The data file features can correspond to multiple dimensions, and the data file features can also include at least the related features of the data attribution number and the data download number at the same time. That is, the hot spot real-time calculation can be performed on the object storage log information and the content distribution log information of the preset local time through the multi-dimensional data model to obtain the data with log information in the preset local time, the corresponding data attribution number and the data download number at the same time.

[0070] In step 303, hot spot data is determined based on the data file features, and the hot spot data is stored in the edge cache.

[0071] After obtaining the data file features, the hotspot analysis model determines the hotspot data based on the data file features, adopts strategy judgment, mainly threshold control, that is, a way of filtering the access data based on threshold conditions to realize the identification of hotspot data. For example, when a certain access data exceeds a certain threshold, it can be determined that the popularity of this access data exceeds a certain degree, which is the hotspot data.

[0072] In the embodiments of the present application, the data attribution number and the data same time download number can be used as threshold conditions to filter the data with log information in a preset local time, and then the hotspot data of the preset local time can be determined in response to the data attribution number and / or the data same time download number reaching the preset threshold.

[0073] The data attribution number can be used to measure how many users the hotspot data belongs to at the current time, and the data same time download number can be used to measure how many times it is downloaded at the same time.

[0074] It should be noted that the hotspot analysis model can also introduce some user behavior estimation, file content judgment and other methods for model training to determine the hotspot data in advance in subsequent improvement. The embodiments of the present application do not limit this.

[0075] In actual application, the determined hotspot data can also be stored in the edge cache to complete the distributed storage of the CDN based on the hotspot data, realize the storage in the edge node based on the source station in a return-to-source manner, and facilitate subsequent implementation of generating and returning the download link based on the hotspot data stored in the edge cache when the user accesses the resource.

[0076] In a preferred embodiment of the present application, after determining the hotspot data based on the data file features, the hotspot data can also be configured with a popularity validity period, that is, the hotspot data stored in the edge cache is the hotspot data configured with the popularity validity period. Generally, the edge cache can exist a transformation cache set, which is stored in the Redis database (a key-value storage system) for example. The configured popularity validity period can be used to measure the storage time of the hotspot data in the edge cache. After storing the hotspot data configured with the popularity validity period in the edge cache, the system interface can adaptively return the download link of the CDN within this validity period, that is, the system can perceive the file popularity decay based on the configured popularity validity period, and the interface layer no longer returns the download link of the CDN.

[0077] Specifically, after receiving a user resource access request, a download link can be generated based on the hotspot data stored in the edge cache and returned in response to the user resource access request. In the case of large data service downlink and random user behavior, non-hotspot data is avoided from occupying the edge node. Under the premise of meeting a certain CDN hit rate / accuracy, recall rate is improved as much as possible, and users can use edge capabilities to improve access quality as much as possible.

[0078] In an embodiment of the present application, a cache set of the edge cache can be obtained, the cache set including hotspot data and a hotness validity period of the hotspot data, and the received user resource access request can include a request access data. At this time, the request access data can be determined as hotspot data in response to the cache set containing the request access data and the hotness validity period of the request access data not being invalid, and a download link for the request access data can be generated based on the edge cache and returned.

[0079] In the embodiments of the present application, the real-time log information obtained can be identified to obtain data file features, and the data file features can be judged to estimate hotspot data, so as to intelligently judge the hotspot data stored in the edge cache from the massive access data. This facilitates the subsequent user to generate and return a download link based on the hotspot data stored in the edge cache when accessing resources. In the case of large data service downlink and random user behavior, non-hotspot data is avoided from occupying the edge node, the hit rate of user access content is improved, and users can use edge capabilities to improve access quality as much as possible.

[0080] In some embodiments of the present application, as Figure 3As shown, the embodiments of the present application can also provide a data pushing system 310, which is configured with a log service module 31, a data stream processing module 32, a message system 33, a database 34, and a computer program for execution in the data pushing system (not shown in the figure), which, when executed by the data pushing system, implements any one of the data pushing methods, specifically can obtain real-time log information, performs pattern recognition on the real-time log information to obtain data file features; determines hot data based on the data file features, stores the hot data to an edge cache; in response to a user resource access request, generates a download link based on the hot data stored to the edge cache, and returns the download link. By analyzing the user's data access model in a pattern recognition manner, i.e., using a multi-dimensional data model to determine data file features, the most popular accessed data files are estimated for CDN pushing, thereby achieving intelligent judgment of massive accessed data to determine the popular files that need to be pushed to the CDN.

[0081] Referring to Figure 4 , a schematic diagram of an application scenario of data pushing provided by the embodiments of the present application is shown, which mainly involves a data pushing system as shown in Figure 3 , which can include a front-end (client) interface 410 and a back-end user system 411, such as Figure 3 , a log service module 31, a data stream processing module 32, a message system 33, a database 34, and a computer program for execution in the data pushing system (not shown in the figure) are mainly configured in the back-end user system, which can provide related interfaces based on data download services, execute the aforementioned computer program, and implement pushing of hot data of the CDN based on the log service module 31, the data stream processing module 32, the message system 33, and the database 34 as shown in Figure 3 . In actual applications, after processing the data, the system interface can be used to return the download link of the CDN to the client adaptively.

[0082] Specifically, after the feature judgment and caching of the hot data, the hot data can be cached based on the CDN. The hot data pushing of the CDN is manifested as a system layer returning a download address to the user in a manner of an interface.

[0083] As Figure 4As shown, the log service module 31, the data stream processing module 32, the message system 33, and the database 34 are system components used in the process of computing hot data, wherein the log service module 31 can provide object storage log information and CDN logs; the data stream processing module 32 usually adopts Flink (a distributed open source computing framework for data stream processing and batch data processing, which can support both stream processing and batch processing based on the same Flink stream execution model); the message system 33 can be embodied as Kafka (a distributed, partitioned, and multi-replica distributed message system); and the database 34 can be embodied as Redis. In actual application, Flink is mainly responsible for Flink stream computing, which can specifically include the stream processing process of data stream preprocessing, hot real-time computing, and hot model experiment and verification, Kafka can be responsible for storing computing information, and Redis can provide a hot data file set, such as an edge cache cache set, to the system layer query.

[0084] The whole process of implementing hot data pushing can be manifested as computing the collected object storage log information and CDN logs based on Flink, and specifically, hot data can be calculated according to some hot analysis models, Kafka is a part of the computation, and some intermediate computation results are stored, and then the hot data can be written into Redis, i.e., the cache set of the edge cache, based on Flink. The system layer will query the Redis result from the cache set when a user requests a download interface, and if the accessed data belongs to the hot set, the download address of the CDN can be returned at this time. The hot data stored in Redis has a validity time, i.e., the hot data is configured with a hot validity period, which can be used to measure the storage time of the hot data in the edge cache and the validity period of the system interface adaptive return of the CDN download link. In actual application, Flink can continuously determine that the accessed data is hot data, i.e., the popularity validity period of the hot data can be continuously updated in real time to enable the accessed data to exist in the Redis set for a long time; in one case, if the popularity of the accessed data turns cold, the Redis set will automatically eliminate it, i.e., the accessed data with an invalid popularity validity period cannot be stored in the edge cache, and then the system layer will return a non-CDN download link to the user.

[0085] In a specific application scenario, as an example, it can be applied in a network disk type data consumption scenario, in which different categories of network disk users have specific data consumption habits, such as resource type consumption, UGC (User-generated Content) type, backup type, etc. In order to better serve the network disk customer group, the download speed experience can be optimized.

[0086] In this scenario, its data downlink service can be represented as a network disk and album service, at this time, the CDN can be used to speed up the download, by placing the data of different user areas as close as possible to the user, using the edge bandwidth capability, to improve the user access speed, but the CDN increases the delay of the first access to the original station, at this time, the mode recognition method can be used to analyze the user's data access model, estimate the popular access data file for CDN push to improve the purpose of data access quality.

[0087] Specifically, in order to ensure the real-time of subsequent hot data determination, the collected data access log is usually real-time log information, and in order to ensure the completeness of the access log, the log information involved in the data access can be acquired when the real-time log information is acquired for pattern recognition, and since the CDN hot spot is online, the user download behavior will be recorded in the object storage log information, that is, the object storage log information and the content distribution CDN log information. Then, the log information involved in the data access can at least include the object storage log information and the content distribution,

[0088] After acquiring the real-time log information, in order to clean the interference data in the massive access log data, the object storage log information and the content distribution log information can be preprocessed respectively. In the embodiment of the present application, the interference data can refer to the interference data in the log information which is not needed for subsequent real-time hot spot calculation.

[0089] In the data preprocessing process, data cleaning operation can be performed, and the data cleaning operation performed can include data deduplication operation and other operations for removing interference data.

[0090] Exemplarily, since the intranet request is not triggered by the user, and the hot data to be determined by the present application refers to the popular file generated by the real user, the preprocessing operation performed on the object storage log information can be represented as an intranet request filtering operation on the object storage log information, and a deduplication processing of the requested resources based on a preset local time. Assuming that the preset local time is t0, the resources can be determined based on the URL, and specifically, the deduplication operation can be performed based on the URL at t0 time. Since the log formats of the content distribution log information CDN log and the object storage log information are different, in order to facilitate subsequent hot real-time calculation, the preprocessing operation performed on the content distribution log can be represented as a parameter format unification operation on the content distribution log, and a deduplication processing of the requested resources based on a preset local time, wherein the parameter format unification operation refers to unifying the parameter format of the CDN log with the parameter format of the object storage log information. For this, the present application is not limited.

[0091] The collected data access log is subjected to pattern recognition, mainly hot spot pattern recognition. The hot spot pattern can be determined based on the proposed data file features. At this time, the hot spot data can be determined based on the data file features, and the purpose of intelligently determining the hot spot data stored in the edge cache from the massive access data is achieved. In actual application, a hot spot analysis model can be obtained. Since the required degree of hotness for measurement can be determined based on multiple dimensions, the obtained hot spot analysis model can be a preset multi-dimensional data model. The obtained preset multi-dimensional model can be used to measure the access hotness of data with log information in combination with multiple dimensions. Specifically, in the hot spot real-time computing streaming process executed by Flink, the real-time log information can be subjected to hot spot real-time computing through the multi-dimensional data model to obtain data file features. The data file features are extracted based on the multi-dimensional data model. Therefore, the data file features can correspond to multiple dimensions, and the data file features can also include at least data attribution number and data same-time download number related features. That is, the object storage log information and content distribution log information of a preset local time can be subjected to hot spot real-time computing through the multi-dimensional data model to obtain data with log information within the preset local time, the corresponding data attribution number, and data same-time download number.

[0092] After the hot spot analysis model obtains the data file features, a strategy judgment is adopted when determining the hot spot data based on the data file features, mainly threshold control, that is, a way of filtering the access data based on threshold conditions to realize the identification of hot spot data. For example, when a certain access data exceeds a certain threshold, it can be determined that the hotness of the access data exceeds a certain degree, which is hot spot data.

[0093] It should be noted that in the streaming process of the hot spot model experiment and verification executed by Flink, specifically, the CDN hot spot data has a judgment standard of byte hit rate monitored by the CDN system, that is, if the byte hit rate is too low, it can be considered that the hot spot data is not accurately determined. At this time, the Flink hot spot model experiment can be equivalent to a similar traffic verification process. The model threshold corresponding to the high byte hit rate can be selected as the threshold condition for use. Specifically, the hot spot model experiment can be performed first, and then the determined model can be selected for Flink hot spot real-time computing.

[0094] In actual application, after the hot spot data is determined, the determined hot spot data can be stored to the edge cache, so that the user can generate and return a download link based on the hot spot data stored to the edge cache when accessing the resource, the non-hot spot data is avoided from occupying the edge node in the case that the data service downlink is large and the user behavior is relatively random, the recall rate is improved as much as possible under the premise of meeting a certain CDN hit rate / accuracy, and the user can improve the access quality by using the edge capability as much as possible.

[0095] In the embodiment of the present application, the mode recognition is used to analyze the data access model of the user, that is, the multi-dimensional data model is used to determine the data file features, the data file with popular access is estimated, and the CDN is pushed, so that the scheme of intelligently judging the popular file to be pushed to the CDN from the massive access data is achieved. Specifically, the real-time log information obtained can be recognized by using the mode recognition to obtain the data file features, and the data file features are determined to estimate the hot spot data, so that the purpose of intelligently judging the hot spot data stored to the edge cache from the massive access data is achieved, and the subsequent user can generate and return a download link based on the hot spot data stored to the edge cache when accessing the resource, the non-hot spot data is avoided from occupying the edge node in the case that the data service downlink is large and the user behavior is relatively random, the hit rate of the user accessing the content is improved, and the user can improve the access quality by using the edge capability as much as possible.

[0096] It should be noted that, for the method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited to the order of the actions described, because according to the embodiments of the present application, certain steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0097] Referring to Figure 5 , a structural block diagram of an embodiment of a data pushing device of the present application is shown, which can specifically include the following modules:

[0098] The data file feature recognition module 501 is configured to obtain real-time log information, and recognize the real-time log information by using the mode recognition to obtain data file features;

[0099] The hot spot data determination module 502 is configured to determine hot spot data based on the data file features, and store the hot spot data to the edge cache;

[0100] The download link generation module 503 is configured to generate a download link based on the hotspot data stored in the edge cache in response to a user resource access request, and return the download link.

[0101] In an embodiment of the present application, the data file feature identification module 501 can include the following sub-modules:

[0102] The multi-dimensional data model acquisition sub-module is configured to acquire a preset multi-dimensional data model, and the preset multi-dimensional model is used to measure the access popularity of data with log information in combination with multiple dimensions.

[0103] The data file feature identification sub-module is configured to perform hotspot real-time calculation on the real-time log information by using the multi-dimensional data model, to obtain data file features.

[0104] In an embodiment of the present application, the real-time log information at least includes object storage log information and content distribution log information; before the real-time log information is subjected to hotspot real-time calculation, the following sub-modules can be further included:

[0105] The log information preprocessing sub-module is configured to perform preprocessing on the object storage log information and the content distribution log information respectively.

[0106] In an embodiment of the present application, the log information preprocessing sub-module can include the following units:

[0107] The first log information processing unit is configured to perform an intranet request filtering operation on the object storage log information, and perform a duplicate removal processing of a request resource based on a preset local time.

[0108] The second log information processing unit is configured to perform a parameter format unification operation on the content distribution log, and perform a duplicate removal processing of a request resource based on the preset local time.

[0109] In an embodiment of the present application, the data file features correspond to multiple dimensions, and at least include a data attribution number and a data same-time download number; the data file feature identification sub-module can include the following units:

[0110] The hotspot calculation unit is configured to perform hotspot real-time calculation on the object storage log information and the content distribution log information of a preset local time by using the multi-dimensional data model, to obtain data with log information in the preset local time, a corresponding data attribution number, and a data same-time download number.

[0111] In an embodiment of the present application, the hotspot data determination module 502 can include the following sub-modules:

[0112] a data filtering submodule, configured to filter data with log information in a preset local time by taking the data attribution number and the data same-time download number as threshold conditions;

[0113] a hotspot data determination submodule, configured to determine hotspot data in the preset local time in response to the data attribution number and / or the data same-time download number reaching a preset threshold.

[0114] In an embodiment of the present application, after determining the hotspot data based on the data file features, the apparatus provided in the embodiment of the present application can further include the following modules:

[0115] an effective period configuration module, configured to configure a hotness effective period for the hotspot data; the hotness effective period is used to measure the storage time of the hotspot data in the edge cache.

[0116] In an embodiment of the present application, the user resource access request includes a request to access data; the download link generation module 503 can include the following submodules:

[0117] a cache set acquisition submodule, configured to acquire a cache set of the edge cache, the cache set including hotspot data and a hotness effective period of the hotspot data;

[0118] a hotspot data determination submodule, configured to determine the request to access data as hotspot data in response to the cache set containing the request to access data and the hotness effective period of the request to access data not being invalid;

[0119] a download link return submodule, configured to generate a download link for the request to access data based on the edge cache, and return the download link.

[0120] In the embodiment of the present application, the apparatus for pushing hotspot data based on an edge node provided in the embodiment of the present application can use the mode recognition method to recognize the acquired real-time log information to obtain data file features, and estimate the hotspot data by judging the data file features, so as to intelligently judge the hotspot data stored in the edge cache from the massive access data, facilitate generating and returning the download link based on the hotspot data stored in the edge cache when the user accesses the resource, avoid non-hotspot data occupying the edge node in the case of large data service downstream and random user behavior, improve the hit rate of user access content, and make the user use the edge capability to improve the access quality as much as possible.

[0121] For the apparatus embodiment, the description is relatively simple because it is basically similar to the method embodiment, and the related parts refer to the part of the method embodiment.

[0122] The embodiment of the present application further provides an electronic device, comprising:

[0123] The computer program is stored in the memory and can be run on the processor, and when the computer program is executed by the processor, each process of the data pushing method embodiment is realized, and the same technical effects can be achieved. To avoid repetition, details are not described herein.

[0124] The embodiment of the present application further provides a computer readable storage medium, and a computer program is stored in the computer readable storage medium. When the computer program is executed by the processor, each process of the data pushing method embodiment is realized, and the same technical effects can be achieved. To avoid repetition, details are not described herein.

[0125] Each embodiment in the specification is described in a progressive manner, and each embodiment mainly describes the difference from other embodiments. The same and similar parts of each embodiment can be referred to each other.

[0126] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device or computer program product. Therefore, the embodiments of the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0127] The embodiments of the present application are described with reference to flowcharts and / or block diagrams according to the method, terminal device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to a general purpose computer, special purpose computer, embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal device produce a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The device for implementing the functions specified in one flow or multiple flows and / or blocks Figure 1 The device for implementing the functions specified in one flow or multiple flows and / or blocks

[0128] These computer program instructions can also be stored in a computer readable storage medium, which can guide the computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The device for implementing the functions specified in one flow or multiple flows and / or blocks Figure 1the function specified in the one or more blocks.

[0129] These computer program instructions can also be loaded into computer or other programmable data processing terminal devices, so that a series of operation steps are performed on the computer or other programmable terminal devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable terminal devices provide a process for implementing the flow Figure 1 the flow or flows and / or blocks Figure 1 the steps of the function specified in the one or more blocks.

[0130] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to cover the preferred embodiments and changes and modifications falling within the scope of the present application.

[0131] It should also be noted that, in the present text, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, article, or terminal device including a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or terminal device. Without more limitations, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or terminal device including the element.

[0132] The above provides a data pushing method, a corresponding data pushing system, a corresponding electronic device, and a corresponding computer readable storage medium. The principles and implementation manners of the present application are described by using specific examples in the present text. The above example descriptions are only used to help understand the method and core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed. In summary, the content of the present description should not be understood as a limitation of the present application.

Claims

1. A data pushing method, the method comprising: acquiring real-time log information, and performing pattern recognition on the real-time log information to obtain data file features; the real-time log information at least comprises object storage log information and content distribution log information, and the pattern recognition is hot spot pattern recognition; determining hot spot data based on the data file features, and storing the hot spot data to an edge cache; the hot spot data is access data with a degree of hotness exceeding a preset threshold, and the preset threshold is determined based on multi-dimensional information related to user download behavior; in response to a user resource access request, generating a download link based on the hot spot data stored to the edge cache, and returning the download link.

2. The method of claim 1, wherein the pattern recognition on the real-time log information to obtain data file features comprises: acquiring a preset multi-dimensional data model, the preset multi-dimensional data model being used to measure the access degree of hotness of data with log information in combination with multiple dimensions; performing hot spot real-time calculation on the real-time log information through the multi-dimensional data model to obtain data file features.

3. The method of claim 2, the real-time log information comprising at least object storage log information and content distribution log information. Before the hot spot real-time calculation on the real-time log information, the method further comprises: respectively pre-processing the object storage log information and the content distribution log information; the pre-processing of the object storage log information and the content distribution log information comprises: performing an intranet request filtering operation on the object storage log information, and performing a request resource deduplication processing based on a preset local time; and / or, performing a parameter format unification operation on the content distribution log, and performing a request resource deduplication processing based on the preset local time.

4. The method of claim 2, wherein the data file features correspond to multiple dimensions, at least comprising data attribution number and data same-time download number; and the hot spot real-time calculation on the real-time log information through the multi-dimensional data model to obtain data file features comprises: performing hot spot real-time calculation on the object storage log information and the content distribution log information of a preset local time through the multi-dimensional data model to obtain data with log information within the preset local time, corresponding data attribution number, and data same-time download number.

5. The method of claim 1 or 4, wherein the determination of hot spot data based on the data file features comprises: taking the data attribution number and the data same-time download number as threshold conditions to filter data with log information within a preset local time; in response to the data attribution number and / or data same-time download number reaching a preset threshold, determining hot spot data of the preset local time.

6. The method of claim 5, wherein after the determination of hot spot data based on the data file features, the method further comprises: configuring a degree of hotness validity period for the hot spot data; the degree of hotness validity period is used to measure the storage time of the hot spot data in the edge cache.

7. The method of claim 6, wherein the user resource access request comprises a request to access data; and the response to the user resource access request, the generation of a download link based on the hot spot data stored to the edge cache, and the return of the download link comprise: obtain a cache set of the edge cache, the cache set comprising hotspot data and a validity period of the hotspot data; determine that the requested access data is hotspot data in response to the cache set containing the requested access data and the validity period of the requested access data not being expired; generate a download link for the requested access data based on the edge cache, and return the download link. 8.A data pushing system configured with a log service module, a data stream processing module, a message system, a database, and a computer program for execution in the data pushing system, wherein the computer program, when executed by the data pushing system, implements the data pushing method according to any one of claims 1 to 7.

9. An electronic device, comprising: a processor, a memory, and a computer program stored on the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the data pushing method according to any one of claims 1 to 7. 10.A computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and wherein the computer program, when executed by a processor, implements the data pushing method according to any one of claims 1 to 7.

Images