Big data storage service method and system

Abstract

The invention relates to the technical field of big data storage, in particular to a big data storage service method and system. The method comprises the following steps of: screening suspected abnormal time periods by analyzing a time sequence growth trend of an advertisement page view in a current period and combining historical comparative analysis and stability of an access amplification; obtaining an access jump abnormity index through correlation analysis of an access page jump sequence in a suspected abnormal time period and a difference between the access page jump sequence and a non-abnormal time period sequence, and determining a normal access behavior in combination with an access behavior abnormal condition of an access user in a current period; and based on the normal access behavior, integrating the condition that the advertisement is accessed by each user and the effective interaction condition, and obtaining the storage priority optimization resource allocation of each advertisement. According to the method, normal access is identified based on the time sequence behavior and jump logic difference, the user interaction depth is analyzed, the storage strategy is dynamically adjusted for the advertisement, resource occupation and waste caused by malicious behaviors are reduced, and the effectiveness of storage resource utilization is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data storage, and particularly relates to a big data storage service method and system. Background Art

[0002] Advertising placement needs to instantaneously record and analyze users' advertising interaction behaviors, such as advertising access behaviors, conversion behaviors, etc., and adjust the advertising content delivered to users in real time according to users' access behaviors, so as to optimize the accuracy of advertising placement.

[0003] To efficiently process and maintain the stability of the advertising placement system, existing methods usually rely on the principle of distributed storage. According to users' access habits, such as recent advertising interaction frequencies, dwell times, etc., storage resources are allocated. For example, relevant advertising data with a relatively high recent interaction frequency of users is stored in edge storage nodes, and the rest of the data with a relatively low interaction frequency is transmitted to the cloud for storage, thereby reducing access latency and improving the effective utilization rate of storage resources.

[0004] During the actual operation of a big data-based storage service system, there is a phenomenon of a sudden increase in access volume caused by burst traffic and malicious access attacks. Burst traffic may be caused by sudden event activities, and its access behavior has storage value compared with malicious access. For example, around shopping festivals, the advertising access volume may increase, and existing methods cannot accurately identify malicious access situations in the surge of browsing, thus affecting the reasonable allocation of storage resources and bringing potential risks of abnormal storage service responses. Summary of the Invention

[0005] In order to solve the technical problem in the prior art that malicious access situations in the surge of browsing cannot be accurately identified, thus affecting the reasonable allocation of storage resources, the purpose of the present invention is to provide a big data storage service method and system, and the specific technical solutions adopted are as follows:

[0006] The present invention provides a big data storage service method, and the method includes:

[0007] Obtain the page jump sequence, the browsing time of each page, the access depth, and the payment mark when each advertisement is accessed by each user once in each cycle of the historical period.

[0008] Analyze the growth trend of the number of accesses to each advertisement in chronological order in the current cycle, and determine the growth period; according to the stability of the increase rate of the number of accesses in the growth period, combined with the deviation from the number of accesses in the same period of the historical cycle, obtain an abnormal probability index; determine the suspected abnormal period of each advertisement based on the abnormal probability index.

[0009] During the suspected abnormal period of each advertisement, obtain the jump anomaly index of a single access through the deviation of the page jump sequence length between a single access and other accesses during the non-suspected abnormal period, as well as the association of each jump; according to the possible anomaly index of each access of the corresponding user during the current cycle for a single access, and the similarity between the page jump sequence and the non-suspected abnormal period, combine the access frequency to obtain the user anomaly index of a single access; determine the normal access of each advertisement during the current cycle according to the user anomaly index and the jump anomaly index.

[0010] In each storage node, obtain the storage priority of each advertisement through the access frequency degree of each advertisement by each user and the number of payment marks, and combine the page viewing time and access depth when the advertisement is normally accessed; reallocate the stored content according to the storage priority of the advertisement.

[0011] Further, the method for obtaining the growth period includes:

[0012] For any advertisement, take the total number of accesses of the advertisement at each moment in time series as the access volume at each moment.

[0013] During the current cycle, calculate the difference in access volume between each moment of the advertisement and the previous moment as the growth index of the advertisement at each moment; screen out the moments when the growth index is less than the value 0, and record the remaining moments as target moments.

[0014] The period composed of consecutive target moments during the current cycle is used as the growth period of the advertisement.

[0015] Further, the method for obtaining the possible anomaly index includes:

[0016] For any advertisement, sequentially take each growth period as the analysis period.

[0017] For any cycle outside the current cycle in the historical period, obtain the co-distribution period in the current cycle by analyzing the distribution of the analysis period in the current cycle; take the difference between the average access volume of the advertisement in the analysis period and the average access volume of the advertisement in the co-distribution period as the access deviation between the analysis period and the cycle.

[0018] Take the average value of the access deviations between the analysis period and all other cycles in the historical period as the historical access deviation degree of the advertisement in the analysis period.

[0019] After calculating the difference in growth index between every two adjacent moments of the advertisement in the analysis period, sum up all the growth index differences and perform a negative correlation mapping to obtain the fluctuation stability of the advertisement in the analysis period.

[0020] Combining the average growth index, historical access deviation degree, and fluctuation stability degree of the advertisement at all times during the analysis period, obtain the abnormal probability index of the advertisement during the analysis period.

[0021] Further, the method for obtaining the jump abnormal index includes:

[0022] Take every two adjacent pages in the page jump sequence during a single access as a jump group, and use the page with a larger serial number in the page jump sequence in the jump group as the jump page; according to the distribution order of the jump groups in the page jump sequence, arrange the jump pages of the jump groups to obtain the jump behavior sequence of a single access.

[0023] For any access of each advertisement during the suspected abnormal period, use all the accesses of the advertisement corresponding to this access during all non-suspected abnormal periods as reference accesses.

[0024] Successively take each serial number in the jump behavior sequence of this access as the target serial number; use the total number of accesses with the target serial number existing in the jump behavior sequence of the reference access as the same jump times of the target serial number; count the total number of accesses with the target serial number existing in the jump behavior sequence of the reference access and the target serial number corresponding to the same jump page as the same behavior times of the target serial number; use the ratio of the same behavior times of the target serial number to the same jump times as the jump correlation index of the target serial number.

[0025] Perform a negative correlation mapping on the abnormal probability index of the suspected abnormal period where this access is located to obtain the normal analysis probability index of this access.

[0026] Use the product of the jump correlation index of the target serial number and the normal analysis probability index of this access as the correlation feature index of the target serial number in this access; calculate the average value of the correlation feature indexes of all serial numbers in the jump behavior sequence of this access, and perform a negative correlation mapping to obtain the jump correlation abnormal index of this access.

[0027] Calculate the difference between the length of the page jump sequence during this access and the average value of the lengths of all page jump sequences in the reference access to obtain the jump times abnormal index of this access.

[0028] Combine the jump correlation abnormal index and the jump times abnormal index of this access to obtain the jump abnormal index of this access.

[0029] Further, the method for obtaining the user abnormal index includes:

[0030] Take the first two pages in the page jump sequence during a single access as an analysis group, and successively incorporate the subsequent pages in the page jump sequence into the analysis group. Each time an incorporation is made, a new analysis group is obtained.

[0031] For any access to an advertisement during a suspected abnormal period, calculate the occurrence frequency of each analysis group of this access during all non-suspected abnormal periods to obtain the jump fixed probability index of each analysis group; take the maximum jump fixed probability index in this access as the similar jump index of this access;

[0032] Take the user corresponding to this access as the analysis user; in the current cycle, take the ratio of the total number of times the analysis user accesses this advertisement to the total number of times all users access this advertisement as the access frequency index of the analysis user; calculate the mean value of the abnormal probability index in all accesses of the analysis user to obtain the access abnormal index of the analysis user; calculate the mean value of the similar jump indexes of all accesses of the analysis user and perform a negative correlation mapping to obtain the jump order abnormal index of the analysis user;

[0033] Combine the access frequency index, access abnormal index, and jump order abnormal index of the analysis user to obtain the user abnormal index of this access.

[0034] Further, the method for determining a normal access includes:

[0035] Normalize the product of the jump abnormal index and the user abnormal index of a single access as the malicious behavior index of the single access; take the access with the malicious behavior index greater than the preset malicious threshold as the malicious access of each advertisement;

[0036] In the current cycle of each advertisement, take all accesses except malicious accesses as the normal accesses of each advertisement.

[0037] Further, the method for obtaining the storage priority includes:

[0038] For any normal access to an advertisement, take the ratio of the number of times the user corresponding to this normal access accesses this advertisement in the current cycle to the total number of times the corresponding user is served this advertisement as the access interest index of this normal access;

[0039] Multiply the mean value of the browsing time of each page during this normal access by the mean value of the access depth of each page to obtain the behavior interest index of this normal access; perform a negative correlation mapping on the malicious behavior index of this normal access to obtain the behavior reliability of this normal access;

[0040] Multiply the behavior reliability, access interest index, and behavior interest index of this normal access to obtain the effective index of this normal access;

[0041] In each storage node, take the total number of times each advertisement is accessed by each user as the node access degree; take the ratio of the total number of times this advertisement is normally accessed to the node access degree as the node access frequency of this advertisement;

[0042] Calculate the mean of all valid metrics for normal access to the advertisement, the product of the number of payment tags and the node access frequency, and obtain the storage priority of the advertisement in the storage node.

[0043] Further, the re - allocation of storage content according to the storage priority of the advertisement includes:

[0044] In each storage node, normalize the storage priority of the advertisement to obtain a storage retention metric;

[0045] When the storage retention metric of the advertisement is greater than the preset storage threshold, retain the relevant data of the corresponding advertisement in the storage node, otherwise transmit the advertisement - related data to cloud storage.

[0046] Further, the method for determining the suspected abnormal period includes:

[0047] In all growth periods of each advertisement, regard the growth period in which the abnormal probability metric is greater than the preset abnormal threshold as the suspected abnormal period.

[0048] The present invention also provides a big - data storage service system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a big - data storage service method as described above are implemented.

[0049] The present invention has the following beneficial effects:

[0050] By analyzing the temporal growth trend of the advertisement access volume in the current period, combining the deviation of the access volume in the same period of the historical period and the stability of the access growth rate, the abnormal probability metric is calculated, and the suspected abnormal periods with relatively abnormal sharp increases in traffic are initially screened, providing a time - period basis for subsequent discrimination of malicious traffic behavior. Within the suspected abnormal periods, further through the correlation analysis of the page - jump sequence and the length difference from the non - abnormal period sequence, the jump - abnormality metric for a single access is quantified. At the same time, combined with the abnormal access behavior of the user in the current period, the normal access behavior is comprehensively determined, excluding the interference of malicious behavior on the data - storage resource allocation. Based on the normal access - behavior data, integrating the access situation of each advertisement by users, the number of payment tags, the page - browsing duration, and the access depth, the storage priority of each advertisement is dynamically calculated, realizing the real - time optimal allocation of edge - storage - node resources. The present invention identifies abnormal access based on temporal - behavior analysis and jump - logic differences, dynamically adjusts the storage strategy for advertisements in combination with the user - interaction depth of normal access, reduces the resource - occupancy waste caused by malicious behavior, improves the rapid response of effective advertisement data, and enhances the effectiveness of storage - resource utilization. Description of the Drawings

[0051] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0052] Figure 1 Flowchart of a big data storage service method provided by an embodiment of the present invention;

[0053] Figure 2 Structural schematic diagram of a distributed storage provided by an embodiment of the present invention. Detailed implementation manners

[0054] To further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following will, in conjunction with the accompanying drawings and preferred embodiments, detail the specific implementation manners, structures, features, and effects of a big data storage service method and system according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.

[0056] The following will specifically describe the specific solutions of a big data storage service method and system provided by the present invention with reference to the accompanying drawings.

[0057] For the distributed big data storage service based on advertising data, through the distributed architecture of cooperation between edge storage nodes and the cloud, please refer to Figure 2 , which shows a structural schematic diagram of a distributed storage provided by an embodiment of the present invention. Edge storage nodes such as CDN nodes, local servers, etc. are deployed adjacent to user terminals to form the first layer of storage in the distributed network. When a user initiates an advertising access request, the edge node can directly call the high-priority data cached locally, reducing the response time to the millisecond level and avoiding the network latency of accessing the cloud across regions. The cloud storage bears a large amount of low-frequency data, supports long-term analysis and global policy generation, and can reduce the bandwidth cost by reducing the frequency of cloud data transmission.

[0058] The resource utilization rate can be improved by dynamically allocating storage resources according to priorities. Please refer to Figure 1 , which shows a flowchart of a big data storage service method provided by an embodiment of the present invention. The method includes the following steps:

[0059] S1: Obtain the page jump sequence, page viewing time, access depth, and payment mark for each advertisement when each user accesses it once in each cycle during the historical period.

[0060] The user data will be transmitted to the edge storage device for storage nearby. The edge storage node stores user access logs, advertisement materials, etc. In the embodiments of the present invention, taking 24 hours as a cycle, the advertisement interaction behavior logs of users in each edge storage point are recorded at each sampling moment, mainly including the page jump sequence when each advertisement is accessed by the user, the viewing time of staying on each page, and the access depth corresponding to the page, and marking the payment page where the advertisement appears based on the label. Analyze the historical situation with data for a week in the historical period to facilitate the subsequent screening of malicious traffic.

[0061] S2: Analyze the growth trend of the number of accesses to each advertisement in chronological order in the current cycle, and determine the growth period; according to the stability of the increase amplitude of the number of accesses during the growth period, combined with the deviation from the number of accesses in the same period in the historical cycle, obtain the possible abnormal indicators; determine the suspected abnormal period of each advertisement based on the possible abnormal indicators.

[0062] During the storage service, there are malicious attack behaviors of forging a large number of requests with abnormal traffic, and due to factors such as sudden event activities such as promotional activities, the normal access behavior of users causes a sudden increase in the advertisement page access volume. Among them, malicious attacks usually lead to the occupation and waste of system storage resources.

[0063] There are certain differences between the two situations of sudden increases in traffic access. When the sudden increase in the access volume of a certain advertisement is caused by the normal behavior of users, the access behavior still conforms to the normal access habits of users. For example, normal accesses are mostly concentrated in the user's leisure time, such as during lunch breaks, after work, or on weekends, with a relatively slow growth rate and a relatively uniform time distribution. However, the access time of malicious attacks is irregular, and the access volume growth rate is faster, and the growth period is more concentrated.

[0064] First, through the growth trend of the access volume, initially screen the abnormal periods with a sudden increase in traffic. Preferably, in the embodiments of the present invention, the method for obtaining the growth period includes:

[0065] For any advertisement, regard the total number of accesses to the advertisement at each moment in chronological order as the access volume at each moment. In the current cycle, calculate the difference in access volume between each moment and the previous moment of the advertisement as the growth index of the advertisement at each moment. When the growth index is less than zero, it indicates that the access volume shows a decreasing trend, and it is not a period with malicious traffic. Therefore, screen out the moments when the growth index is less than the value 0, and record the remaining moments as the target moments.

[0066] Merge the target moments. That is, if there are adjacent moments that are both target moments, merge them to reflect the possible continuously abnormal access part. The period composed of consecutive target moments in the current cycle is used as the growth period of the advertisement. There may be a malicious attack stage in the growth period.

[0067] Abnormal attacks are different from normal user behaviors. If the access volume of a single growth period varies greatly from that of users in the same period within the historical cycle, and its access volume increase rate is fast and the volatility is small, then the possibility of abnormal attacks in this period is greater. Preferably, in the embodiment of the present invention, the method for obtaining the abnormal probability index includes:

[0068] For any advertisement, each growth period is sequentially used as the analysis period, and all growth periods are analyzed. For any cycle outside the current cycle in the historical period, by analyzing the distribution of the analysis period in the current cycle, the co-distribution period in this cycle is obtained, that is, the time series of the historical cycle and the current cycle are aligned at both the head and the tail for analysis, and the part of the same time period as the analysis period is selected as the co-distribution period. For example, if the analysis period is from 0 to 1 in the current cycle, the co-distribution period is from 0 to 1 in each historical cycle.

[0069] Further, the difference between the average access volume of the advertisement in the analysis period and the average access volume of the advertisement in the co-distribution period is used as the access deviation between the analysis period and this cycle, and the difference in access rules is reflected through the difference in access volume between cycles. Combining all historical cycles, the average value of the access deviations between the analysis period and all other cycles in the historical period is used as the historical access deviation degree of the advertisement in the analysis period. The higher the historical access deviation degree, the more serious the deviation of the access situation in the analysis period from the history, and the higher the possibility of abnormality.

[0070] Furthermore, in combination with the analysis of the growth rate situation, after calculating the difference in growth indices between every two adjacent moments of the advertisement in the analysis period, the sum of all growth index differences is obtained and negatively correlated mapping is performed to obtain the fluctuation stability degree of the advertisement in the analysis period, which characterizes the stability of the increase rate at different moments. The smaller the overall difference, the more stable the increase rate is reflected. It should be noted that negatively correlated mapping is a well-known technical means in the art, such as using an inverse proportion form, and is not limited here.

[0071] Finally, by combining the average growth index, historical access deviation degree, and fluctuation stability degree of the advertisement at all times during the analysis period, an abnormal possibility index of the advertisement during the analysis period is obtained. In the embodiment of the present invention, the average growth index, historical access deviation degree, and fluctuation stability degree at all times during the analysis period are multiplied and normalized to obtain the abnormal possibility index of the advertisement during the analysis period. It should be noted that normalization is a well-known technical means in the art, and the choice of normalization can be linear normalization or standard normalization, etc. The specific normalization method is not limited here.

[0072] The larger the average value of the growth index, the greater the increase in the number of accesses during the analysis period. The larger the historical access deviation degree, the higher the degree of abnormality of the analysis period compared to the historical cycle. The larger the stable fluctuation degree, the more stable the increase at different times. Therefore, the possibility of an abnormal attack is higher.

[0073] Finally, it can be determined whether there is a malicious attack period through threshold judgment. According to the abnormal possibility index, the suspected abnormal periods are screened. In the embodiment of the present invention, among all the growth periods of each advertisement, the growth periods with an abnormal possibility index greater than the preset abnormal threshold are used as suspected abnormal periods, and the non-suspected abnormal periods can be regarded as normal periods. Among them, the preset abnormal threshold can be set to 0.5, and the specific value can be adjusted by the implementer according to the specific implementation scenario and is not limited here.

[0074] S3: In the suspected abnormal periods of each advertisement, by analyzing the deviation of the page jump sequence length between a single access and other accesses in the non-suspected abnormal periods and the correlation of each jump, an abnormal jump index of the single access is obtained; according to the abnormal possibility index of each access of the corresponding user during the current cycle and the similarity between the page jump sequence and the non-suspected abnormal periods, combined with the access frequency, a user abnormal index of the single access is obtained; according to the user abnormal index and the abnormal jump index, the normal accesses of each advertisement during the current cycle are determined.

[0075] In the suspected abnormal periods, malicious behavior is analyzed and screened out through access behavior. Malicious access behavior is usually automatically executed by a set program, which has a high degree of regularity and is quite different from the access behavior of normal users. When a normal user accesses an advertisement, the page jump process usually conforms to a certain process or access habit. For example, the user's access habit may be: jump from the advertisement activity page to the product detail page, then to the order page, and finally to the payment page, etc. However, the maliciously accessed page usually frequently targets a certain page for access. Even if there are page jumps, the jump behavior is quite different from the access habits of normal users. Therefore, the access behavior of normal users and malicious traffic are distinguished by combining the differences in jump behavior.

[0076] First, perform anomaly analysis on the access jump behavior. Although the page jump behavior of users is random, there is a certain process rule in the page jump sequence of each access jump. For example, from the advertisement product details page to the shopping cart interface, etc. Therefore, in the access behavior of this advertisement, the higher the correlation between each jump behavior, the more in line with the user's access habits the jump behavior is. And considering the proximity of the number of jumps to that during normal periods, an anomaly index for jump analysis is obtained.

[0077] Preferably, in the embodiment of the present invention, the method for obtaining the jump anomaly index includes:

[0078] First, disassemble the page jump sequence for jump analysis, which is convenient for subsequent analysis of each jump situation. Take every two adjacent pages in the page jump sequence during a single access as a jump group, and each jump group represents a jump process. Take the page with a larger serial number in the page jump sequence in the jump group as the jump page, that is, the destination page for each jump. According to the distribution order of the jump groups in the page jump sequence, arrange the jump pages of the jump groups to obtain the jump behavior sequence of a single access.

[0079] For example, there are 5 pages from the homepage of an advertisement to the payment completion. Among them, there may be users who access all pages in order from beginning to end. The page jump sequence of their access behavior is [1, 2, 3, 4, 5], with a total of 4 jumps. The jump groups are 1 - 2, 2 - 3, 3 - 4, 4 - 5, and the jump pages for each jump are 2, 3, 4, 5 respectively. Or there are users who have repeatedly jumped between the product details page and the evaluation page. The page jump sequence is [1, 2, 3, 2, 3, 4, 5], with 6 jumps. The jump pages for each jump are 2, 3, 2, 3, 4, 5 respectively. At this time, the two users have the same destination for the first two jumps and there is a behavior correlation. Particularly, there are also cases where users click into the advertisement and then exit immediately. At this time, the page jump sequence is [1], with 0 jumps.

[0080] Furthermore, analyze any access of each advertisement during the suspected abnormal period. First, take all the accesses of the advertisement corresponding to this access during all non - suspected abnormal periods as reference accesses, and analyze by using the access situation of the advertisement during normal periods as a reference. Take each serial number in the jump behavior sequence of this access as the target serial number in turn, and analyze the situation of the destination page for each jump in turn.

[0081] Take the total number of accesses with the target serial number in the jump behavior sequence of the reference accesses as the same - jump times of the target serial number, that is, in the jump order of all reference accesses, the number of jump situations where the serial number is the same as the target serial number. For example, when the target serial number is 2, take the number of accesses with serial number 2 in the jump behavior sequence of the reference accesses as the same - jump times.

[0082] Statistically count the total number of accesses in the jump behavior sequence of the reference access where there is a target serial number and the jump pages corresponding to the target serial number are the same, which is used as the number of same-behavior times of the target serial number, that is, the number of accesses where the jump pages are also the same under the same serial number in the jump behavior sequence. Finally, take the ratio of the number of same-behavior times of the target serial number to the number of same-jump times as the jump correlation index of the target serial number, which reflects the degree of association for each jump situation when compared and analyzed with normal accesses.

[0083] Further, perform a negative correlation mapping on the abnormal probability index of the suspected abnormal period in which the access is located to obtain the normal analysis probability index of the access. Combining the time period abnormal analysis, if the abnormal probability of the behavior corresponding to a single access in the corresponding time period is smaller, that is, the normal analysis probability index is larger, it indicates that the contribution degree of this access behavior to the analysis of normal access behavior is higher.

[0084] Therefore, take the product of the jump correlation index of the target serial number and the normal analysis probability index of the access as the correlation feature index of the target serial number in this access, which reflects the correlation feature of each jump. Combine the correlations of all jump situations, calculate the mean value of the correlation feature indexes of all serial numbers in the jump behavior sequence of this access, and perform a negative correlation mapping to obtain the jump correlation abnormal index of this access. When the correlation features of multiple jump situations in this access are smaller, it indicates that the degree of association with normal jumps is lower and it is more likely to be a malicious behavior. Therefore, the larger the jump correlation abnormal index.

[0085] Furthermore, calculate the difference between the length of the page jump sequence during this access and the mean value of the lengths of all page jump sequences in the reference access to obtain the jump number abnormal index of this access. When the difference in the overall jump length is higher, it reflects that the jump abnormality degree is more significant.

[0086] Finally, combine the jump correlation abnormal index and the jump number abnormal index of this access to obtain the jump abnormal index of this access. In the embodiment of the present invention, take the product of the jump correlation abnormal index and the jump number abnormal index as the jump abnormal index of this access. The larger the jump abnormal index, the higher the probability that this access is a malicious behavior analyzed from the jump behavior.

[0087] Secondly, analyze the user abnormality of the access. If the frequency of the user corresponding to a single access to the advertisement in the current cycle is higher and the abnormal degree of the time period where each access is located is higher, then the possibility that the user has an access abnormality in the current cycle is greater. And the greater the difference between the jump page during the access of this user in the current cycle and the access behavior of accessing this advertisement in the normal time period, the greater the possibility that this access behavior is a malicious behavior from the user aspect.

[0088] Preferably, in the embodiment of the present invention, the method for obtaining the user abnormality index includes:

[0089] Considering that the advertisement interface settings are relatively fixed, the partial order of user page jumps is highly consistent. For example, there is only a single jump process between the advertisement introduction page and the advertisement details page. At this time, the similarity of the user's page sequence should be relatively high. Therefore, considering the similarity of the order of page jump sequences between user access and normal periods, the situation where a single access conforms to the user's access habits is analyzed.

[0090] First, take the first two pages in the page jump sequence during a single access as an analysis group, and then incorporate the subsequent pages in the page jump sequence into the analysis group in turn. Each time a page is incorporated, a new analysis group is obtained. For example, if the page jump sequence is "abcd", the analysis groups are "ab", "abc", and "abcd".

[0091] For any advertisement and any single access during a suspected abnormal period, the higher the frequency of occurrence of the analysis group during normal period access, the more normal the user's access behavior at this time. Calculate the occurrence frequency of each analysis group of this access in all non-suspected abnormal periods to obtain the jump fixation probability index of each analysis group. Take the maximum jump fixation probability index in this access as the similarity jump index of this access. For multiple analysis groups, take the highest frequency as the possible degree of fixed-order jumps. The larger the similarity jump index, the higher the similarity degree of the user's jump habits.

[0092] Take the user corresponding to this access as the analysis user. In the current cycle, take the ratio of the total number of times the analysis user accesses this advertisement to the total number of times all users access this advertisement as the access frequency index of the analysis user. The higher the user frequency of accessing this advertisement, the higher the probability of abnormal access for this user.

[0093] Further, in the current cycle, calculate the average value of the abnormal probability index in all accesses of the analysis user to obtain the access abnormality index of the analysis user. Combining the access abnormality index, the more periods with a relatively high probability of abnormal attacks in the current cycle of user access, the higher the probability of user abnormality.

[0094] Further, in the current cycle, calculate the average value of the similarity jump indexes of all accesses of the analysis user and perform a negative correlation mapping to obtain the jump order abnormality index of the analysis user. Combining the similarity jump indexes of all accesses of the analysis user, the smaller the overall similarity jump index, the less similar the user's access behavior habits are to those in the normal period, and the higher the probability of abnormality.

[0095] Finally, by combining and analyzing the user access frequency index, access anomaly index, and jump sequence anomaly index of the user, the user anomaly index of the access is obtained. In the embodiment of the present invention, the product of the user access frequency index, access anomaly index, and jump sequence anomaly index of the analyzed user is used as the user anomaly index of the access. The larger the user anomaly index, the higher the possibility that the access is a malicious behavior analyzed from the user behavior.

[0096] Based on the malicious behavior analysis of the two aspects, according to the user anomaly index and the jump anomaly index, the normal access of each advertisement in the current period is determined. In the embodiment of the present invention, the product of the jump anomaly index of a single access and the user anomaly index is normalized and used as the malicious behavior index of the single access. The larger the index, the more abnormal the access behavior. The access with a malicious behavior index greater than the preset malicious threshold is used as the malicious access of each advertisement. The preset malicious threshold is set to 0.6, and the specific value can be adjusted by the implementer himself.

[0097] In the current period of each advertisement, all accesses except the malicious accesses are used as the normal accesses of each advertisement, so as to distinguish the normal access behavior from the malicious traffic attack.

[0098] S4: In each storage node, based on the degree of access frequency of each advertisement by each user and the number of payment marks, combined with the page view time and access depth of each advertisement when it is normally accessed, the storage priority of each advertisement is obtained; the storage content is redistributed according to the storage priority of the advertisement.

[0099] The data cached in each edge storage node meets the requirement of quick response. Therefore, for the efficient situation that can be generated by normally accessing advertisements, the priority analysis of all advertisements can be carried out. The advertisements with higher storage priority are cached in the edge storage node closer to the traffic police, so that the demand for their advertisement delivery can be quickly responded to, and the cached data related to the advertisements with lower priority degree is removed from the edge storage node to the cloud storage, so as to make full use of the storage resources.

[0100] The higher the access frequency of the advertisement by the user and the higher the effective degree it brings, the greater the beneficial degree of the advertisement arrival. Inefficient advertisement access behaviors usually have a shorter access duration. The corresponding user access behaviors are usually to exit immediately after access, and there are fewer jump behaviors. On the contrary, if a certain advertisement access behavior has multi-level page jumps, and the user has a greater scrolling depth and longer stay time on the page, it means that the user is more interested in the content output by the advertisement, and the more likely it is to occur a purchase behavior, that is, the corresponding current advertisement access behavior is more effective.

[0101] Preferably, in the embodiment of the present invention, the method for obtaining the storage priority includes:

[0102] For any normal access to any advertisement, the ratio of the number of times the corresponding user accesses the advertisement in the current period to the total number of times the corresponding user is served the advertisement is used as the access interest index for the normal access. The more consistent the number of clicks on the served advertisement by the user, the higher the degree of the user's access interest in the advertisement is reflected.

[0103] Furthermore, the mean value of the browsing time of each page during the normal access is multiplied by the mean value of the access depth of each page to obtain the behavior interest index for the normal access. The longer the browsing time of the page and the greater the exploration depth of the page, the higher the degree of interest in the advertisement.

[0104] By performing a negative correlation mapping on the malicious behavior index of the normal access, the behavior reliability of the normal access is obtained. The reliability of each access participating in the effective analysis is reflected by the degree of malicious behavior. The smaller the malicious behavior index, the higher the normal and reliable degree of the behavior.

[0105] Further, the behavior reliability, access interest index, and behavior interest index of the normal access are multiplied to obtain the effective index of the normal access. When the behavior reliability, access interest index, and behavior interest index are higher, it reflects that the interest value generated by the user's access to the advertisement this time is higher, and the effectiveness of the advertisement is greater.

[0106] In each storage node, the total number of times each advertisement is accessed by each user is used as the node access degree, and the ratio of the total number of times the advertisement is normally accessed to the node access degree is used as the node access frequency of the advertisement. The high-frequency situation of the advertisement being accessed by each user is reflected by the proportion of the advertisement in the total number of accessed advertisements. The higher the node access frequency, the higher the possibility of being efficiently accessed again in the future.

[0107] Finally, the mean value of the effective indexes of all normal accesses of the advertisement, the product of the number of payment marked quantities, and the node access frequency are calculated to obtain the storage priority of the advertisement in the storage node. When the degree of interest that the corresponding advertisement in the node can generate is higher, the frequency of being accessed and paid is higher, and the frequency in each advertisement access is higher, the storage priority of the advertisement is higher.

[0108] Based on the storage priority of each advertisement, the cache content of each storage node is adjusted. In the embodiment of the present invention, in each storage node, the storage priority of the advertisement is normalized to obtain the storage retention index. When the storage retention index of the advertisement is greater than the preset storage threshold, the relevant data of the corresponding advertisement is retained in the storage node, otherwise the advertisement-related data is transmitted to the cloud storage. The storage content of the edge storage point can be dynamically allocated regularly, such as updated once per minute.

[0109] In summary, the present invention analyzes the temporal growth trend of the advertisement access volume in the current period, combines the deviation of the access volume in the same time period of the historical period and the stability of the access growth rate, calculates the abnormal probability index, and preliminarily screens the suspected abnormal time periods with relatively abnormal sudden increase in traffic, providing a time period basis for subsequent distinguishing malicious traffic behaviors. In the suspected abnormal time periods, further through the correlation analysis of the page jump sequence and the length difference from the sequence in the non-abnormal time periods, the jump abnormal index of a single access is quantified. At the same time, combined with the abnormal situation of the user's access behavior in the current period, the normal access behavior is comprehensively determined, and the interference of malicious behaviors on the data storage resource allocation is excluded. Based on the normal access behavior data, the access situation of each advertisement by users, the number of payment marks, the page browsing duration, and the access depth are integrated, and the storage priority of each advertisement is dynamically calculated to realize the real-time optimal allocation of the resources of the edge storage nodes. The present invention identifies abnormal access based on temporal behavior analysis and jump logic differences, dynamically adjusts the storage strategy for advertisements in combination with the user interaction depth of normal access, reduces the resource occupancy waste caused by malicious behaviors, improves the rapid response of effective advertisement data, and enhances the effectiveness of storage resource utilization.

[0110] The present invention also provides a big data storage service system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned big data storage service method are implemented.

[0111] It should be noted that the above sequence of the embodiments of the present invention is only for description and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] Each embodiment in this specification is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and the focus of each embodiment is to illustrate the differences from other embodiments.

Claims

1. A method for big data storage service, characterized in that The method includes: Obtaining the page jump sequence, each page browsing time, access depth, and payment mark when each advertisement is accessed by each user once in each cycle during the historical period; Analyzing the growth trend of the number of times each advertisement is accessed in chronological order in the current cycle to determine the growth period; based on the stability of the increase rate of the number of times accessed during the growth period, combined with the deviation from the number of times accessed in the same period of the historical cycle, obtaining the abnormal probability index; determining the suspected abnormal period of each advertisement according to the abnormal probability index; In the suspected abnormal period of each advertisement, obtaining the jump abnormality index of a single access through the deviation of the page jump sequence length between a single access and other accesses in the non-suspected abnormal period and the association of each jump; according to the abnormal probability index of each access of the corresponding user in the current cycle for a single access, and the similarity between the page jump sequence and the non-suspected abnormal period, combined with the access frequency, obtaining the user abnormality index of a single access; determining the normal access of each advertisement in the current cycle according to the user abnormality index and the jump abnormality index; In each storage node, obtaining the storage priority of each advertisement through the access frequency degree of each advertisement by each user and the number of payment marks, combined with the page browsing time and access depth when the advertisement is normally accessed; reallocating the stored content according to the storage priority of the advertisement.

2. The method for a big data storage service according to claim 1, wherein The method for obtaining the growth period includes: For any advertisement, taking the total number of times the advertisement is accessed at each moment in chronological order as the access volume at each moment; In the current cycle, calculating the difference between the access volume at each moment and the previous moment of the advertisement as the growth index of the advertisement at each moment; screening out the moments when the growth index is less than the value 0, and recording the remaining moments as target moments; The period composed of consecutive target moments in the current cycle is used as the growth period of the advertisement.

3. The method for a big data storage service according to claim 2, wherein, The method for obtaining the abnormal probability index includes: For any advertisement, successively taking each growth period as the analysis period; For any cycle outside the current cycle in the historical period, obtaining the co-distribution period in the cycle by analyzing the distribution of the analysis period in the current cycle; taking the difference between the average access volume of the advertisement in the analysis period and the average access volume of the advertisement in the co-distribution period as the access deviation between the analysis period and the cycle; Taking the average value of the access deviations between the analysis period and all other cycles in the historical period as the historical access deviation degree of the advertisement in the analysis period; After calculating the difference in the growth index between every two adjacent moments of the advertisement in the analysis period, obtaining the sum of all growth index differences and performing a negative correlation mapping to obtain the fluctuation stability degree of the advertisement in the analysis period; Combining the average value of the growth index of the advertisement at all moments in the analysis period, the historical access deviation degree, and the fluctuation stability degree, obtaining the abnormal probability index of the advertisement in the analysis period.

4. The method for a big data storage service according to claim 1, wherein The method for obtaining the jump abnormality index includes: Take every two adjacent pages in the page jump sequence during a single visit as a jump group, and take the page with a larger sequence number in the jump group in the page jump sequence as the jump page; according to the distribution order of the jump groups in the page jump sequence, arrange the jump pages of the jump groups to obtain the jump behavior sequence of a single visit. For any visit of each advertisement during the suspected abnormal period, take all visits of the advertisement corresponding to this visit during all non-suspected abnormal periods as reference visits. Successively take each sequence number in the jump behavior sequence of this visit as the target sequence number; take the total number of visits with the target sequence number existing in the jump behavior sequence of the reference visit as the same jump times of the target sequence number; count the total number of visits with the target sequence number existing in the jump behavior sequence of the reference visit and the corresponding jump pages of the target sequence number being the same as the same behavior times of the target sequence number; take the ratio of the same behavior times of the target sequence number to the same jump times as the jump correlation index of the target sequence number. Perform a negative correlation mapping on the abnormal possible index of the suspected abnormal period where this visit is located to obtain the normal analysis possible index of this visit. Take the product of the jump correlation index of the target sequence number and the normal analysis possible index of this visit as the correlation feature index of the target sequence number in this visit; calculate the mean value of the correlation feature indexes of all sequence numbers in the jump behavior sequence of this visit, and perform a negative correlation mapping to obtain the jump correlation abnormal index of this visit. Calculate the difference between the length of the page jump sequence during this visit and the mean value of the lengths of all page jump sequences in the reference visits to obtain the jump times abnormal index of this visit. Combine the jump correlation abnormal index and the jump times abnormal index of this visit to obtain the jump abnormal index of this visit.

5. The method for a big data storage service according to claim 1, wherein The method for obtaining the user abnormal index includes: Take the first two pages in the page jump sequence during a single visit as an analysis group, and successively incorporate the subsequent pages in the page jump sequence into the analysis group. Each time an incorporation is made, a new analysis group is obtained. For any visit of any advertisement during the suspected abnormal period, calculate the occurrence frequency of each analysis group of this visit during all non-suspected abnormal periods to obtain the jump fixed possible index of each analysis group; take the maximum jump fixed possible index in this visit as the similar jump index of this visit. Take the user corresponding to this visit as the analysis user; in the current cycle, take the ratio of the total number of visits of the analysis user to this advertisement to the total number of visits of all users to this advertisement as the access frequency index of the analysis user; calculate the mean value of the abnormal possible indexes of all visits of the analysis user to obtain the access abnormal index of the analysis user; calculate the mean value of the similar jump indexes of all visits of the analysis user and perform a negative correlation mapping to obtain the jump order abnormal index of the analysis user. Combine the access frequency index, access abnormal index, and jump order abnormal index of the analysis user to obtain the user abnormal index of this visit.

6. The method for a big data storage service according to claim 1, characterized in that The method for determining a normal visit includes: Normalize the product of the jump abnormal index and the user abnormal index of a single visit as the malicious behavior index of a single visit; take the visit with the malicious behavior index greater than the preset malicious threshold as the malicious visit of each advertisement. In the current cycle of each advertisement, all accesses except malicious accesses are regarded as normal accesses to each advertisement.

7. The method for a big data storage service according to claim 6, wherein The method for obtaining the storage priority includes: For any normal access to any advertisement, the ratio between the number of times the corresponding user accesses the advertisement in the current cycle and the total number of times the corresponding user is served the advertisement is used as the access interest index of this normal access; Multiply the average value of the browsing times of each page during this normal access by the average value of the access depths of each page to obtain the behavior interest index of this normal access; perform a negative correlation mapping on the malicious behavior index of this normal access to obtain the behavior reliability of this normal access; Multiply the behavior reliability, access interest index, and behavior interest index of this normal access to obtain the effective index of this normal access; In each storage node, the total number of times each advertisement is accessed by each user is used as the node access degree; the ratio between the total number of times the advertisement is normally accessed and the node access degree is used as the node access frequency of this advertisement; Calculate the mean value of the effective indexes of all normal accesses to this advertisement and the product of the number of payment marked quantities and the node access frequency to obtain the storage priority of this advertisement in the storage node.

8. The method for a big data storage service according to claim 1, wherein The re - allocation of storage content according to the storage priority of the advertisement includes: In each storage node, normalize the storage priority of the advertisement to obtain the storage retention index; When the storage retention index of the advertisement is greater than the preset storage threshold, retain the relevant data of the corresponding advertisement in the storage node, otherwise transmit the advertisement - related data to cloud storage.

9. The method for a big data storage service according to claim 1, wherein, The method for determining the suspected abnormal period includes: Among all the growth periods of each advertisement, the growth periods with the abnormal probability index greater than the preset abnormal threshold are regarded as suspected abnormal periods.

10. A big data storage service system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a big data storage service method as described in any one of claims 1 - 9.

Images