Resource scheduling method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a resource scheduling method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: receiving an application programming interface (API) request sent by a terminal, wherein the API request comprises a device fingerprint identifier of the terminal; querying a historical operation record according to the device fingerprint identifier of the terminal and the user identity information; clustering according to the queried historical operation records of the user to obtain a request group, and determining that redundant and repeated requests exist in the request group; and for redundant and repeated requests, a request range for merging and multiplexing is determined, for requests with consistent parameters, a cache result is directly multiplexed, and for requests with different parameters but related services, the requests are submitted to a task queue for batch processing. According to the method and the device, the occupation of server resources by redundant requests can be effectively reduced, and the overall performance and the user experience of API services can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to the technical field of resource scheduling and cloud computing, and specifically to a resource scheduling method and device, electronic equipment, storage medium and program product. BACKGROUND

[0002] When switching between mobile devices and desktop devices, the slight behavioral differences of users can affect the specific micro-service API(Application Programming Interface, application programming interface) call mode. When users switch between different devices, some repeated or similar API calls may be generated, such as obtaining user information, querying order lists, etc. However, how to utilize these data to achieve intelligent resource scheduling is a technical problem to be solved. SUMMARY

[0003] The present disclosure provides a resource scheduling method and device, electronic equipment, storage medium and program product.

[0004] In a first aspect, the present disclosure provides a resource scheduling method, comprising:

[0005] receiving an application programming interface (API) request sent by a terminal, wherein the API request comprises a device fingerprint identifier of the terminal;

[0006] querying historical operation records of the user on different devices according to the device fingerprint identifier of the terminal and user identity information;

[0007] clustering API requests of the user on different devices according to the queried historical operation records of the user, obtaining a request group, and determining that there are redundant and repeated requests in the request group;

[0008] for the redundant and repeated requests, determining a request combination that meets a parameter similarity threshold and is consistent with a business association according to a similarity degree of request parameters and a business scenario association;

[0009] for a first type of request with consistent parameters in the request combination, directly reusing a cache result of the first type of request, and for a second type of request with different parameters but related to a business in the request combination, encapsulating request information of the second type of request into a task object and submitting the task object to a task queue for resource scheduling.

[0010] In a second aspect, the present disclosure provides a resource scheduling device, comprising:

[0011] a receiving module configured to receive an application programming interface (API) request sent by a terminal, wherein the API request comprises a device fingerprint identifier of the terminal;

[0012] querying, according to the device fingerprint identifier and the user identity information of the terminal, a historical operation record of the user on different devices;

[0013] clustering, according to the queried historical operation record of the user, API requests of the user on different devices to obtain a request group;

[0014] determining that there are redundant and repetitive requests in the request group;

[0015] determining, for the redundant and repetitive requests, a request combination that meets a parameter similarity threshold and is consistent with a business association according to a similarity degree of request parameters and a business scenario association;

[0016] directly reusing a cache result of a first type of request that is consistent in parameters in the request combination, and encapsulating request information of a second type of request that is different in parameters but related in business into a task object and submitting the task object to a task queue for resource scheduling.

[0017] In a third aspect, an electronic device is provided, including:

[0018] one or more processors;

[0019] The processor is configured to invoke instructions to cause the electronic device to perform the method of the first aspect.

[0020] In a fourth aspect, an electronic device is provided, including:

[0021] In a fifth aspect, a program product is provided, including at least one of a program and instructions, which are executed by an electronic device to implement the steps of the method of the first aspect.

[0022] According to the technical solution of the present disclosure, the occupation of server resources by redundant requests can be effectively reduced, the system response speed can be improved, unnecessary computing resource consumption can be reduced, the operation cost can be reduced, the resource scheduling speed of the business system can be improved, the quality of the big data product can be improved, and in particular, the response time of the business request platform can be improved, thereby the overall performance and user experience of the API service can be improved.

[0023] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate exemplary embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure.

[0025] Figure 1 is a flowchart of a stereovision generation method according to an exemplary embodiment.

[0026] Figure 2 is a flowchart of a resource scheduling method according to an exemplary embodiment.

[0027] Figure 3 is a flowchart of a resource scheduling method according to an exemplary embodiment.

[0028] Figure 4 is a flowchart of a resource scheduling method according to an exemplary embodiment.

[0029] Figure 5 is a block diagram of a resource scheduling apparatus according to an exemplary embodiment.

[0030] Figure 6 is a block diagram of a resource scheduling apparatus according to an exemplary embodiment.

[0031] Figure 7 is a block diagram of a resource scheduling apparatus according to an exemplary embodiment.

[0032] Figure 8 is a block diagram of a resource scheduling apparatus according to an exemplary embodiment.

[0033] Figure 9 is a block diagram of an apparatus 900 for resource scheduling according to an exemplary embodiment. DETAILED DESCRIPTION

[0034] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description of the exemplary embodiments is intended to apply to various alternative embodiments of the present disclosure. The following description of exemplary embodiments is only meant to be illustrative of the present disclosure. It is not intended to limit the scope of the present disclosure to the exact details shown.

[0035] The embodiments of the present disclosure are not exhaustive, but only illustrate some embodiments, and are not specific limitations on the protection scope of the present disclosure. In the case of no contradiction, each step in an embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily, for example, the scheme after removing part of the steps in an embodiment can also be implemented as an independent embodiment, and the order of the steps in an embodiment can be exchanged arbitrarily, in addition, the optional implementation in an embodiment can be combined arbitrarily; in addition, the embodiments can be combined arbitrarily, for example, part or all of the steps of different embodiments can be combined arbitrarily, an embodiment can be combined with the optional implementation of other embodiments.

[0036] In each embodiment of the present disclosure, the terms and / or descriptions between the embodiments are consistent if there is no special description and logical conflict, and can be referred to each other, and the technical features in different embodiments can be combined to form a new embodiment according to their inherent logical relationship.

[0037] It should be noted that the acquisition, transmission, storage, use, processing and the like of data in the technical solutions of the present disclosure comply with the relevant provisions of national laws and regulations, and do not violate public order and good customs.

[0038] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present disclosure are authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0039] It should be noted that in the embodiments of the present disclosure, some existing industry solutions such as software, components, models and the like may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0040] The resource scheduling method, device, electronic device, storage medium and program product of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0041] It should be noted that the execution subject of the resource scheduling method of the embodiments of the present disclosure can be a resource scheduling device, which can be realized by software and / or hardware, and the device can be configured in an electronic device, which can include but is not limited to a server end and the like.

[0042] Figure 1 is a flowchart of a stereovision generation method according to an exemplary embodiment, as shown in Figure 1 The resource scheduling method can include but is not limited to the following steps.

[0043] In step 101, an API (Application Programming Interface) request sent by a terminal is received, which can include a device fingerprint identification of the terminal.

[0044] In some embodiments, the terminal sending the API request described above can be understood as a user terminal (also referred to as a user device). In some embodiments, the device fingerprint identification described above can be generated based on key information of the user terminal. In some embodiments, the key information can include, but is not limited to, one or more of the following: hardware configuration information of the user terminal, software environment data, network parameter data, etc. For example, the key information can include hardware configuration information of the user terminal, software environment data, and network parameter data. Optionally, the hardware configuration information can include CPU (Central Processing Unit) model, memory size, graphics card model, etc.; the software environment data can include operating system version number, installed application list, etc.; and the network parameter data can include network connection type, IP (Internet Protocol) address, DNS (Domain Name System) server, etc. It is worth noting that the device fingerprint identification described above can not only indicate the key information of the user terminal of the electronic device, but also strengthen the user identity verification through the device fingerprint identification.

[0045] In some embodiments, the device fingerprint identification can be generated by the user terminal based on the key information of the user terminal. In other embodiments, the device fingerprint identification can be generated by the server side based on the key information of the user terminal, and the server side can send the device fingerprint identification to the corresponding user terminal, so that the user terminal carries the device fingerprint identification in the API request when sending the API request.

[0046] In a possible implementation, the optional implementation of generating the device fingerprint identifier based on the key information of the user terminal includes the following steps: device information collection library (such as DeviceInfo) can be used to extract device characteristic values from the hardware configuration information, software environment data and network parameter data of the terminal; the device fingerprint identifier is generated according to the device characteristic values combined with a hash algorithm (such as SHA256 hash algorithm). Optionally, an encryption algorithm (such as AES-256 encryption algorithm, but not limited to) can be used to encrypt the device fingerprint identifier to obtain an encrypted device fingerprint identifier; the encrypted device fingerprint identifier is sent to the server side through the HTTPS (Hypertext Transfer Protocol Secure, HyperText Transfer Protocol Secure) protocol; after the server side receives the encrypted device fingerprint identifier, a preset key (such as AES-256 key) is used to decrypt the encrypted device fingerprint identifier to obtain the original device fingerprint identifier; it is queried in the device fingerprint database (such as MongoDB device fingerprint database, but not limited to) whether the original device fingerprint identifier exists; if the original device fingerprint identifier does not exist in the device fingerprint database, the original device fingerprint identifier is stored in the device fingerprint database and marked as a new device; if the original device fingerprint identifier exists in the device fingerprint database, the last active time of the original device fingerprint identifier is updated.

[0047] Exemplarily, the CPU model, memory size, graphics card model and other hardware configuration information of the user terminal can be acquired, and the operating system version number, installed application list and other software environment data can be collected, and the network connection type, IP address, DNS server and other network parameters can be read, and the device information collection library is used to extract unique device characteristic values from the raw data. According to the device characteristic values, a unique device fingerprint identifier is generated by combining a SHA256 hash algorithm, and an AES-256 encryption algorithm is used to encrypt the device fingerprint, and the encrypted device fingerprint identifier is packaged with the current timestamp, random salt value (such as salt value) and other parameters to form a standardized API request load. The API request load is sent to the server side through the HTTPS protocol, and the server side receives the request and parses the load content, extracts the encrypted device fingerprint identifier, and uses a preset AES-256 key to decrypt it to obtain the original device fingerprint identifier, and queries the MongoDB device fingerprint database to determine whether the fingerprint identifier already exists. If the device fingerprint identifier does not exist in the database, it is stored in the database and marked as a new device. Optionally, the device risk assessment module can be triggered to perform an initial security score on the device, and a decision tree algorithm is used to calculate a security score of 0-100 according to the device type, operating system version, network environment and other factors. If the device fingerprint identifier already exists, the last active time of the device fingerprint identifier is updated, and the security score is adjusted according to the historical behavior data, and factors such as login frequency, geographic location change, use time period can be considered when adjusting the security score.

[0048] For example, when embedding a device fingerprint collection module in the client SDK (Software Development Kit) of a user terminal, first call the operating system API to obtain device hardware information, including CPU model (such as Intel Core i7-1165G7), memory size (such as 16GB), hard disk serial number (such as WD-WXA1A1234567), etc. Then read the operating system version number (such as Windows 1021H2), installed software list (such as Microsoft Office, Adobe Photoshop). Then get the IP address (such as 192.168.1.100), MAC address (such as 00:1A:2B:3C:4D:5E), network type (such as WiFi 5GHz) through the network library. Combine these information, use SHA256 hash function to generate a unique device fingerprint identifier, such as "a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6". To ensure data transmission security, AES-256 encryption algorithm can be used to encrypt the device fingerprint identifier, and the encryption key can be a preset string, such as "SecretKey123456". Pack the encrypted device fingerprint identifier with current timestamp (such as 1629784800), random salt value (such as "randomSalt9876") and other parameters into JSON (JavaScript Object Notation) format API request payload. Send the API request to the server side through HTTPS protocol. After receiving the API request, the server side decrypts the device fingerprint identifier using the same AES-256 algorithm and key. Compare the decrypted original device fingerprint identifier with the records stored in the database to determine whether it is a new device. If it is a new device, store it in the device_fingerprints collection of the MongoDB database, and calculate the initial security score using the risk assessment algorithm based on device attributes, with a score range of 0-100. If the device fingerprint identifier already exists, update the last active time field and adjust the security score based on historical behavior data such as login frequency and operation type using weighted average algorithm.

[0049] In step 102, according to the device fingerprint identifier of the terminal and the user identity information, the historical operation records of the user on different devices are queried.

[0050] In some embodiments, the historical operation records of the user on different devices can be queried in a server-side user behavior log database according to the device fingerprint identifier and the user identity information of the terminal. For example, the user identity information can be carried in the API request sent by the terminal. In some embodiments, the historical operation records can include operation time stamps, operation type identifiers and operation object content features within a preset time length.

[0051] In a possible implementation, an API request carrying a device fingerprint identifier and user identity information is acquired, which can be sent by a user terminal; a SQL query statement is constructed according to the user identity information and the device fingerprint identifier, which can include a user identifier, a device fingerprint identifier and a time range condition; the SQL query statement is executed in a user behavior log database to obtain historical operation records of the user on multiple devices; and the historical operation records are filtered by time range to extract operation time stamps, operation type identifiers and operation object content features, wherein the operation object content features can include a data type, a data size and a data content digest of the operation. Optionally, the extracted operation time stamps, operation type identifiers and operation object content features can be processed by grouping and sorting algorithms to obtain a multi-device user behavior time series data set; wherein the multi-device user behavior time series data set can contain operation sequences of each device, and each operation record can contain a time stamp, an operation type, an operation object feature and a device identifier.

[0052] Exemplarily, the device fingerprint identifier and the user identity information can be extracted from the API request, and the original user identity information is directly used as a query condition. According to the user identity information and the device fingerprint identifier, a SQL query statement is constructed, such as "SELECT * FROM user_behavior_logs WHERE user_id = 'user ID' AND device_fingerprint = 'device fingerprint identifier' AND timestamp >= DATE_SUB(NOW(), INTERVAL 30 DAY)", a query operation is performed in the user behavior log database, and the historical operation records of the user on different devices in the last 30 days are obtained. The query result is filtered in a time range, and only operation records in a preset time length are retained, from which key information such as operation timestamp, operation type identifier and operation object content feature is extracted, and the operation object content feature can include data type, data size, data content summary and the like. Grouping and sorting operations can be used to arrange the extracted key information, first grouped according to the device fingerprint identifier, and then sorted according to the operation type and time sequence in each device group, to generate a multi-device user behavior time series dataset, which contains the operation sequence of each device, and each operation record can contain a timestamp, an operation type, an operation object feature and a device identifier.

[0053] For example, when processing an API request, the device fingerprint (e.g., "DF123456") and user identity information can be extracted from the request header. Then, based on the device fingerprint and user identity information, an SQL query statement is constructed (e.g., "SELECT * FROM user_behavior_logs WHERE user_id = 'user@example.com' AND device_fingerprint IN ('DF123456', 'DF789012') AND timestamp >= DATE_SUB(NOW(), INTERVAL 30 DAY)"), which contains multiple device fingerprints (e.g., 'DF123456', 'DF789012') commonly used by the user. After executing the query, 1000 historical operation records of the user on different devices in the last 30 days are obtained. Key information is extracted from each record, including operation timestamp (e.g., "2023-08-28 14:30:15"), operation type identifier (e.g., "FILE_UPLOAD"), and operation object content features (e.g., {"dataType": "PDF", "fileSize": "2.5MB", "contentHash": "a1b2c3d4e5f6"}). Then, using the pandas library in Python, the data is processed, first grouping the data by device using df.groupby('device_fingerprint'), and then sorting within each group using df.sort_values(['operation_type', 'timestamp']). The final multi-device user behavior time series dataset contains operation sequences for two devices, with 600 records for device "DF123456" and 400 records for device "DF789012", each record formatted as {"timestamp": "2023-08-28 14:30:15", "operation_type": "FILE_UPLOAD", "object_feature": {"dataType": "PDF", "fileSize": "2.5MB", "contentHash": "a1b2c3d4e5f6"}, "device_fingerprint": "DF123456"}. This structured dataset facilitates subsequent behavior analysis and anomaly detection.

[0054] In step 103, based on the queried user historical operation records, the API requests of the user on different devices are clustered to obtain request groups, and it is determined that there are redundant and repeated requests within the request groups.

[0055] In some embodiments, according to the queried user historical operation record, a content similarity-based clustering algorithm can be used to classify the API requests of the user on different devices, identify the request groups with similar functions and similar parameters, and combine the time interval and frequency characteristics of the requests to determine whether there are redundant and repeated requests in the request groups.

[0056] In a possible implementation, API request information in the user historical operation record is acquired, and feature extraction is performed on the API request information to obtain a request feature vector; a content similarity-based clustering algorithm is used in combination with the request feature vector to cluster the API request information in the user historical operation record, to obtain a request group.

[0057] For example, API request information in the user historical operation record can be acquired, and keywords and values are extracted from the URL, parameters, and request body of the API request information; a unified request feature vector is constructed according to the keywords and values, different processing methods are used for different types of parameters: numerical parameters are normalized, discrete type parameters are one-hot encoded, and text type parameters are vectorized using the TF-IDF algorithm. A content similarity-based clustering algorithm (such as a cosine similarity-based K-means clustering algorithm, but not limited thereto) is used to cluster the request feature vector. Optionally, the optimal K value can be determined by the elbow rule, the initial clustering center is set, and the clustering center is adjusted through iterative calculation until the clustering result converges or the preset maximum number of iterations is reached. Time series analysis is performed on the request group obtained by clustering, the time interval and frequency distribution of the requests in the request group are calculated, the sliding window technology is used to identify the high-frequency repeated request mode in a short time, the window size of the sliding window is set to the preset time length, and the sliding step is set to the preset time interval. According to the time interval and frequency characteristics, it is determined whether there are redundant and repeated requests in the request group. If the request interval is less than the preset time threshold and the frequency is higher than the preset frequency threshold, the request is marked as a potential redundant or repeated request. The request similarity between different devices is determined, if the request similarity between two devices exceeds the preset similarity threshold, the request is regarded as the same request, and if the request similarity does not exceed the preset similarity threshold, the request is regarded as a normal request across devices.

[0058] For example, time series analysis is performed on the clustered request groups, the time interval and frequency distribution of requests within the group are calculated, and the sliding window technique is used to identify high-frequency repeated request patterns in a short period of time. The window size is set to 1 hour, the sliding step is 5 minutes, and the time threshold is 10 seconds and the frequency threshold is 6 times per minute based on historical data statistical analysis. According to the time interval and frequency characteristics, it is determined whether there are redundant and repeated requests in the request group. If the request interval is less than 10 seconds and the frequency is higher than 6 times per minute, it is marked as a potential redundant or repeated request. At the same time, considering the request differences between different devices, the request similarity threshold between devices is set to 0.8 to avoid misjudgment of normal requests across devices. If the request similarity between two devices exceeds 0.8, it is considered as the same request. In processing user historical operation records, first extract features from API requests, for example, for the request URL " / api / user / profile", extract the keywords "user" and "profile"; for numerical parameters such as "age=30", normalize to get 0.375 (assuming the age range is 0-80); for categorical parameters such as "gender=male", perform one-hot encoding to get [1, 0]; for text parameters such as "description=active user", use TF-IDF algorithm to get vector [0.3, 0.5, 0.2]. Combine these features into a unified request feature vector [1, 0, 0.375, 0.3, 0.5, 0.2]. Use K-means clustering algorithm to cluster the feature vectors, calculate the within-cluster sum of squares for different K values such as 2 to 10, draw the elbow curve, and determine the optimal K value as 4. Set 4 initial cluster centers, after 50 iterations, the clustering result converges. Perform time series analysis on each cluster group, use a 1-hour sliding window, slide every 5 minutes, and calculate the time interval and frequency of requests within the window. For example, in a certain 1-hour window, 120 requests are found, the average time interval is 30 seconds, and the highest frequency is 8 times per minute. Based on historical data statistics, set the time threshold to 10 seconds and the frequency threshold to 6 times per minute. It is determined that 30 of the requests have an interval of less than 10 seconds, and 40 requests occur in a 5-minute period (exceeding the threshold of 6 times per minute), which are marked as potential redundant or repeated requests. Finally, considering the user's operations on the mobile phone and tablet devices, the cosine similarity of the requests between the two devices is calculated to get 0.85, which exceeds the threshold of 0.8, so these cross-device requests are considered as the same request to avoid misjudgment as abnormal behavior.

[0059] In a possible implementation, the operation type identifier, operation object content feature, and request parameter information of the API request can be vectorized into feature vectors, a similarity threshold is set, requests with a cosine similarity between the feature vectors greater than the threshold are identified, and are classified into a request group with similar functions and similar parameters. In combination with the time interval and frequency features of the requests, it is determined whether there are redundant and repetitive requests in the request group. For example, the operation type identifier is obtained, the TF-IDF algorithm is used to extract features from the operation type identifier, and the operation object content feature is converted into a numerical vector. The request parameter information is processed by feature engineering to obtain a uniform format feature vector. The cosine similarity calculation method is used to compare the feature vectors two by two according to the uniform format feature vector, and the similarity between the feature vectors is determined. The spectral clustering algorithm is used to classify the requests with a similarity greater than a preset threshold into the same group to form a request group set. The time series analysis is performed on each request group in the request group set, and the time interval and frequency features of the requests in the group are calculated. If the time interval is less than a time threshold and the frequency is higher than a frequency threshold, it is determined that there are redundant and repetitive requests.

[0060] For example, when processing an API request, first, the operation type identifier such as "user login" is used to calculate the feature value by using the TF-IDF algorithm, and a vector [0.5, 0.8, 0.3] is obtained. The operation object content feature such as "user ID: 12345" is converted into a numerical vector [1, 2, 3, 4, 5]. The request parameters are processed by feature engineering, for example, the numerical parameter "age: 30" is normalized to 0.375 (assuming that the age range is 0-80), the categorical parameter "gender: male" is one-hot encoded to [1, 0], and the text parameter "description: active user" is converted into [1, 1, 0] by using the bag-of-words model. Finally, a uniform format feature vector [0.5, 0.8, 0.3, 1, 2, 3, 4, 5, 0.375, 1, 0, 1, 1, 0] is generated. Then, the cosine similarity between different requests is calculated, for example, the similarity between two requests is 0.92, which is higher than the preset threshold 0.8, and it is determined that the requests are similar. The spectral clustering algorithm is used to convert the similarity matrix into a graph structure, the eigenvalues and eigenvectors of the Laplacian matrix are calculated, and finally the requests are divided into three groups. The time series analysis is performed on each request group, for example, a request group has 100 requests in 10 minutes, the average time interval is 3 seconds, and the frequency is 20 times per minute, which exceeds the set time threshold 10 seconds and the frequency threshold 6 times per minute, and therefore it is determined that the request group has redundant and repetitive requests.

[0061] In step 104, for the redundant and repetitive requests, a request combination that meets the parameter similarity threshold and is consistent with the business scenario is determined according to the similarity of the request parameters and the association between the business scenarios.

[0062] In some embodiments, the request parameters are vectorized to obtain a vector form of the request parameters, and a similarity calculation is performed on the vector form of the redundant and repetitive requests to obtain a parameter similarity matrix; context information of the redundant and repetitive requests is extracted, and a business scenario feature vector is generated based on the context information; a business scenario correlation between the redundant and repetitive requests is determined according to the similarity between the business scenario feature vectors of the redundant and repetitive requests; and a request combination that meets a parameter similarity threshold and conforms to a business correlation is determined from the redundant and repetitive requests according to the parameter similarity matrix and the business scenario correlation. For example, the request parameters can be converted into a vector form using a TF-IDF algorithm, and the parameter similarity matrix can be calculated using a cosine similarity algorithm. According to a preset business scenario correlation rule, such as "continuous operation of the same user ID" or "query request of the same product ID", the similarity matrix is filtered to screen out a request combination that meets the parameter similarity threshold of 0.9 and conforms to the business correlation.

[0063] In a possible implementation, the optional implementation of vectorizing the request parameters to obtain a vector form of the request parameters and performing a similarity calculation on the vector form of the redundant and repetitive requests to obtain a parameter similarity matrix includes the following steps: extracting parameter name, data type, and value range meta information of the request, constructing a parameter feature vector, calculating the cosine similarity between the parameter feature vectors of each pair of requests to obtain a parameter similarity score, and thus obtaining the parameter similarity matrix.

[0064] For example, the Porter stem extraction algorithm can be used to process the parameter name, and the Porter stem extraction algorithm is used to remove stop words and convert the data type into a numerical code; the value range in the request is normalized, and the normalization includes using the minimum-maximum normalization for numerical parameters, using one-hot encoding for enumeration parameters, and calculating the length proportion for string parameters; the TF-IDF algorithm is used to calculate the weight of each parameter feature, and the TF-IDF algorithm regards each request as a document, the parameter name as a term, and constructs a parameter feature vector; the cosine similarity of the feature vectors of each pair of requests is calculated, and the cosine similarity is calculated by dividing the product of the vector dot product by the vector length to obtain a similarity score; if the similarity score is greater than a preset threshold, it is determined that the two request parameters are similar, and the preset threshold can be determined by clustering analysis of historical request data.

[0065] Exemplarily, parameter information is parsed from the request, extracting each parameter's name, data type, and value range. Parameter names are processed using the Porter stemming algorithm and stop words are removed. Data types are converted to numeric encodings, and value ranges are normalized: numeric parameters are normalized using min-max normalization, enumeration parameters are one-hot encoded, and string parameters are calculated using length ratios. This generates a set of parameter metadata, with missing parameters padded with a special marker -1. Based on this processed metadata, the TF-IDF algorithm is used to calculate the weight of each parameter feature. Each request is treated as a document, with parameter names as terms. A parameter feature vector is constructed and weighted using predefined parameter importance weights. Zero padding is then used to unify all feature vectors to the dimension of the longest vector, forming a complete feature vector for the request. Cosine similarity is calculated for each pair of request feature vectors. A similarity score is calculated by dividing the vector dot product by the product of the vector modulus. The score ranges from 0 to 1, with higher scores indicating greater similarity. Set a similarity threshold, perform cluster analysis on historical request data, and select the average intra-class similarity as the threshold. If the calculated similarity score is greater than the threshold, the two request parameters are judged to be similar, otherwise they are judged to be dissimilar. The judgment result is stored in the similarity matrix for subsequent processing.

[0066] For example, in processing API requests, first parse the parameter information, such as for the request "user_id=123&age=30&city=XXX", extract the parameter names and process them using the Porter stem extraction algorithm to obtain "user", "age", and "city". Convert the data types to numerical encodings: 1 for string type, 2 for integer type, and 3 for floating point type, resulting in [1, 2, 1] here. Normalize the value range: map "age" to the 0-1 interval to obtain 0.375 (assuming an age range of 0-80), and calculate the string length proportion of "city" to obtain 0.6 (assuming the maximum city name length is 15). Calculate the parameter weights using the TF-IDF algorithm, treating the entire request as a document and the parameter names as terms, resulting in the weight vector [0.4, 0.3, 0.5]. Combine the predefined parameter importance weights [0.8, 0.6, 0.7] to weight the feature vector, resulting in [0.32, 0.18, 0.35]. Use zero padding to unify all feature vectors to the longest dimension of 5, forming the final feature vector [0.32, 0.18, 0.35, 0, 0]. Calculate the cosine similarity between the feature vectors of the two requests, resulting in a similarity score of 0.85. By performing K-means clustering (K=5) on 1000 historical request data, select the average intra-class similarity of 0.8 as the threshold. Since 0.85 is greater than 0.8, it is determined that the two request parameters are similar, and the result "1" is stored in the corresponding position of the similarity matrix.

[0067] In one possible implementation, the optional implementation of determining the business scenario correlation between redundant and repetitive requests described above can include the following steps: extracting the context information of the request, constructing a business scenario feature vector, calculating the cosine similarity between the business scenario feature vectors of each pair of requests to obtain a business correlation score, and thereby obtaining the business scenario correlation.

[0068] For example, context information in API request information is obtained, which can include user portrait labels, page classification identifiers, and function module identifiers. An original feature set is established according to the context information, which can include one-hot encoded user portrait labels, numerically encoded page classification identifiers, and function module identifiers. Vectorization processing is performed on the original feature set to obtain a fixed-dimensional feature vector. Normalization processing is performed on the feature vector, and a weighting operation is performed on the feature vector according to a predefined weight to obtain a weighted feature vector. The cosine similarity between the weighted feature vectors is calculated to obtain a business correlation score. If the business correlation score is greater than a preset threshold, it is determined that there is business correlation between the two API requests.

[0069] For example, the context information extracted from the request header, cookie and server log can include user portrait label, page classification, function module, etc. The user portrait label is one-hot encoded, the page classification is converted into a numerical encoding of 0-9, the function module is converted into a numerical encoding of 10-99, and the missing context information is filled with a special value -1 to construct an original context feature set. The Word2Vec algorithm can be used to vectorize the original context features, map the user portrait label, page classification and function module to a 100-dimensional high-dimensional space, obtain a fixed-dimensional feature vector, and process the feature vector using L2 normalization. At the same time, a predefined weight [0.5, 0.3, 0.2] is introduced to weight the features of the user portrait, page classification and function module respectively. The cosine similarity of the business scenario feature vectors of each pair of requests is calculated, and the business association score is obtained by dividing the vector dot product by the product of the vector magnitudes. The score range is between 0 and 1, and the higher the score, the stronger the business scenario association. A business association threshold is set, and the 75th percentile of the intra-class similarity is selected as the threshold by clustering analysis of historical request data. If the calculated association score is greater than the threshold, it is determined that the business scenarios of the two requests have association, otherwise it is determined to be unrelated. The determination result is stored in the business association matrix for subsequent analysis.

[0070] For example, when processing an API request, first extract the user ID "user123" from the request header, get the page classification "product_page" from the cookie, and read the function module "search_function" from the server log. The user portrait label is obtained by querying the user portrait database as ["male", "age_30_40", "high_income"], and one-hot encoding is performed to obtain [1, 0, 0, 1, 0, 1]. The page classification "product_page" is mapped to the numerical value 3, and the function module "search_function" is mapped to 15. These features are vectorized using a pre-trained Word2Vec model (word vector dimension 100) to obtain a 300-dimensional vector. After applying L2 normalization, multiply by the predefined weight [0.5, 0.3, 0.2] to obtain the final feature vector. The cosine similarity of the feature vectors of the two requests is calculated, and the similarity score is 0.82. By K-means clustering (K=10) on 10,000 historical request data, the 75th percentile of the intra-class similarity is calculated as 0.75 as the threshold. Since 0.82 is greater than 0.75, it is determined that the business scenarios of the two requests have association, and the result "1" is stored in the corresponding position of the business association matrix.

[0071] In step 105, for the first type of request with consistent parameters in the request combination, the cached results of the first type of request are directly reused, and for the second type of request with different parameters but related services in the request combination, the request information of the second type of request is encapsulated into a task object and submitted to a task queue for resource scheduling.

[0072] In the embodiments of the present disclosure, the consistent parameters can be understood as the parameter similarity between the requests being greater than or equal to a parameter similarity threshold. That is, the similarity of the parameters between the first type of requests is greater than or equal to the parameter similarity threshold. The second type of request with different parameters but related services in the request combination can refer to the request with a parameter similarity less than the parameter similarity threshold but a business correlation score greater than or equal to a preset threshold.

[0073] Optionally, for the first type of request, a unique identification key can be generated to query the Redis cache, and the cached results of the first type of request are directly reused; if the cached results corresponding to the unique identification key do not exist in the Redis cache, the first type of request is sent to a processing module and the processing result is stored in the cache. For the second type of request with different parameters but related services in the request combination, a task object is obtained by encapsulating the request information; a priority is calculated according to the request time of the task object and a predefined business importance weight, and the task object is submitted to a task queue; the task objects in the task queue are scheduled by a batch processing module in order of priority, a specified number of task objects are taken out from the task queue for parallel execution to obtain an execution result, and the task state is updated according to the execution result and returned.

[0074] For example, in processing redundant and repetitive requests, the TF-IDF algorithm can be used to convert the request parameters into vector form, such as converting "user ID = 1001 & product ID = 2002 & operation type = query" into a vector of [0.5, 0.3, 0.2, 0.4, 0.6]. The cosine similarity between requests is calculated, such as the similarity of two requests is 0.95, which is higher than the preset threshold 0.9. The business scenario association rule is applied, such as "continuous operation of the same user ID", and the continuous 3 query requests of user ID 1001 are filtered out. For requests with consistent parameters, a unique identification key "user_1001_item_2002_query" is generated, and the Redis cache is queried. If it exists and does not exceed the expiration time of 1 hour, the cache result is directly returned. If it does not exist, the request is processed and the result is stored in the Redis in the form of key-value pair, and the expiration time of 3600 seconds is set. For requests with different parameters but related to business, such as 3 different product queries of user 1001, they are encapsulated into task objects, and based on the request time difference within 10 minutes and the predefined query operation weight 0.5, the priority score 8.5 is calculated, and submitted to the task queue. The batch processing module takes out 10 tasks with the highest priority from the queue each time, and executes these query operations in parallel. After completion, the task state is updated to "processed" and the result set is returned.

[0075] In the above embodiment, by carrying the user terminal device fingerprint identification in the API request and sending it to the electronic device, the electronic device can query the user historical operation record according to the device fingerprint identification and the user identity information, and cluster the API requests of the user on different devices according to the queried user historical operation record to obtain request groups with similar functions and similar parameters. When it is determined that there are redundant and repetitive requests in the request group, the request range of the request group is determined to be merged and reused. For requests with consistent parameters, the cache result is directly reused. For requests with different parameters but related to business, they are submitted to the task queue for batch processing. This can effectively reduce the occupation of server resources by redundant requests, improve the system response speed, reduce unnecessary computing resource consumption, improve the resource scheduling speed of the business system, improve the quality of big data products, especially the response time of the business request platform, thereby improving the overall performance and user experience of the API service.

[0076] Figure 2 is a flowchart of a resource scheduling method according to an example embodiment. As shown in Figure 2 the resource scheduling method can include but is not limited to the following steps.

[0077] In step 201, an application programming interface (API) request sent by a terminal is received, and the API request includes a device fingerprint identification of the terminal.

[0078] Optionally, step 201 can be implemented by any of the implementation manners of the embodiments of the present disclosure respectively, and the embodiments of the present disclosure do not make any limitation thereto, and will not be repeated here.

[0079] In step 202, according to the device fingerprint identification of the terminal and the user identity information, the historical operation records of the user on different devices are queried.

[0080] Optionally, step 202 can be implemented by any of the implementation manners of the embodiments of the present disclosure respectively, and the embodiments of the present disclosure do not make any limitation thereto, and will not be repeated here.

[0081] In step 203, according to the queried historical operation records of the user, the API requests of the user on different devices are clustered to obtain request groups, and it is determined that there are redundant and repetitive requests in the request groups.

[0082] Optionally, step 203 can be implemented by any of the implementation manners of the embodiments of the present disclosure respectively, and the embodiments of the present disclosure do not make any limitation thereto, and will not be repeated here.

[0083] In step 204, for the redundant and repetitive requests, according to the similarity of the request parameters and the business scenario correlation, a request combination that meets the parameter similarity threshold and conforms to the business correlation is determined.

[0084] Optionally, step 204 can be implemented by any of the implementation manners of the embodiments of the present disclosure respectively, and the embodiments of the present disclosure do not make any limitation thereto, and will not be repeated here.

[0085] In step 205, for the first type of request in the request combination with consistent parameters, the cached result of the first type of request is directly reused, and for the second type of request in the request combination with different parameters but related to the business, the request information of the second type of request is encapsulated into a task object and submitted to a task queue for resource scheduling.

[0086] Optionally, step 205 can be implemented by any of the implementation manners of the embodiments of the present disclosure respectively, and the embodiments of the present disclosure do not make any limitation thereto, and will not be repeated here.

[0087] In step 206, in the batch processing link of the task queue, the request groups with common parameters are identified from the task queue.

[0088] In the embodiments of the present disclosure, a request set in a task queue can be acquired, similarity data is obtained from the request set by a parameter similarity matrix and a business association matrix, a threshold is determined by using the similarity data by a hierarchical clustering algorithm, and if the similarity is greater than the threshold, the request group with common parameters is identified. For example, a similar request set is extracted from the task queue, a threshold is determined by using a hierarchical clustering algorithm through a parameter similarity matrix and a business association matrix, and a request group with common parameters is identified.

[0089] In step 207, the common parameters of the request group are extracted to generate a parameter template, and a dependent data change period is calculated according to a business process and a data dependency relationship in combination with the parameter template.

[0090] In the embodiments of the present disclosure, the common parameters of the request group can be extracted by using an Apriori algorithm to generate a parameter template. According to a business process graph and a data dependency graph, a data dependency chain of the request group is analyzed by a graph traversal algorithm to obtain a dependent data change period.

[0091] For example, according to a pre-constructed business process graph and a data dependency graph, a data dependency chain of each request group can be analyzed by using a graph traversal algorithm to calculate a dependent data change period.

[0092] In step 208, for a request with a dependent data change period greater than or equal to a preset threshold, the calculation result of the request is stored in a distributed memory database based on a time stamp caching mechanism.

[0093] In some embodiments, the preset threshold can be set based on a data update frequency. For example, the data update frequency can be determined by time series analysis by using an ARIMA model to set the preset threshold. For a request group with a dependent data change period greater than or equal to the preset threshold, a time stamp-based caching mechanism is implemented, for example, the cache validity period is set to 80% of the data change period, the calculation result is stored in a Redis distributed cache, and a proactive invalidation trigger is set.

[0094] In step 209, for a request with a dependent data change period less than a preset threshold, a dynamic caching strategy is used to cache the calculation result of the request.

[0095] In the embodiments of the present disclosure, for the request group with a data change period less than a preset threshold, a dynamic caching strategy is adopted to cache the calculation result of the request. Optionally, the cache duration can be set to 50% of the data change period, and the calculation frequency is improved through a task scheduler based on a priority queue to realize quasi-real-time data updating. The task scheduler dynamically adjusts the execution order according to the request priority and data change frequency to ensure that the data with high priority and frequent change is processed in time. Optionally, in the cache updating process, a double cache mechanism is adopted to avoid read-write conflict, and data consistency can be ensured.

[0096] For example, in the task queue batch processing process, 100 similar requests are first extracted from the queue, the hierarchical clustering algorithm is used to analyze the parameter similarity matrix and the business association matrix, the clustering threshold is determined to be 0.8, and 3 request groups with common parameters are identified. Then, the Apriori algorithm is applied, the support threshold is set to 0.6, the common parameters such as "user_id" and "product_category" are extracted, and the parameter template is generated. Based on the pre-constructed business process graph and data dependency graph, the depth-first search algorithm is used to analyze the data dependency chain of each request group, and the data change period is calculated to be 10 minutes, 30 minutes and 2 hours respectively. Through time series analysis by ARIMA(1,1,1) model, the data updating frequency in the next 1 hour is predicted, and the preset threshold is set to 15 minutes. For the request group with a 2-hour change period, the system realizes Redis cache based on timestamp, the validity period is set to 96 minutes (80% of 2 hours), and a proactive invalidation trigger is set. For the request group with a 10-minute change period, a dynamic caching strategy is adopted, the cache duration is 5 minutes, and the data updating is triggered by the priority queue task scheduler every 2 minutes. The scheduler dynamically adjusts the execution order according to the request priority (1-10) and the data change frequency to ensure that the tasks with a priority greater than 8 or a change period less than 5 minutes are processed in priority. In the cache updating process, a double Buffer mechanism is used to write new data to the standby cache, and after the writing is completed, the master and standby caches are switched atomically to avoid read-write conflict and ensure data consistency.

[0097] In the above embodiments, by identifying the request group with common parameters from the task queue in the batch processing link of the task queue, the data change period is calculated based on the common parameters of the request group combined with the business process and the data dependency relationship, the data change period is compared with the preset threshold, the calculation result of the request with a larger data change period is cached based on the timestamp caching mechanism, and the calculation result of the request with a smaller data change period is cached by using the dynamic caching strategy to realize quasi-real-time data updating and ensure that the frequently changed data is processed in time.

[0098] Optionally, in some embodiments, such as Figure 3As shown, on the basis of the method described in any of the above embodiments, the resource scheduling method further includes but is not limited to the following steps.

[0099] In step 301, the intermediate result data generated by the batch processing link of the task queue is split to form a plurality of independent data subsets.

[0100] In an embodiment of the present disclosure, the consistency hashing algorithm can be used to split the intermediate result data generated by the batch processing link of the task queue into a plurality of independent data subsets. For example, in batch processing, 1 TB of intermediate result data is analyzed first, and the data is split into 100 subsets using the consistency hashing algorithm, each subset being about 10 GB.

[0101] In step 302, for each data subset, a routing strategy based on geographical location and network operator factors is used to select the corresponding microservice cluster node, and the data subset is distributed to the corresponding microservice cluster node in the form of a computing task, so that the microservice cluster node dynamically applies and releases computing resource containers for different types of computing tasks.

[0102] In an embodiment of the present disclosure, the geographical location and network operator information of the microservice cluster node can be obtained, and a database containing the node latitude and longitude coordinates, the data center to which it belongs, and the network operator type can be established. The K-D tree algorithm can be used to match the geographical location to obtain a routing strategy based on geographical location and network operator factors. According to the priority list of the routing strategy, the optimal microservice cluster node can be determined to achieve dynamic task distribution. If the data subset is transmitted to the target node for completion, the computing results of each node can be collected through the MapReduce mode; if the computing results of each node are collected, the final output result can be formed by summarizing.

[0103] For example, to analyze the intermediate result data generated by the batch processing stage, a consistent hashing algorithm is used to split the data into multiple independent data subsets. A suitable partition key, such as user ID or geographic location information, is selected based on data access pattern analysis to ensure uniform distribution of data. A database of geographic location and network operator information for microservice cluster nodes is constructed, recording the latitude and longitude coordinates, data center affiliation, and network operator type of each node. Node status and network performance indicators, including latency, packet loss rate, and throughput, are updated regularly through periodic probing. A routing strategy based on geographic location and network operator factors is designed, using a K-D tree algorithm to optimize geographic location matching and calculate the geographic distance between data subsets and microservice nodes. A priority ranking list is generated by combining network operator matching degrees. A dynamic task distribution mechanism is implemented, selecting the optimal microservice cluster node based on the priority ranking list of the routing strategy, and using a weighted round-robin load balancing algorithm to distribute computing tasks and corresponding data subsets to the target node. During data transmission, the LZ4 compression algorithm can be used to reduce data volume, and an incremental synchronization mechanism can be used to reduce bandwidth occupancy. After task execution is complete, the computing results of each node can be collected and aggregated using the MapReduce mode to form the final output.

[0104] For example, in batch processing, 1TB of intermediate result data is analyzed, and the data is split into 100 subsets using a consistent hashing algorithm, with each subset being approximately 10GB. By analyzing data access logs, user ID is selected as the partition key to ensure uniform distribution of data. An information database containing 1000 microservice nodes is constructed, recording the latitude and longitude, data center affiliation, and network operator of each node. Network probing is performed every 5 minutes to update node status and performance indicators such as latency (<50ms), packet loss rate (<0.1%), and throughput (>100Mbps). The routing strategy uses a K-D tree algorithm to divide the geographic space into multiple regions, quickly matching the nearest nodes, for example, for data subsets located in City A, the nearest City B node may be selected as the preferred option. The task distribution uses a weighted round-robin algorithm, with weights based on the current load and network conditions of the nodes, such as a performance better node with a weight of 3, a general node with a weight of 2, and a high load node with a weight of 1. Data transmission uses the LZ4 algorithm for compression, with an average compression ratio of 2:1, compressing 10GB of data to 5GB. Incremental synchronization is used, transmitting only the changed data blocks, typically reducing data transmission by 70%. After task completion, the results are aggregated using the MapReduce mode, with the Map phase processed in parallel on each node and the Reduce phase aggregated on the master node, ultimately generating 500MB of result data.

[0105] In the above embodiments, the intermediate result data generated by the batch processing link is split to form multiple independent data subsets; for each data subset, a routing strategy based on geographical location and network operator factors is used to dynamically select the nearest microservice cluster node for distribution of the computing task, thereby reducing network transmission delay and bandwidth occupancy.

[0106] It should be noted that in some embodiments, on the node server of the microservice cluster, computing resource containers can be dynamically applied and released for different types of computing tasks. The containers can be elastically scaled according to the priority and execution progress of the tasks, and the tasks can be distributed to multiple containers for parallel computing through a load balancing strategy to improve task execution efficiency and system throughput. In a possible implementation, the type and resource requirement information of the computing task is obtained, and a task resource configuration template is constructed according to the information. The template can include resource quotas for CPU-intensive, memory-intensive and IO-intensive tasks. The task resource configuration template can be used to dynamically apply containers of corresponding specifications through a resource scheduler, and the computing task can be deployed to the containers for execution. The importance, time urgency and resource consumption of the computing task can be used to calculate a comprehensive priority score of the computing task, which is used for resource allocation and scheduling decisions. The execution progress and resource utilization of the computing task can be monitored in real time, and if the execution progress is below a preset threshold or the CPU usage exceeds a preset upper limit, the number and specifications of the containers can be dynamically adjusted. A load balancing algorithm based on consistent hashing can be used to split the computing task into multiple subtasks, and the subtasks can be distributed to multiple containers for parallel computing according to the load of each node and the network state.

[0107] For example, according to the type and resource requirement of the computing task, a task resource configuration template can be constructed through historical task data analysis, and different resource quotas can be set for CPU-intensive, memory-intensive and IO-intensive tasks. A Kubernetes-based resource scheduler can be used to dynamically apply containers of corresponding specifications and deploy tasks into the containers for execution. A task priority evaluation mechanism can be established, and a weighted scoring method can be used to calculate the comprehensive priority score of the task by combining factors such as task importance weight 0.4, time urgency weight 0.3 and resource consumption weight 0.3, which can be used for subsequent resource allocation and scheduling decisions. The task execution progress and resource utilization can be monitored in real time, and when the task execution speed is lower than expected or the CPU usage exceeds 80%, the HorizontalPodAutoscaler (HPA) container elastic scaling mechanism is triggered to dynamically adjust the number and specifications of the containers, while considering the upper limit of node resources to avoid excessive expansion. A load balancing algorithm based on consistent hashing can be designed to split large computing tasks into multiple subtasks, and according to the load and network status of each node, the subtasks are allocated to multiple containers for parallel computing. After the task is completed, the container resources can be automatically released by the resource recycler, and the resource pool state can be updated. For task execution exceptions, a failure retry mechanism is implemented, for example, up to 3 retries, and if it still fails, the task is marked as abnormal and the management module is notified.

[0108] For example, on the node server of the microservice cluster, historical task data is first analyzed to construct a resource configuration template: CPU-intensive tasks are configured with 4 cores and 8 GB of memory, memory-intensive tasks are configured with 2 cores and 16 GB of memory, and IO-intensive tasks are configured with 2 cores, 4 GB of memory and 100 GB of SSD. After receiving a new task, the Kubernetes-based resource scheduler immediately applies for a container that meets the requirements if it identifies it as a CPU-intensive computing task. At the same time, the task priority evaluation mechanism uses a weighted scoring method to calculate the priority, for example, a certain task has an importance score of 9 points with a weight of 0.4, a time urgency score of 7 points with a weight of 0.3, and a resource consumption score of 6 points with a weight of 0.3, resulting in a final priority score of 7.5 points. Real-time monitoring of task execution is performed, and when the CPU usage is detected to exceed 80%, the HorizontalPodAutoscaler mechanism is triggered to expand the number of containers from 1 to 3. The load balancer uses a consistent hashing algorithm to split a large computing task into 10 subtasks, which are allocated to 3 containers for parallel computing. If an exception occurs during task execution, the system automatically retries, and after 3 retries, if it still fails, the task is marked as abnormal and an alarm message is sent. After the task is successfully completed, the resource recycler releases the container resources within 30 seconds, updates the resource pool state, and displays that the number of available CPU cores increases by 12 and the available memory increases by 24 GB, and the system throughput increases from 100 tasks / hour to 250 tasks / hour.

[0109] In the above embodiments, the task is distributed to multiple containers for parallel computing through load balancing strategy, which can further improve the efficiency of task execution and system throughput.

[0110] Optionally, in some embodiments, as shown in Figure 4 Based on the method of any of the above embodiments, the resource scheduling method further includes but is not limited to the following steps.

[0111] In step 401, the computing result data distributed on multiple micro-service cluster nodes is merged and integrated, and the result obtained after merging and integrating is cached in a distributed memory database.

[0112] In an embodiment of the present disclosure, the computing result data sent by the multiple micro-service cluster nodes is received, and the computing result data is generated when the node server executes the distributed computing task. The MapReduce operation can be performed according to the computing result data, which can include the classification and preprocessing of data in the Map stage, and the merging of the same data in the Reduce stage to obtain the integrated result data set. The integrated result data set can be written into the Redis cluster, which can use a key-value pair storage structure, wherein the key is generated using the MD5 hash algorithm, and the value can include data content, version number and timestamp.

[0113] For example, a distributed data collector is designed to obtain the computing result data from multiple node servers, which can use the HadoopMapReduce model to merge and integrate the data. The Map stage classifies and preprocesses the data, and the Reduce stage merges the same data to generate the final result data set. The integrated result data is written into the Redis cluster as a distributed memory database, which uses a key-value pair storage structure. The key is generated using the MD5 hash algorithm, and the value includes data content, version number and timestamp, which realizes fast reading and updating operation. Optionally, the optimistic lock mechanism can be used to handle concurrent writing, and the version number is compared to solve the data conflict.

[0114] In step 402, when the received API request hits the cache in the distributed memory database, the hit cache result is returned to respond to the API request.

[0115] In the embodiments of the present disclosure, a data request sent by a user can be acquired, which can be issued when the user queries data. A cache key can be generated using an MD5 hash algorithm according to parameters in the data request, and cache data matching the cache key is searched in a Redis cluster. It is judged whether a modification of an underlying data source occurs, and if the modification of the underlying data source occurs, a modification event can be captured through a database trigger, and change information is sent to a message queue. Memory sharding operations are performed on large data sets, and the large data sets are divided into multiple small storage units, and the storage units are distributed on multiple nodes of the Redis cluster.

[0116] For example, a cache query middleware is constructed, a user request is intercepted, a cache key is generated using an MD5 hash algorithm according to request parameters, matching cache data is searched in a Redis cluster, if a hit is made, a result is directly returned, otherwise the request is forwarded to a computing service. A cache update mechanism is implemented, a modification event of an underlying data source can be captured through a database trigger, change information is sent to a message queue, an incremental update process is triggered, and an LRU (Least Recently Used) algorithm is used to set a cache expiration policy, and expired data is cleaned up regularly. Memory sharding technology is used to optimize cache capacity management, and large data sets are divided into multiple small storage units. Data consistency can be ensured through a two-level cache and version number mechanism, a database is updated first, then a cache is updated, and synchronization of the cache and the data source is ensured.

[0117] For example, in practical applications, the distributed data collector obtains the calculation result data from 50 node servers, each of which generates an average of 100 MB of data. The Hadoop MapReduce model processes these data in parallel on 10 servers, the Map stage classifies the data by user ID, and the Reduce stage combines the data of the same user to finally generate a 5 GB result data set. Data writing is performed by a Redis cluster composed of 20 nodes, and the data is distributed to different nodes using a consistent hashing algorithm. The key of each data uses the MD5 algorithm to generate a 32-bit string, and the value contains the data content in JSON format, an 8-bit version number, and a 13-bit timestamp. The optimistic locking mechanism handles concurrent writes by comparing version numbers, and if the version numbers do not match, it retries up to 3 times. The cache query middleware intercepts user requests, calculates the MD5 hash value of the request parameters as the cache key, and finds the matching data in Redis, with an average query time of less than 5 ms. The database trigger monitors changes to the core table and sends modification events to the Kafka message queue, and three consumers process messages in parallel to update the cache data. The LRU algorithm manages cache expiration, retaining data accessed within the last 7 days, and cleaning expired data every hour. The memory sharding technique splits data sets larger than 1 MB into 256 KB small blocks for storage. The two-level cache mechanism updates the data to the database first, and then updates the Redis cache, with an average time consumption of 20 ms, ensuring data consistency.

[0118] In the above embodiment, the calculation result data distributed on multiple node servers is merged and integrated, and the final result is cached in a distributed in-memory database. When subsequent requests hit the cache, the cached result can be returned directly to reduce the amount of repeated calculation and server response latency. Using incremental update and expiration mechanism can improve the consistency of cache data and underlying data source.

[0119] Figure 5 is a block diagram of a resource scheduling apparatus according to an example embodiment. As shown in Figure 5 the resource scheduling apparatus can include a receiving module 501, a querying module 502, a clustering module 503, a first determining module 504, a second determining module 505, and a scheduling module 506.

[0120] The receiving module 501 is configured to receive an application programming interface (API) request sent by a terminal, the API request including a device fingerprint identifier of the terminal.

[0121] The querying module 502 is configured to query historical operation records of a user on different devices according to the device fingerprint identifier of the terminal and user identity information.

[0122] The clustering module 503 is configured to cluster API requests of a user on different devices according to the queried historical operation records of the user, to obtain a request group.

[0123] In some embodiments, the clustering module 503 is configured to: acquire API request information in the historical operation records of the user, and perform feature extraction on the API request information to obtain a request feature vector; and perform clustering on the API request information in the historical operation records of the user by using a content similarity-based clustering algorithm in combination with the request feature vector, to obtain the request group.

[0124] The first determination module 504 is configured to determine that there are redundant and repetitive requests in the request group. In some embodiments, the first determination module 504 is configured to: perform time series analysis on the request group, and calculate a time interval and a frequency of requests in the request group; and determine that there are redundant and repetitive requests in the request group according to the time interval and the frequency of requests in the request group.

[0125] The second determination module 505 is configured to determine, for the redundant and repetitive requests, a request combination that meets a parameter similarity threshold and conforms to business association according to a similarity degree of request parameters and the business association. In some embodiments, the second determination module 505 is configured to: perform vectorization processing on the request parameters for the redundant and repetitive requests, to obtain a vector form of the request parameters, and perform similarity calculation on the vector form of the redundant and repetitive requests, to obtain a parameter similarity matrix; extract context information of the redundant and repetitive requests, and generate a business scenario feature vector based on the context information; determine business scenario association between the redundant and repetitive requests according to a similarity between the business scenario feature vectors of the redundant and repetitive requests; and determine, from the redundant and repetitive requests, the request combination that meets the parameter similarity threshold and conforms to the business association, according to the parameter similarity matrix and the business scenario association.

[0126] The scheduling module 506 is configured to, for a first type of request in the request combination that has consistent parameters, directly reuse a cache result of the first type of request, and for a second type of request in the request combination that has different parameters but is related in business, encapsulate request information of the second type of request into a task object and submit the task object to a task queue for resource scheduling.

[0127] Optionally, in some embodiments, as shown in FIG. 5B, the system further includes a request parameter extraction module 501 and a request parameter feature extraction module 502. Figure 6As shown, the resource scheduling apparatus can include an identification module 607, an extraction module 608, and a cache module 609. The identification module 607 is configured to identify a request group with common parameters from the task queue at the batch processing link of the task queue. The extraction module 608 is configured to extract the common parameters of the request group to generate a parameter template, and calculate a dependent data change period according to the business process and the data dependency, in combination with the parameter template. The cache module 609 is configured to, for a request with a dependent data change period greater than or equal to a preset threshold, store the calculation result of the request in a distributed in-memory database based on a timestamp-based cache mechanism; and for a request with a dependent data change period less than the preset threshold, cache the calculation result of the request using a dynamic cache strategy. Wherein, Figure 6 The modules 601-606 in Figure 5 The modules 501-506 in

[0128] Optionally, in some embodiments, as Figure 7 As shown, the resource scheduling apparatus can include a splitting module 710 and a distribution module 711. The splitting module 710 is configured to split the intermediate result data generated by the batch processing link of the task queue to form a plurality of independent data subsets. The distribution module 711 is configured to, for each data subset, select a corresponding micro-service cluster node using a routing strategy based on geographical location and network operator factors, and distribute the data subset to the corresponding micro-service cluster node in the form of a calculation task, so that the micro-service cluster node dynamically applies for and releases a calculation resource container for different types of calculation tasks. Wherein, Figure 7 The modules 701-709 in Figure 6 The modules 601-609 in

[0129] Optionally, in some embodiments, as Figure 8 As shown, the resource scheduling apparatus can include an integration module 812 and a response module 813. The integration module 812 is configured to merge and integrate the calculation result data distributed on the plurality of micro-service cluster nodes, and cache the result obtained after the merging and integrating to a distributed in-memory database. The response module 813 is configured to, when a received API request hits the cache in the distributed in-memory database, return the hit cache result to respond to the API request. Wherein, Figure 8 The modules 801-811 in Figure 7 The modules 701-711 in

[0130] As for the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0131] Figure 9is a block diagram of an apparatus 900 for resource scheduling according to an example embodiment. For example, the apparatus 900 can be provided as a server. Referring to FIG. 9, the apparatus 900 includes a processing component 922, a memory 932, a communication component 934, and a bus 950. Figure 9 The processing component 922 includes one or more processors. The memory 932 includes a random access memory (RAM), a read only memory (ROM) and / or another type of memory to store instructions and data. The processing component 922 is configured to execute instructions and to perform processing tasks for the apparatus 900. The bus 950 can include a path that is used to move data between and among the memory 932, the processing component 922, and the communication component 934. The bus 950 can include one or more busses of different types, such as a memory bus or memory column, a peripheral bus, an external bus, a crossbar switch, and / or the like.

[0132] The apparatus 900 can also include a power component 926 configured to perform power management for the apparatus 900, a wired or wireless network interface 950 configured to connect the apparatus 900 to a network, and an input / output (I / O) interface 958. The apparatus 900 can operate based on an operating system stored in the memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0133] In example embodiments, a non-transitory computer readable storage medium is also provided including instructions, such as the memory 932 including instructions, that are executable by the processing component 922 of the apparatus 900 to cause the apparatus 900 to perform the methods described above. For example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and / or optical data storage device, etc.

[0134] In example embodiments, a program product including at least one of a program, instructions, which when executed by an electronic device, implements the steps of the methods described above.

[0135] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the present disclosure cover any and all variations of the features described herein including modifications and adaptations thereof. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0136] It is to be understood that the present disclosure is not limited to the precise construction described and as shown in the attached figures, and that changes and modifications can be effected therein by those skilled in the art without departing from the scope of the disclosure. The scope of the present disclosure is limited solely by the claims that follow.

Claims

1. A resource scheduling method, characterized in that: include: Receiving an application programming interface (API) request sent by a terminal, wherein the API request includes a device fingerprint identifier of the terminal; Query the user's historical operation records on different devices based on the device fingerprint identification and user identity information of the terminal; Clustering the user's API requests on different devices based on the retrieved historical operation records of the user to obtain request groups, and determining whether there are redundant and repeated requests within the request groups; For the redundant and repeated requests, determine a request combination that meets the parameter similarity threshold and conforms to the business relevance based on the similarity of the request parameters and the business scenario relevance; For the first type of requests with consistent parameters in the request combination, the cached results of the first type of requests are directly reused. For the second type of requests with different parameters in the request combination but business-related, the request information of the second type of requests is encapsulated into a task object and submitted to the task queue for resource scheduling.

2. The method according to claim 1, characterized in that Clustering the user's API requests on different devices based on the retrieved user historical operation records to obtain request groups includes: Obtaining API request information from the user's historical operation records, and performing feature extraction on the API request information to obtain a request feature vector; A clustering algorithm based on content similarity is used in combination with the request feature vector to cluster the API request information in the user's historical operation records to obtain the request group.

3. The method according to claim 1, characterized in that Determining whether redundant and repeated requests exist within the request group includes: Performing time series analysis on the request group to calculate the time interval and frequency of requests within the request group; According to the time interval and frequency of requests in the request group, it is determined that redundant and repeated requests exist in the request group.

4. The method according to claim 1, wherein The step of determining, for the redundant and repeated requests, a request combination that meets a parameter similarity threshold and complies with business relevance based on the similarity of request parameters and business scenario relevance, includes: For the redundant and repeated requests, vectorize the request parameters to obtain the vector form of the request parameters, and perform similarity calculation on the vector form of the redundant and repeated requests to obtain a parameter similarity matrix; Extracting context information of the redundant and repeated requests, and generating a business scenario feature vector based on the context information; Determining the business scenario relevance between the redundant and repeated requests based on the similarity between the business scenario feature vectors of the redundant and repeated requests; According to the parameter similarity matrix and the business scenario relevance, a request combination that meets a parameter similarity threshold and conforms to business relevance is determined from the redundant and repeated requests.

5. The method according to claim 1, wherein The method further comprises: In a batch processing phase of the task queue, a request group having common parameters is identified from the task queue; Extracting common parameters of the request group to generate a parameter template, and calculating the dependent data change cycle in combination with the parameter template based on the business process and data dependency relationship; For requests whose dependency data change period is greater than or equal to a preset threshold, a timestamp-based caching mechanism stores the calculation results of the requests in a distributed memory database; For requests whose dependency data change period is less than the preset threshold, a dynamic caching strategy is used to cache the calculation results of the request.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Splitting the intermediate result data generated by the batch processing phase of the task queue to form multiple independent data subsets; For each of the data subsets, a routing strategy based on geographic location and network operator factors is adopted to select the corresponding microservice cluster node, and the data subset is distributed to the corresponding microservice cluster node in the form of computing tasks, so that the microservice cluster node can dynamically apply for and release computing resource containers for different types of computing tasks.

7. The method according to claim 6, characterized in that The method further comprises: Merge and integrate the calculation result data distributed on multiple microservice cluster nodes, and cache the results obtained after the merger and integration in a distributed memory database; When the received API request hits the cache in the distributed memory database, the cache result of the hit is returned to respond to the API request.

8. A resource scheduling device, characterized in that: include: A receiving module, configured to receive an application programming interface (API) request sent by a terminal, wherein the API request includes a device fingerprint identifier of the terminal; A query module, configured to query the user's historical operation records on different devices based on the device fingerprint identifier and user identity information of the terminal; A clustering module is used to cluster the API requests of the user on different devices based on the retrieved user historical operation records to obtain request groups; A first determining module, configured to determine whether redundant and repeated requests exist within the request group; A second determination module is configured to determine, for the redundant and repeated requests, a request combination that meets a parameter similarity threshold and conforms to business relevance based on the similarity of request parameters and business scenario relevance; The scheduling module is used to directly reuse the cached results of the first type of requests with consistent parameters in the request combination, and for the second type of requests with different parameters but business-related in the request combination, encapsulate the request information of the second type of requests into a task object and submit it to the task queue for resource scheduling.

9. An electronic device, characterized in that: include: one or more processors; The processor is configured to call instructions to enable the electronic device to execute the method according to any one of claims 1 to 7.

10. A storage medium storing instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 7.

11. A program product, comprising at least one of a program and instructions, characterized in that: When at least one of the program and the instruction is executed by an electronic device, the steps of the method according to any one of claims 1 to 7 are implemented.

Images