Independent site data processing method and device and electronic equipment

By identifying the source and information of the requested data, determining priority and applying traffic governance policies to process requested data from independent sites, the problem of inefficient processing of requested data is solved, and data processing efficiency and user experience are improved.

CN119003910APending Publication Date: 2024-11-22BUSINESS LINE COMMERCIAL PTE LTD

Patent Information

Application Number
CN202411060842.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Independent sites are inefficient in processing requested data, resulting in reduced efficiency of data services provided by users.

Method used

By identifying the source and information of the requested data, determine the importance of the requested data, and allocate it to the corresponding container according to different priorities, use traffic governance strategies to process the requested data, including blacklist filtering, cache filtering, effect degradation, and local flow restriction. and global current limiting strategies.

Benefits of technology

It improves the efficiency of processing requested data, prioritizes the processing of requested data with high response time requirements, filters redundant information, and improves user experience and system stability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003910A_ABST
    Figure CN119003910A_ABST
Patent Text Reader

Abstract

The invention provides an independent site data processing method and device and electronic equipment. The independent site data processing method comprises the steps of receiving request data; identifying a corresponding request source and request information according to the request data; determining an identifier of the request data according to the request source, wherein the identifier is used for representing the importance degree of the corresponding request data; distributing the request data to a container corresponding to the identifier; and processing the request data according to the flow management strategy in the container, and outputting a request result corresponding to the request data. The invention relates to the technical field of data analysis, and can improve the efficiency of processing request data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and in particular to an independent site data processing method, device and electronic equipment. Background Art

[0002] At present, with the development of the Internet, more and more organizations tend to use Internet platforms to receive requests sent by users and provide corresponding services to users. For example, with the development of e-commerce, the emerging business model of cross-border e-commerce has also developed rapidly in recent years. Independent sites are a new form of cross-border e-commerce. In operation, they have high autonomy and are more flexible, avoiding many constraints of platform rules. Compared with traditional e-commerce platforms, the merchants running online stores in independent sites are also of different types, such as high-quality merchants and ordinary merchants; the volume of users browsing online stores for online transactions is also different. In e-commerce platforms, information services are usually provided to users through the search system of independent sites. Users usually send data operation requests to e-commerce platforms through the search system of independent sites to obtain data services provided by e-commerce platforms. Due to the large number of Internet users in independent sites, independent sites usually receive a large amount of request data from different sources and different contents in a unit time, resulting in reduced stability of the search system, which in turn leads to low efficiency of organizations in providing data services to users. Summary of the invention

[0003] In view of the above, it is necessary to propose an independent site data processing method, device and electronic device to solve the technical problem of low efficiency in processing request data.

[0004] The present application provides an independent site data processing method, which is applied to an electronic device, and the method includes: receiving request data; identifying the corresponding request source and request information according to the request data; determining an identifier of the request data according to the request source, and the identifier is used to characterize the importance of the corresponding request data; allocating the request data to a container corresponding to the identifier; processing the request data according to the traffic management strategy in the container, and outputting a request result corresponding to the request data.

[0005] In some embodiments, determining the identifier of the request data according to the request source includes: when the request source is a first merchant, determining the identifier as a first identifier; when the request source is a second merchant, determining the identifier as a second identifier; when the request source is a user, determining the identifier of the request data as a third identifier; wherein the importance of the request data corresponding to the first identifier is higher than the importance of the request data corresponding to the second identifier, and the importance of the request data corresponding to the second identifier is higher than the importance of the request data corresponding to the third identifier.

[0006] In some embodiments, allocating the request data to the container corresponding to the identifier includes: when the identifier is the first identifier, allocating the request data to the first container; when the identifier is the second identifier, allocating the request data to the second container; when the identifier is the third identifier, allocating the request data to the third container corresponding to the third identifier according to the type of the request information.

[0007] In some embodiments, the types of request information include query requests, screening requests, and aggregation requests.

[0008] In some embodiments, the traffic management strategy includes any one or more of the following: a blacklist screening strategy, a cache screening strategy, an effect degradation strategy, a local flow limiting strategy, and a global flow limiting strategy.

[0009] In some embodiments, the effect degradation strategy includes: determining the corresponding maximum request volume per unit time based on the request source; when the number of request data corresponding to the request source is greater than the maximum request volume per unit time, determining that the request information is a preset request parameter.

[0010] In some embodiments, the local flow limiting strategy includes: determining the maximum request volume per unit time according to the request source; when the number of request data corresponding to the request source is greater than the maximum request volume per unit time, determining the data to be limited from the request data; intercepting the data to be limited; and sending traffic abnormality alarm information to the request source.

[0011] In some embodiments, the global flow limiting strategy includes: when the amount of the request data is greater than the maximum request amount per unit time, determining the data to be limited from the request data; intercepting the data to be limited; and sending traffic abnormality alarm information to the request source.

[0012] An embodiment of the present application also provides an independent site data processing device, which includes: a receiving module for receiving request data; a determination module for identifying a corresponding request source and request information based on the request data; the determination module is also used to determine an identifier of the request data based on the request source, and the identifier is used to characterize the importance of the corresponding request data; the determination module is also used to allocate the request data to a container corresponding to the identifier; the determination module is also used to process the request data according to the traffic management policy in the container, and output a request result corresponding to the request data.

[0013] An embodiment of the present application further provides an electronic device, comprising: a memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the independent site data processing method.

[0014] It can be seen from the above technical solutions that the embodiment of the present application determines the priority of the request data according to the request source and request information in the acquired request data, allocates corresponding traffic management strategies to the request data of different priorities, and then processes the corresponding request data based on the traffic management strategy to obtain the updated request data, and finally outputs the request result corresponding to the updated request data. In this way, the request data with higher response time requirements can be processed first, and different traffic management strategies can be implemented for the request data with different priorities to filter the redundant information in the request data, thereby improving the efficiency of processing the request data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is an application scenario diagram of an independent site data processing method provided in one embodiment of the present application.

[0016] Figure 2 This is an architecture diagram of an independent station provided in one embodiment of the present application.

[0017] Figure 3 It is a flowchart of an independent site data processing method provided by an embodiment of the present application.

[0018] Figure 4 This is a flowchart of a method for determining an identifier of request data provided by an embodiment of the present application.

[0019] Figure 5 It is a flowchart of a method for allocating request data to corresponding containers provided by an embodiment of the present application.

[0020] Figure 6 It is a functional module diagram of an independent site data processing device provided in one embodiment of the present application.

[0021] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to more clearly understand the purpose, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In the following description, many specific details are set forth to facilitate a full understanding of the present application, and the embodiments described are only a part of the embodiments of the present application, rather than all of the embodiments.

[0023] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which the present application belongs. The terms used herein in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0025] The embodiment of the present application provides an independent site data processing method, which can be applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a programmable gate array (Field-Programmable Gate Array, FPGA), a digital processor (Digital Signal Processor, DSP), an embedded device, etc.

[0026] An electronic device can be any electronic product that can interact with a customer, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0027] The electronic device may also include a network device and / or a client device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

[0028] The network where the electronic device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0029] like Figure 1The figure shows an application scenario diagram of an independent site data processing method provided by an embodiment of the present application. The independent site data processing method provided by the present application can be applied to an electronic device 100. Among them, the electronic device 100 can be a device such as a computer or server applied to an independent site. The user 200 using the independent site inputs request data to the electronic device 100. Among them, the request data is used to characterize the user 200's need to operate the data in the independent site. Exemplarily, when the user 200 is a merchant in the independent site, the request data can be used to characterize the merchant's need to aggregate and analyze the data in the independent site; when the user 200 is a consumer in the independent site, the request data can be used to characterize the consumer's need to query the data in the independent site. The electronic device 100 is also used to determine the identifier of the request data according to the request source, allocate the request data to the container corresponding to the identifier, process the request data according to the traffic management policy in the container, and output the request result corresponding to the request data to the user 200.

[0030] like Figure 2 The following is an architecture diagram of an independent site provided by an embodiment of the present application. The independent site 300 provided by the present application can be operated on Figure 1 The electronic device 100 shown is used to provide a trading platform for merchants and users of cross-border e-commerce. The independent site 300 includes a client 310, a business system 320, a search system 330, and a backend 340. Specifically, the client 310 is used to receive the request data input by the user 200 into the independent site, the business system 320 is used to determine the business data to be searched according to the request information in the request data, and the search system 330 is used to determine the corresponding business data from the backend 340 according to the request information, wherein the backend 340 is used to store the business data in the independent site 300.

[0031] The client 310 includes a web page 311, an application 312, and a program interface 313; the web page 311 may be a portal website corresponding to the independent site 300 on the Internet, which is used to access the web page based on a browser. Figure 1 The user 200 shown provides transaction services and data within the independent site 300. The user 200 browses the web page 311 through a browser to use the transaction services in the independent site 300 or browse the data in the independent site 300. The application 312 may be an application running in the electronic device 100. The user 200 may use the transaction services in the independent site 300 or browse the data in the independent site 300 based on the application 312. The program interface 313 may be a program that can be called by other programs or software, and the program interface 313 may be integrated in third-party software. In this way, when using third-party software, the user can call the independent site by calling the program interface 313 to use the transaction services or browse the data in the independent site.

[0032] Among them, the backend 340 includes a distributed search engine 341, a cache engine 342, a configuration center 343 and a traffic control component 344. Among them, the distributed search engine 341 is used to store data in the independent site 300. For example, the data stored in the distributed search engine 341 can be product data, transaction data or user portrait data in the independent site 300; the cache engine 342 is used to cache user queries and search results (or index data and intermediate search results) with high frequency, so as to provide independent data processing acceleration for each physical computer or virtual computer in the independent site 300, which can reduce the server's dependence on slow storage devices such as disks and improve data access speed; the configuration center 343 can be a collection of programs with configuration information storage and management, which is used to centrally manage various configuration information of the website, wherein the configuration information includes system parameters, function options, database connection information, third-party service access credentials, etc. of the independent site 300, which can reduce the dispersion and redundancy of the configuration information of the independent site 300, and thus improve the management efficiency of the configuration information in the independent site 300; the configuration center 343 can also be used to dynamically update the configuration information, so that the new configuration information can be ensured to take effect without restarting the independent site 300, which can improve the flexibility and response speed of the independent site 300. The flow control component 344 can be any software or program framework with flow management functions. For example, the flow control component 344 can be Sentinel software. Specifically, the flow control component 344 is used to execute multiple flow control strategies on the flow data received by the independent site 300, wherein the flow control strategies include QPS flow limiting (limiting the number of requests per second), concurrent flow limiting, thread pool flow limiting, etc. Multiple flow control strategies can control the access volume of the service and prevent the service from crashing due to overload. The flow control component 344 also monitors the number of request data received by the independent site 300 per second, the response time to the request data and other indicators in real time, and can promptly identify abnormal situations of the independent site 300.

[0033] like Figure 3 , is a flowchart of an independent site data processing method provided in an embodiment of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. An independent site data processing method provided in an embodiment of the present application includes the following steps.

[0034] S20, receiving request data.

[0035] In one embodiment of the present application, the request data may be the request data received by the server and input by the user through the client. The request data is used to initiate a request, and is a carrier for the user to initiate a request to the independent site through the client, and is used to perform preset operations on the data in the independent site. The preset operations include but are not limited to adding, deleting, modifying, querying or analyzing the data stored in the independent site. Exemplarily, when an e-commerce platform is running in the independent site, and the user of the e-commerce platform needs to analyze the business data in the e-commerce platform, the user can send the request data to the back-end server of the e-commerce platform through the client of the e-commerce platform, so as to obtain the business data corresponding to the request data in the server by using the request data, so as to analyze the business data to obtain the data analysis results; when the merchants settled in the e-commerce platform of the independent site need to query the product information and perform product analysis, the merchants can input the corresponding request data to the independent site through the client of the independent site, and the search system queries the product data corresponding to the request data from the back-end of the independent site according to the request data, and uses the business system in the independent site to process the product data to realize product analysis.

[0036] Specifically, the request data includes the type of request, the requested resource, and the protocol version used when making the request. The request data includes the request header, which includes the type of client (User-Agent), the type of content the client expects to receive (Accept), authentication information (Authorization), etc. The above information can ensure that the server understands the request data sent by the user to the server through the client and returns the corresponding response data.

[0037] Specifically, the request data also includes the request information that the client of the independent site needs to submit to the server. The request information is located in the request body and exists in the form of form data, JSON, XML and other formats. By submitting the request information, the independent site can implement operations such as creation, deletion, update, and query of backend resources. Some content in the request data can also be used to control the behavior of the request data. For example, the Content-Type field in the request header is used to specify the data type of the request body, which can ensure that the server correctly parses the data in the request body. For another example, the Cache-Control field in the request header can be used to control the cache policy, thereby affecting the response speed and cache effect of the request. Support complex interaction. Through the request data, users can perform complex interaction logic with the server through the client. For example, in a Web application, the client can send an asynchronous request to the server through AJAX technology to update the page content without reloading the entire page. The above interaction method relies on the request data to transmit the request information, receive the response data, and update the page status accordingly. Security and authentication. The request data can also be used to implement security and authentication mechanisms. For example, in an HTTPS request, the request data will be encrypted for transmission to prevent the data from being intercepted or tampered with during transmission. In addition, the Authorization field in the request header can be used to pass authentication information (such as a token) to implement client authentication and authorization.

[0038] Exemplarily, when the request data is in HTTP format, the request data includes specific information sent by the client of the independent site to the backend of the independent site, including: request line, which is the first line of the HTTP request and contains three key information: request method: such as GET, POST, PUT, DELETE, etc., which is used to indicate the operation that the user expects to perform on the resources stored in the backend of the independent site through the client of the independent site. For example, the GET method is used to request resources, the POST method is used to submit data, and the DELETE method is used to delete certain data. Specifically, when the request data is in HTTP format, and when the request method recorded in the request line is the GET method, the request data is used to represent the user's need to request data resources from the backend of the independent site through the client of the independent site. For example, when the user is a merchant settled in the independent site, the merchant can query the product information stored in the backend of the independent site by sending the request data to the client; when the request method recorded in the request line is the POST method, the request data is used to represent the user's need to submit data to the backend of the independent site through the client of the independent site. For example, when the user is a merchant settled in the independent site, the merchant can upload the product data to be put on the shelf to the backend of the independent site for storage by sending the request data to the client. The request line also includes a uniform resource locator (URL) to indicate the specific path of the requested resource. Specifically, the URL contains a query string (Query String) for passing additional parameters. The request line also includes an HTTP protocol version: such as HTTP / 1.1, indicating the HTTP protocol version used by the request. Request Header The request header contains additional information about the request, expressed in the form of a key-value pair. This information helps the server understand the specific requirements of the request and the client's environment. Common request headers include: Host: Specifies the host name and port number of the requested resource. User-Agent: Identifies the client program that initiates the request, usually containing the client type, version, and operating system information. Accept: The content type that the client can receive, such as text / html, application / json, etc. Content-Type: When the request body contains data, this header is used to specify the MIME type of the data, such as application / x-www-form-urlencoded (form data), application / json (JSON data), etc. Authorization: Contains credentials for authentication, such as the Bearer token Referer: Indicates the URL of the request source page, used for anti-hotlinking and statistics, etc. Request Body: The request body is optional and contains specific data sent by the client to the server.This data usually exists in the form of form data, JSON, XML and other formats. The request body is usually used with request methods such as POST and PUT to submit data to the server. There is a blank line between the request header and the request body, which is used to indicate that the request header has ended and the subsequent content is the content of the request body. The above information together constitutes the entire content of the HTTP request, enabling the client to communicate and exchange data effectively with the server.

[0039] S21, identifying a corresponding request source and request information according to the request data.

[0040] In an embodiment of the present application, the request data may correspond to different data types, and thus the request data may have different data formats, wherein the request source and the request information are located at different positions in the request data of different data formats.

[0041] Exemplarily, when the data format of the request data is HTTP format, the first request line in the request data is used to characterize the type of the request data (such as GET or POST), the URI (Uniform Resource Identifier) ​​of the request, and the version of the HTTP protocol (such as HTTP / 1.1); the request header is located after the request line, and the format of the request header is a key-value pair of "field name: field value", the field name and the field value are separated by a colon, and one or more spaces are included before the field value; all the content after the request header is the request information. Specifically, the request header in the HTTP request data contains information about the request data itself and the client environment, so the source of the request and the request information can be identified based on the HTTP data request header. For example, the request header of the HTTP request data contains a field named User-Agent, and the information corresponding to this field is used to characterize the client information that initiates the HTTP request data, such as the browser type, version, operating system, etc., so the request source can be determined based on the information corresponding to this field in the request header. Specifically, the request information corresponding to the request data can be determined based on the data in the request body.

[0042] Exemplarily, when the request data is in the URL format, the request source can be the IP address in the URL data; Device ID: For mobile applications or IoT devices, the device ID (such as UUID, IMEI, etc.) can be used as a unique identifier; User ID: If the request is initiated by the user and the system can identify the user (such as through a login token, Cookie, etc.), the user ID can be used as part of the identifier; API key: For API requests that require authentication, the API key can also be used as part of the request source to generate the identifier. The request information can be the specific content in the URL request, including: request method (such as GET, POST, PUT, etc.), different request methods may correspond to different business logic, so the request method can be part of the identifier; Request path or URL: The URL (or path part) of the request can indicate the requested resource or operation; Request parameters: The query parameters in the URL or the parameters in the request body can be used as part of the identifier; Request header: Some request headers (such as Content-Type, User-Agent, Authorization, etc.) contain information used to process the request, so they can be used to generate the identifier.

[0043] S22: Determine an identifier of the request data according to the request source, where the identifier is used to represent the importance of the corresponding request data.

[0044] In one embodiment of the present application, an independent site may receive a large amount of request data from different users in a short period of time, some of which have high timeliness requirements, while other request data have lower timeliness requirements. For example, when a user sends request data for changing a password to the backend of the independent site through the client of the independent site to change the login password corresponding to the user's account on the independent site, in order to ensure that the modified login password can take effect immediately to improve the security of the user's account, it can be determined that the timeliness corresponding to the request data is high; for example, when a user sends request data for adding data to the backend of the independent site through the client of the independent site to add data to the independent site, since the result of adding data does not need to be displayed or used in real time, it can be determined that the timeliness corresponding to the request data is low. When processing request data, in order to determine the priority of each request data among the multiple request data received in a short period of time, thereby determining the priority of subsequent processing of the request data, and then determining the traffic management strategy that needs to be applied to the request data in the process of processing each request data according to the priority, the identifier of the request data can be determined according to the request source, and the identifier is used to characterize the importance of the corresponding request data.

[0045] In one embodiment of the present application, the request source is used to characterize the identity of the user who sends the request data. When a large amount of request data is received in a short period of time, the importance of the request data can be determined according to the identity of the user who sends the request data. For example, when the request data is the request data sent by the user to the independent site through the client of the independent site, if the request source of the request data indicates that the user who sends the request data is an important merchant in the independent site, then the request data is processed with the highest priority; if the request source of the request data indicates that the user who sends the request data is an ordinary merchant settled in the independent site, then the request data is processed with a relatively high priority; if the request source of the request data indicates that the user who sends the request data is a user who conducts transaction activities through the independent site, then the request data is processed last. Among them, the important merchant in the independent site can be the merchant with the highest sales volume settled in the independent site, or the merchant with the highest product evaluation in the independent site, or the merchant with the highest number of transactions, which is not limited in this application. Among them, the ordinary merchant can be the remaining merchants in the independent site except the important merchants. Since the sales volume of ordinary merchants may be lower than that of important merchants, or the product evaluation of ordinary merchants is lower than that of important merchants, the importance of the request data submitted by ordinary merchants is lower than that of the request data submitted by important merchants. Among them, ordinary users who are not merchants in independent sites may be consumers who use independent sites to conduct transaction activities or data retrieval activities. Since consumer users have lower requirements for the timeliness of response data feedback when conducting product retrieval or product transactions, the importance of request data submitted by consumer users is lower than that of request data submitted by ordinary merchants.

[0046] Exemplarily, the identifier of the request data may be req_high_level, req_medium_level, and req_low_level, wherein req_high_level is used to indicate that the request data corresponding to the identifier is of the highest importance; req_medium_level is used to indicate that the request data corresponding to the identifier is of medium importance; and req_low_level is used to indicate that the request data corresponding to the identifier is of the lowest importance.

[0047] In one embodiment of the present application, for a specific method of determining the identifier of the request data according to the request source, see Figure 4 The corresponding detailed description.

[0048] S23, allocating the request data to a container corresponding to the identifier.

[0049] In one embodiment of the present application, the container may be a cache subspace pre-set in the cache, for storing a pre-set traffic management policy. Different containers correspond to different identifiers, and different containers store different traffic management policies. In order to apply different traffic management policies to different request data, the request data may be allocated to the container corresponding to the identifier.

[0050] In one embodiment of the present application, determining the traffic management policy corresponding to the request data includes: parsing the request identifier. First, it may be necessary to parse the request identifier to extract information useful for policy matching. For example, if the request identifier contains a user ID, device ID, or API path, the above information can be used to narrow the search scope of the policy match; use the parsed request identifier information to find a matching policy in the pre-stored traffic management policy set. In the process of searching for policies, the policy set can be traversed, and the following operations are performed on each policy: condition matching, confirming whether the conditions of the traffic management policy match the identifier (or part thereof) of the request data; the conditions of the traffic management policy can be based on user ID, device ID, API path, request time, request volume and other factors; priority sorting: if multiple policies match, which policy to apply can be determined based on the priority of the policy, where the priority can be specified by the policy configurator, so that policy conflicts caused by all policies being at the same priority can be avoided; applying the policy, after determining the matching traffic management policy, the policy can be applied to the processing of the request data. Specifically, the processing of the request data The process includes: load balancing, distributing requests to different backend server instances according to the strategy; current limiting, confirming whether the request data exceeds the preset rate limit, and rejecting or delaying the request accordingly; circuit breaking, if the backend service is unstable, deciding whether to fuse the service and return a backup response according to the strategy; downgrading, when the system load is too high or resources are insufficient, downgrading the request according to the strategy; logging and monitoring, recording detailed information of requests and responses according to the strategy for auditing and performance monitoring; processing responses, processing requests according to the applied traffic management strategy, and generating corresponding response data, where the response data includes normal data, error messages or specific status codes to indicate the processing results of the request.

[0051] In one embodiment of the present application, for a specific method of allocating request data to a container corresponding to an identifier, see Figure 5 The corresponding detailed description.

[0052] S24: Process the request data according to the traffic management policy in the container, and output a request result corresponding to the request data.

[0053] In one embodiment of the present application, when determining the container corresponding to the request data, the request source, the content of the request information, and the overall architecture and requirements of the system can be comprehensively considered. Specifically, the traffic management strategy includes any one or more of the following: blacklist screening strategy, cache screening strategy, effect degradation strategy, local flow limiting strategy, and global flow limiting strategy.

[0054] In one embodiment of the present application, the blacklist screening strategy includes: when the request source of the received request data is in the pre-set blacklist, a blacklist alarm message is output and the operation corresponding to the request data is blocked; when the request source of the received request data does not exist in the pre-set blacklist, the operation corresponding to the request data is continued. The blacklist is used to record the request data corresponding to the user name, user ID or specific type that is blocked or controlled, so that abnormal request data from certain ports can be blocked, and the stability of the server can be ensured when it is attacked by the network.

[0055] In one embodiment of the present application, the cache screening strategy includes: when the request source of the received request data is in the pre-set cache source, it indicates that the request data hits the cache strategy, and the request data can be cached; when the request source of the received request data is not in the pre-set cache source, it indicates that the request data does not hit the cache strategy, and there is no need to cache the request data, and the operation corresponding to the request data can continue. Among them, the cache source is used to indicate the request source of the request data that needs to be cached. When the request source corresponding to a certain request data is in the cache source, it indicates that the request data issued by the request source does not need high real-time performance. It is mainly to judge whether a single request uses cache, cache key generation method, cache time, etc. For scenarios where data real-time performance is not high, such as C-end search list pages and category pages, using cache can greatly improve response speed and reduce the pressure on the underlying Elasticsearch engine.

[0056] In one embodiment of the present application, the effect degradation strategy includes: determining the corresponding maximum request amount per unit time according to the request source; when the number of request data corresponding to the request source is greater than the maximum request amount per unit time, determining that the request information is a preset request parameter. Specifically, when the service load is too high or an abnormal situation occurs, temporarily shut down certain functions or services to ensure the availability of core functions. Set priorities, set service priorities according to business importance, and give priority to ensuring the operation of core services when resources are insufficient. Functional downgrade, shut down or reduce requests for non-core functions to ensure the overall stability of the system.

[0057] In one embodiment of the present application, the local flow limiting strategy includes: determining the maximum request amount per unit time according to the request source; when the number of request data corresponding to the request source is greater than the maximum request amount per unit time, determining the data to be limited from the request data; intercepting the data to be limited; sending traffic abnormality alarm information to the request source to control and limit the request traffic to prevent the system from being overwhelmed by too many requests.

[0058] In one embodiment of the present application, the global flow limiting strategy includes: when the number of request data is greater than the maximum number of requests per unit time, determining the data to be limited from the request data; intercepting the data to be limited; sending a traffic abnormality alarm message to the request source to set the maximum concurrency number, limit the number of requests processed simultaneously, and ensure that the system does not crash due to too many requests.

[0059] In one embodiment of the present application, the traffic management strategy also includes one or more of the following: QPS limit, setting a query rate per second (QPS) limit to prevent burst traffic from impacting the system. Token bucket algorithm, using the token bucket algorithm to smooth burst traffic, allowing the system to still process some requests at traffic peaks. Fuse, when a service fails or is abnormal, quickly cut off the request to the service to avoid affecting the entire system. Set error thresholds and time windows, when the error rate of the service exceeds the preset threshold, the fuse opens and rejects the request, reducing access to the faulty service. Half-open state, within a period of time after the fuse is opened, enter the half-open state and tentatively let go of a portion of the traffic to check whether the service has been restored. Fault tolerance, improve the system's tolerance for unreliable networks or services, and ensure the continuous availability of the system. Retry mechanism, retry when the request fails, and improve the success rate of the request. Timeout control, set the timeout time for the service call to avoid a single request from occupying resources for a long time. Compensation operation, roll back or perform other compensation operations when an error occurs to ensure data consistency and integrity. Traffic coloring and grayscale release: color the traffic at the API gateway layer to achieve grayscale release and link tracking. Traffic coloring: add different labels (such as version number, user group, etc.) to requests from different sources or types so that they can be distinguished and processed in the system. Gradually launch the new version of the service to some users, observe its performance and stability, and ensure that the version upgrade is gradually completed without affecting most users. Monitoring and early warning: monitor the operating status and performance indicators of the system in real time to discover and handle potential problems in a timely manner. Establish a monitoring system and use tools such as Prometheus and Grafana to monitor the system's key indicators such as CPU, memory, and network in real time. Set early warning thresholds: trigger an early warning when the system performance indicators exceed the preset thresholds so that timely intervention and processing can be carried out. When determining the traffic governance strategy corresponding to the request data, it is necessary to comprehensively consider multiple aspects such as load balancing, current limiting, circuit breaking, degradation, fault tolerance, traffic coloring and grayscale release, as well as monitoring and early warning. By reasonably combining and configuring these strategies, it can be ensured that the system can maintain a stable and efficient operating state when facing various complex requests.

[0060] In one embodiment of the present application, a method for processing request data is determined according to a traffic governance policy to generate updated request data and corresponding response data. Specifically, request data is processed based on the traffic governance policy, and the traffic governance policy is evaluated. Among them, the traffic governance policy evaluation includes: load balancing: selecting a suitable backend service instance to process the request according to the load balancing policy; current limiting: checking whether the request exceeds the preset current limiting threshold. If exceeded, the request may be directly rejected or queued; fuse: check whether the target service is in a fuse state, if so, it may directly return a fuse response or redirect to a backup service; downgrade: decide whether to downgrade the function according to the system load and the importance of the request; fault tolerance: set the timeout time and retry mechanism of the request, and prepare to retry when the request fails; traffic coloring and gray release: perform traffic coloring according to the source or characteristics of the request, and decide whether to route the request to the new version of the service; request processing: if the request passes all traffic governance policy checks, the server will process the request according to the normal process. This may include verifying the request data, querying the database, calling other services, etc. Modify the metadata or parameters in the request according to the traffic management policy, such as adding specific request headers, modifying routing information, etc.

[0061] In one embodiment of the present application, the request result corresponding to the updated request data is the response data generated after processing the request data. If a traffic management policy (such as fuse, downgrade, etc.) is applied during the processing, the response data includes information related to the traffic management policy, such as an error code, an error message, or a downgraded response content. The generated response data is sent to the client.

[0062] In this way, the request data received by the independent site from different data sources in a short period of time can be processed in layers. By limiting concurrent requests, the number of connections or the transmission rate, the independent site can be prevented from receiving too much request data and causing performance degradation or system crash; it can also help the independent site to use resources more effectively and improve the efficiency of computing resource utilization in the independent site; it can also ensure that the computing resources in the independent site are tilted towards the request data sent by more important or more demanding users, avoiding some users from causing excessive pressure on the entire system, thereby improving the overall service quality, ensuring the stability and real-time nature of data transmission, and improving the user experience; on the other hand, flow control can effectively prevent certain types of attacks, such as DDoS attacks. By limiting malicious traffic, flow control can protect the system from attacks and enhance the security of the system.

[0063] Exemplarily, the client sends an HTTP GET request to the server, requesting a certain resource. The server applies traffic management policies, such as load balancing, current limiting, etc. The request successfully passes all policy checks and is distributed to a suitable backend service instance for processing. Request reception and policy evaluation: The server receives the request and evaluates whether it meets the requirements of the traffic management policy (such as load balancing, current limiting, etc.). If the request passes the policy evaluation, the server forwards the request to the backend service instance for processing. The backend service instance performs necessary operations (such as querying the database, calculating, etc.) and prepares response data. The backend service instance generates response data based on the processing results, such as HTTP status code, response header, and response body. The server sends the generated response data to the client.

[0064] It can be seen from the above technical solutions that the embodiment of the present application determines the priority of the request data according to the request source and request information in the acquired request data, allocates corresponding traffic management strategies to the request data of different priorities, and then processes the corresponding request data based on the traffic management strategy to obtain the updated request data, and finally outputs the request result corresponding to the updated request data. In this way, the request data with higher response time requirements can be processed first, and different traffic management strategies can be implemented for the request data with different priorities to filter the redundant information in the request data, thereby improving the efficiency of processing the request data.

[0065] like Figure 4 As shown, it is a flowchart of a method for determining an identifier of request data according to the request source provided by an embodiment of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. The method for determining an identifier of request data according to the request source provided by an embodiment of the present application includes the following steps.

[0066] S30: When the request source is the first merchant, determine that the identifier is the first identifier.

[0067] In one embodiment of the present application, the first merchant is used to represent the merchant with the highest importance in the e-commerce platform or the trading platform. For example, the first merchant can be the merchant with the highest sales volume in the e-commerce platform, or the first merchant can also be the merchant with the highest score in the e-commerce platform. The present application does not limit the specific setting method of the first merchant.

[0068] In an embodiment of the present application, since the request source of the request data corresponding to the first identifier is the first merchant, the request data corresponding to the first identifier has the highest importance.

[0069] S31: When the request source is a second merchant, determine that the identifier is a second identifier.

[0070] In one embodiment of the present application, the second merchant is used to represent a merchant with higher importance in the e-commerce platform or trading platform, and the importance of the second merchant is lower than that of the first merchant. For example, the second merchant can be an ordinary merchant in the e-commerce platform. The present application does not limit the specific setting method of the first merchant.

[0071] In one embodiment of the present application, since the request source of the request data corresponding to the second identifier is the second merchant, the importance of the request data corresponding to the second identifier is higher, and the importance of the request data corresponding to the second identifier is lower than the importance of the request data corresponding to the first identifier.

[0072] S32: When the request source is a user, determine that the identifier of the request data is a third identifier.

[0073] In one embodiment of the present application, the request source is used by the user to characterize that the corresponding request data is the request data sent by an ordinary user in the e-commerce platform or the trading platform, so it can be determined that the identifier of the request data is the third identifier. Among them, the importance of the user is lower than that of the first merchant and the second merchant, so the request data corresponding to the third identifier is the request data with the lowest importance. Specifically, the importance of the request data corresponding to the first identifier is higher than the importance of the request data corresponding to the second identifier, and the importance of the request data corresponding to the second identifier is higher than the importance of the request data corresponding to the third identifier.

[0074] In one embodiment of the present application, the types of the request information include query request, screening request and aggregation request. Specifically, when the request information is a query request, the identifier is determined to be the third identifier; when the request information is a screening request, the identifier is determined to be the fourth identifier; when the request information is an aggregation request, the identifier is determined to be the fifth identifier.

[0075] like Figure 5 , is a flowchart of a method for allocating request data to a container corresponding to an identifier provided by an embodiment of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. The method for allocating request data to a container corresponding to an identifier provided by an embodiment of the present application includes the following steps.

[0076] S40: When the identifier is the first identifier, allocate the request data to a first container.

[0077] In one embodiment of the present application, when the identifier of the request data is the first identifier, it indicates that the request source of the request data is an important merchant in the e-commerce platform or trading platform. The request data corresponding to the first identifier can be allocated to the first container, wherein the first container is used to store the traffic management policy applied to the request data corresponding to the first identifier.

[0078] S41: When the identifier is the second identifier, allocate the request data to a second container.

[0079] In one embodiment of the present application, when the identifier of the request data is the second identifier, it indicates that the request source of the request data is an ordinary merchant in an e-commerce platform or a trading platform. The request data corresponding to the second identifier can be allocated to the second container, wherein the second container is used to store the traffic management policy applied to the request data corresponding to the second identifier.

[0080] S42: When the identifier is the third identifier, allocate the request data to a third container corresponding to the third identifier according to the type of the request information.

[0081] In one embodiment of the present application, when the identifier of the request data is the third identifier, it indicates that the request source of the request data is an ordinary user in the e-commerce platform or trading platform. The request data corresponding to the third identifier can be allocated to the third container, wherein the third container is used to store the traffic management policy applied to the request data corresponding to the third identifier.

[0082] See also Figure 6 , Figure 6 1 is a functional module diagram of an independent site data processing device provided in an embodiment of the present application. The independent site data processing device 51 includes a receiving module 510 and a determining module 511. The module / unit referred to in the present application refers to a series of computer-readable instruction segments that can be executed by the processor 13 and can complete fixed functions, which are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0083] The receiving module 510 is used to receive request data.

[0084] The determination module 511 is used to identify the corresponding request source and request information according to the request data.

[0085] The determining module 511 is further configured to determine an identifier of the request data according to the request source, wherein the identifier is used to represent the importance of the corresponding request data.

[0086] The determining module 511 is further configured to allocate the request data to a container corresponding to the identifier.

[0087] The determination module 511 is further used to process the request data according to the traffic management policy in the container and output a request result corresponding to the request data.

[0088] In some embodiments, the determination module 511 is also used to determine that the identifier is the first identifier when the request source is the first merchant; to determine that the identifier is the second identifier when the request source is the second merchant; and to determine that the identifier of the request data is the third identifier when the request source is the user; wherein the importance of the request data corresponding to the first identifier is higher than the importance of the request data corresponding to the second identifier, and the importance of the request data corresponding to the second identifier is higher than the importance of the request data corresponding to the third identifier.

[0089] In some embodiments, the determination module 511 is also used to allocate the request data to the first container when the identifier is the first identifier; allocate the request data to the second container when the identifier is the second identifier; and allocate the request data to the third container corresponding to the third identifier according to the type of the request information when the identifier is the third identifier.

[0090] See also Figure 7 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 100 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 is used to execute the computer-readable instructions stored in the memory to implement an independent site data processing method described in any of the above embodiments.

[0091] In an embodiment of the present application, the electronic device 100 further includes a bus, a computer program stored in the memory 12 and executable on the processor 13, such as an independent site data processing program.

[0092] Figure 7 Only the electronic device 100 having the memory 12 and the processor 13 is shown, and those skilled in the art can understand that Figure 7 The structure shown does not constitute a limitation on the electronic device 100 , and the electronic device 100 may include fewer or more components than shown in the figure, or combine some components, or arrange the components differently.

[0093] Combination Figure 3 The memory 12 in the electronic device 100 stores a plurality of computer-readable instructions to implement the independent site data processing method, and the processor 13 can execute the plurality of instructions to implement: receiving request data; identifying the corresponding request source and request information according to the request data; determining an identifier of the request data according to the request source, the identifier being used to characterize the importance of the corresponding request data; allocating the request data to a container corresponding to the identifier; processing the request data according to the traffic management strategy in the container, and outputting a request result corresponding to the request data.

[0094] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 3 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0095] Those skilled in the art will appreciate that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100. The electronic device 100 may be a bus-type structure or a star-type structure. The electronic device 100 may also include more or less other hardware or software than shown in the diagram, or a different arrangement of components. For example, the electronic device 100 may also include input and output devices, network access devices, etc.

[0096] It should be noted that the electronic device 100 is only an example, and other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and included here by reference.

[0097] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium can be non-volatile or volatile. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 100, such as a mobile hard disk of the electronic device 100. In other embodiments, the memory 12 can also be an external storage device of the electronic device 100, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 100. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 100, such as a code of an independent site data processing program, etc., but also can be used to temporarily store data that has been output or is to be output.

[0098] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 100, and uses various interfaces and lines to connect various components of the entire electronic device 100, and executes or executes programs or modules stored in the memory 12 (for example, executing independent site data processing programs, etc.), and calls data stored in the memory 12 to execute various functions of the electronic device 100 and process data.

[0099] The processor 13 executes the operating system of the electronic device 100 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned embodiments of the independent site data processing method, for example Figure 3 Steps shown.

[0100] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 100. For example, the computer program may be divided into a receiving module 510 and a determining module 511.

[0101] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, and includes a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute a part of an independent site data processing method described in each embodiment of the present application.

[0102] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware devices to complete through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0103] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory and other memory, etc.

[0104] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0105] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. Figure 7 Only one arrow is used in the figure, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0106] An embodiment of the present application also provides a computer-readable storage medium (not shown), in which computer-readable storage medium is stored computer-readable instructions, and the computer-readable instructions are executed by a processor in an electronic device to implement the independent site data processing method described in any of the above embodiments.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0108] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0109] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0110] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. An independent site data processing method, applied to electronic equipment, characterized in that: The method comprises: Receiving request data; wherein the request data is used to perform a preset operation on the data in the independent site; Identify the corresponding request source and request information according to the request data; Determine an identifier of the request data according to the request source, where the identifier is used to represent the importance of the corresponding request data; Allocating the request data to a container corresponding to the identifier; Process the request data according to the traffic management policy in the container, and output the request result corresponding to the request data.

2. The independent site data processing method according to claim 1, characterized in that: Determining the identifier of the request data according to the request source includes: When the request source is the first merchant, determining the identifier to be the first identifier; When the request source is a second merchant, determining the identifier to be a second identifier; When the request source is a user, determining that the identifier of the request data is a third identifier; The importance of the request data corresponding to the first identifier is higher than that of the request data corresponding to the second identifier, and the importance of the request data corresponding to the second identifier is higher than that of the request data corresponding to the third identifier.

3. The independent site data processing method according to claim 2, characterized in that: The allocating the request data to the container corresponding to the identifier comprises: When the identifier is the first identifier, allocating the request data to a first container; When the identifier is the second identifier, allocating the request data to a second container; When the identifier is the third identifier, the request data is allocated to a third container corresponding to the third identifier according to the type of the request information.

4. The independent site data processing method according to claim 3, characterized in that: The types of request information include query requests, screening requests and aggregation requests.

5. The independent site data processing method according to claim 1, characterized in that: The traffic management strategy includes any one or more of the following: blacklist screening strategy, cache screening strategy, effect degradation strategy, local flow limiting strategy and global flow limiting strategy.

6. The independent site data processing method according to claim 5, characterized in that: The effect degradation strategy includes: Determine the maximum number of requests per unit time according to the request source; When the amount of request data corresponding to the request source is greater than the maximum request amount within the unit time, the request information is determined to be a preset request parameter.

7. The independent site data processing method according to claim 5, characterized in that: The local current limiting strategy includes: Determine the maximum number of requests per unit time according to the request source; When the amount of request data corresponding to the request source is greater than the maximum amount of requests within the unit time, determining the data to be limited from the request data; Intercepting the data to be limited; Send traffic anomaly alarm information to the request source.

8. The independent site data processing method according to claim 5, characterized in that: The global current limiting strategy includes: When the amount of the request data is greater than the maximum request amount per unit time, determining the data to be limited from the request data; Intercepting the data to be limited; Send traffic anomaly alarm information to the request source.

9. An independent site data processing device, characterized in that: The device comprises a module for implementing the independent site data processing method according to any one of claims 1 to 8, and the device comprises: A receiving module, used for receiving request data; A determination module, used to identify a corresponding request source and request information according to the request data; The determining module is further used to determine an identifier of the request data according to the request source, wherein the identifier is used to represent the importance of the corresponding request data; The determination module is further used to allocate the request data to the container corresponding to the identifier; The determination module is also used to process the request data according to the traffic management policy in the container and output the request result corresponding to the request data.

10. An electronic device, characterized in that: The electronic device comprises a processor and a memory, and the processor is used to implement the independent site data processing method as claimed in any one of claims 1 to 8 when executing a computer program stored in the memory.

Citation Information

Patent Citations

  • Transaction request grouping processing method and device, electronic equipment and storage medium

    CN116028225A

  • Server traffic control method and device, processor and electronic equipment

    CN116633866A

Cited By

  • Third-party tool docking operation and maintenance and management and control method and system of independent station

    CN119203256A