Data processing method and device
By labeling and routing simulated traffic data to separate processing devices, the method addresses the issue of excessive load on production machines in online micro-financing verification systems, enhancing system efficiency and accuracy.
Patent Information
- Application Number
- CN202211110128.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-13
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-09-13
AI Technical Summary
Current verification schemes in online micro-financing do not distinguish between verification traffic and production customer traffic, leading to excessive load on production machines and potential CPU overload.
A data processing method that labels simulated traffic data according to preset rules and routes it to different processing devices, while real traffic data is forwarded to production machines, reducing the load on these devices by isolating verification traffic.
The method effectively reduces the load on production machines by processing simulated traffic on separate devices, ensuring accurate verification without overburdening production systems.
Smart Images

Figure CN115484313B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present specification relate to the field of data processing technology, and in particular, to a data processing method. Background Art
[0002] The risk control domain is the underlying core of online merchants' micro-financing. A large number of production changes occur daily, including offline data switching, policy model release, etc. In order to protect production safety and prevent the impact of changes that do not meet expectations, we automatically return production logs, call the production environment and the pre-effective environment in real time during the pre-effective stage of the change, and compare the results returned by the two to achieve the purpose of verifying the correctness of the change in advance, which is what we call "traffic simulation."
[0003] The current verification scheme design does not distinguish between verification traffic and production customer traffic, and both are run on production machines. Therefore, it will affect the load of production machines. Excessive verification traffic may even cause the machine CPU and load to be too high. Summary of the invention
[0004] In view of this, the embodiments of this specification provide three data processing methods. One or more embodiments of this specification also relate to a data processing device, a computing device, a computer-readable storage medium and a computer program to solve the technical defects existing in the prior art.
[0005] According to a first aspect of an embodiment of this specification, a data processing method is provided, including:
[0006] Determine initial flow data, wherein the initial flow data includes simulated flow data and real flow data;
[0007] The simulated traffic data is labeled according to preset labeling rules, and according to preset routing rules, the labeled simulated traffic data is forwarded to the first type of processing device and the second type of processing device, and the real traffic data is forwarded to the first type of processing device.
[0008] According to a second aspect of an embodiment of this specification, a data processing method is provided, including:
[0009] Determine initial traffic data for different servers, wherein the initial traffic data includes simulated traffic data and real traffic data;
[0010] The simulated traffic data is labeled according to preset labeling rules, and according to preset routing rules, the labeled simulated traffic data is forwarded to the first type server and the corresponding second type server, and the real traffic data is forwarded to the corresponding first type server.
[0011] According to a third aspect of an embodiment of this specification, a data processing method is provided, including:
[0012] Determine initial traffic data of different clients, wherein the initial traffic data includes simulated traffic data and real traffic data;
[0013] Processing the real traffic data and labeling the simulated traffic data according to a preset labeling rule;
[0014] According to the preset routing rules, the marked simulation traffic data is forwarded to the second server and the third server.
[0015] According to a fourth aspect of an embodiment of this specification, there is provided a data processing device, including:
[0016] A data determination module is configured to determine initial flow data, wherein the initial flow data includes simulated flow data and real flow data;
[0017] The data forwarding module is configured to label the simulated traffic data according to preset labeling rules, and forward the labeled simulated traffic data to the first type processing device and the second type processing device according to preset routing rules, and forward the real traffic data to the first type processing device.
[0018] According to a fifth aspect of the embodiments of this specification, a computing device is provided, including:
[0019] Memory and processor;
[0020] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above data processing method are implemented.
[0021] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned data processing method are implemented.
[0022] According to a seventh aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.
[0023] The embodiments of this specification provide a data processing method and device, wherein the data processing method includes: determining initial traffic data, wherein the initial traffic data includes simulated traffic data and real traffic data; marking the simulated traffic data according to a preset marking rule, and forwarding the marked simulated traffic data to a first type of processing device and a second type of processing device according to a preset routing rule, and forwarding the real traffic data to the first type of processing device. By receiving the initial traffic data and marking the simulated traffic data therein, the simulated traffic can be forwarded to the first type of processing device and the second type of processing device according to the marking, so that the first type of processing device and the second type of processing device can process it, reducing the load of the first type of device that processes the real data. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a scenario diagram of a data processing method provided by an embodiment of this specification;
[0025] Figure 2 is a flow chart of a data processing method provided by an embodiment of this specification;
[0026] Figure 3a is a flow chart of another data processing method provided by an embodiment of this specification;
[0027] Figure 3b is a flow chart of another data processing method provided by an embodiment of this specification;
[0028] Figure 3c It is a scenario diagram of another data processing method provided by an embodiment of this specification;
[0029] Figure 4a is a structural schematic diagram of a data processing device provided by an embodiment of this specification;
[0030] Figure 4b is a schematic diagram of the structure of another data processing device provided by an embodiment of this specification;
[0031] Figure 4c is a structural schematic diagram of another data processing device provided by an embodiment of this specification;
[0032] Figure 5 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION
[0033] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.
[0034] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0035] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0036] First, the terms involved in one or more embodiments of this specification are explained.
[0037] Traffic simulation: Automatically return production logs. During the pre-effective stage of changes, the production environment and the pre-effective environment are called in real time, and the results returned by the two are compared to verify the correctness of the changes in advance.
[0038] Pre-effective change: The stage before the change is officially implemented for customers, which will not have any impact on production.
[0039] IP address: (Internet Protocol Address) refers to the Internet Protocol address, also translated as Internet Protocol address.
[0040] Internet Protocol version 4: (English: Internet Protocol version 4, IPv4), also known as the fourth version of the Internet Communication Protocol, is the fourth revised version in the development process of the Internet Protocol and the first widely deployed version of this protocol.
[0041] IPv6: It is the abbreviation of "Internet Protocol Version 6" in English. It is the next-generation IP protocol designed by the Internet Engineering Task Force (IETF) to replace IPv4.
[0042] RPC protocol: a protocol that requests services from a remote computer program over the network without having to understand the underlying network technology.
[0043] In this specification, three data processing methods are provided. This specification also involves a data processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
[0044] See also Figure 1 , Figure 1 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes a traffic pool 111, a client 120, an intermediate machine 130, a checking machine 140, a production machine 150 and a production machine 160.
[0045] The user sends real traffic data to the intermediate machine 130 through the client 120, and the intermediate machine 130 obtains simulated traffic data from the task table 112 of the traffic pool 111. The intermediate machine 130 determines whether the traffic data is simulated traffic data or real traffic data based on the port number and other information of the obtained traffic data, and marks the simulated traffic data to forward the simulated traffic data to the verification machine 140 and the production machine 150, and forward the real traffic data to the production machine 160. After the verification machine 140 completes the processing and the production machine 150 completes the processing, it can be determined whether the configuration strategy of the verification machine 140 is normal by comparing the processing results of the verification machine 140 and the production machine 150.
[0046] By marking the simulated traffic data, the simulated traffic can be forwarded to the first type processing device and the second type processing device according to the marking, so that the first type processing device and the second type processing device can process it, reducing the load of the first type device that processes real data.
[0047] See also Figure 2 , Figure 2 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.
[0048] Step 202: Determine initial traffic data, wherein the initial traffic data includes simulated traffic data and real traffic data.
[0049] Among them, the initial traffic data can be the traffic data of the business and the traffic data of the traffic pool. For example, the initial traffic data includes the traffic data sent by user A to the server, and also includes the traffic data obtained from the traffic pool; the simulated traffic data can be the traffic data obtained from the traffic pool, which is used to simulate the traffic data sent by real users; the real traffic data can be the traffic data sent by user A to the server.
[0050] In actual applications, the traffic data sent by the user to the server, that is, the real traffic data, and the simulated traffic data can be obtained so as to mark the real traffic data or the simulated traffic data.
[0051] For example, user A sends traffic data 1 and traffic data 2 to the server, and obtains traffic data 3 and traffic data 4 from the traffic table, then traffic data 1 and traffic data 2, as well as traffic data 3 and traffic data 4 are determined to be initial traffic data.
[0052] The embodiments of this specification obtain all the traffic data, so that the simulated traffic data or the real traffic data can be identified in the subsequent steps.
[0053] In one achievable manner, determining the initial flow data includes:
[0054] The traffic data sent to the first type of processing device is intercepted, and the traffic data sent to the first type of processing device is used as initial traffic data.
[0055] Among them, the first type of processing device can be understood as a production machine, that is, a user-oriented server, and the configuration and policy of the production machine are the user-oriented configuration and policy; the second type of processing device can be understood as a test machine, and the configuration and policy of the test machine are the configuration and policy to be tested.
[0056] In actual applications, multiple applications are run in one machine. Through RPC interception, the application will be intercepted when it is called by other applications as a server or when it calls other applications as a client.
[0057] For example, by intercepting flow data 1 and flow data 2 sent by user A to the server through RPC, and intercepting flow data 3 and flow data 4 obtained from the flow table through RPC, it is determined that flow data 1 and flow data 2, and flow data 3 and flow data 4 are initial flow data.
[0058] The embodiments of this specification can collect all the traffic data sent to the first type of processing device by intercepting the traffic data sent to the first type of processing device, thereby improving the data collection efficiency.
[0059] Step 204: Label the simulated traffic data according to preset labeling rules, and forward the labeled simulated traffic data to the first type processing device and the second type processing device according to preset routing rules, and forward the real traffic data to the first type processing device.
[0060] Among them, the preset labeling rules can be rules for identifying the target traffic data and labeling it, for example, determining whether the user data traffic is the target traffic based on the port number, and labeling the target traffic; the first type of processing device can be understood as a production machine, that is, a user-oriented server, and the configuration and policy of the production machine are the user-oriented configuration and policy; the second type of processing device can be understood as a test machine, and the configuration and policy of the test machine are the configuration and policy to be tested; the preset routing rules can be rules for forwarding traffic data.
[0061] In actual applications, the data used for traffic simulation needs to be forwarded to the production environment and the pre-effective environment, and the results returned by the two need to be compared to achieve the purpose of verifying the correctness of the changes in advance. Therefore, the simulated traffic data needs to be annotated so that the simulated traffic data can be forwarded. It should be noted that the real traffic can also be annotated.
[0062] For example, the preset identification rule is that user traffic data with port number 5555 will be identified as simulated traffic data, so the simulated traffic data will be identified, and then the network address of the first type of processing device and the network address of the second type of processing device corresponding to the simulated traffic data with port number 5555 can be determined through the routing rule, and the simulated traffic data will be forwarded to the first type of processing device according to the network address of the first type of processing device, and the simulated traffic data will be forwarded to the second type of processing device according to the network address of the second type of processing device.
[0063] The embodiments of this specification mark the simulated traffic data so that the simulated traffic data can be identified, and forward the simulated traffic data to the first type processing device and the second type processing device, thereby reducing the load of the first type processing device that processes the real traffic data and achieving business isolation.
[0064] In one achievable manner, the step of labeling the simulated traffic data according to a preset labeling rule includes:
[0065] Determine target parameter information according to a preset marking rule, and determine simulated flow data from the initial flow data according to the target parameter information;
[0066] The simulated traffic data is labeled.
[0067] The target parameter information may be parameter information of the simulated traffic data, such as the port number and special parameters of the simulated traffic data.
[0068] In practical applications, it can be determined whether the traffic data is simulated traffic data based on the information carried by the traffic data.
[0069] For example, when receiving initial traffic data A1, the port number of initial traffic data A1 is determined to be 1234, the method information of initial traffic data A1 is determined to be method b, and the special parameter carried by initial traffic data A1 is determined to be parameter c. When receiving initial traffic data A2, the port number of initial traffic data A2 is determined to be 5555, the method information of initial traffic data A2 is determined to be method b, and the special parameter carried by initial traffic data A2 is determined to be parameter c. If the target port information is 1234, the target method information is method b, and the target setting information is parameter c, it can be determined that initial traffic data A2 is not simulated traffic data, and initial traffic data A1 is determined to be simulated traffic data.
[0070] The embodiments of the present specification determine the simulated flow data through the parameter information of the flow data, thereby improving the data recognition speed and thus improving the data processing efficiency.
[0071] In an implementable manner, the target parameter information includes target port information, target method information and target setting information;
[0072] Accordingly, determining the simulated flow data from the initial flow data according to the parameter information includes:
[0073] Determining initial port information, initial method information and initial setting information of the initial flow data;
[0074] In a case where the initial port information, the initial method information and the initial setting information of the initial flow data match the target port information, the target method information and the target setting information, the initial flow data is determined to be the simulation flow data.
[0075] Among them, the initial port information can be the port information of the initial traffic data, for example, the port number is 1234; the initial method information can be the method information of the programming language referenced by the initial traffic data, such as, the method information is method b; the initial setting information can be the special parameters carried by the initial traffic data, for example, parameter c; accordingly, the target port information is the port information determined according to the preset labeling rules; the target method information is the method information determined according to the preset labeling rules; the target setting information is the setting information determined according to the preset labeling rules, that is, the special parameters.
[0076] In actual applications, when the specified interface, specified method and specified parameters are met, the business mark is added to the context, that is, coloring is performed.
[0077] For example, when receiving initial traffic data A, the port number of initial traffic data A is determined to be 1234, the method information of initial traffic data A is determined to be method b, and the special parameter carried by initial traffic data A is determined to be parameter c. If the target port information is 1234, the target method information is method b, and the target setting information is parameter c, then it can be determined that initial traffic data A is simulated traffic data.
[0078] For another example, when initial traffic data A is received, the port number of initial traffic data A is determined to be 5555, the method information of initial traffic data A is determined to be method b, and the special parameter carried by initial traffic data A is determined to be parameter c. If the target port information is 1234, the target method information is method b, and the target setting information is parameter c, it can be determined that initial traffic data A is not simulated traffic data.
[0079] The embodiments of this specification determine the simulated traffic data based on the port information, method information and setting information. Multiple parameter information can avoid recognition errors and improve the accuracy of data processing.
[0080] In an achievable manner, forwarding the annotated simulated traffic data to the first type processing device and the second type processing device according to a preset routing rule includes:
[0081] Determine the application name and flow identifier of the labeled simulated flow data, and determine the forwarding address in the preset routing rule according to the application name and the flow identifier;
[0082] The marked simulation traffic data is forwarded to the first type processing device and the second type processing device according to the forwarding address.
[0083] Among them, the application name can be the application name to be accessed by the traffic data, for example, the application name is APP1; the traffic identifier can be an identifier for marking the simulated data traffic; the forwarding address can be an IP address, for example, an IPV4 address or an IPV6 address.
[0084] In actual applications, the machine to which the traffic data is to be sent can be determined based on the corresponding application name and traffic identifier.
[0085] For example, if the application name is APP1 and the traffic identifier is service 1, the traffic data is determined to be simulated data traffic in the preset routing rules and needs to be forwarded to a first type of processing device and a second type of processing device. The network addresses of the first type of processing device and the second type of processing device are determined according to the application name, and the marked simulated traffic data is forwarded to the first type of processing device and the second type of processing device according to the network address.
[0086] The embodiment of this specification determines the type of traffic data through identification, and determines which machine to forward to through application name. Since the traffic data is sent to a non-production machine through identification, the load on the production machine is reduced.
[0087] In an achievable manner, determining the forwarding address in the preset routing rule according to the application name and the traffic identifier includes:
[0088] Determine the target application name and target identifier according to the preset routing rule;
[0089] When the application name and the identifier of the marked simulation traffic data match the target application name and the target identifier, a first forwarding address and a second forwarding address are determined.
[0090] Among them, the target identifier can be the traffic identifier carried by the traffic data that needs to be forwarded, the target application name can be the application name carried by the traffic data that needs to be forwarded, the first forwarding address can be the network address of the first type of processing device, and the second forwarding address can be the network address of the second type of processing device.
[0091] In actual applications, the machine to which the traffic data is forwarded can be determined based on the application name and whether it has an identifier.
[0092] For example, if the application name is APP1 and the traffic identifier is service 1, then the traffic data is determined in the preset routing rules to be simulated data traffic and needs to be forwarded to the first type of processing device and the second type of processing device, then the network address of the first type of processing device is determined according to the application name: the first forwarding address. And the network address of the second type of processing device is determined according to the application name: the second forwarding address, and the marked simulated traffic data is forwarded to the first type of processing device according to the first forwarding address, and the marked simulated traffic data is forwarded to the second type of processing device according to the second forwarding address.
[0093] The embodiments of this specification determine the type of traffic data through identification and determine which machine to forward to through application name, thereby reducing the load on the production machine.
[0094] In an achievable manner, forwarding the marked simulated traffic data to the first type processing device and the second type processing device according to the forwarding address includes:
[0095] The annotated simulated traffic data is forwarded to a first type of processing device according to the first forwarding address, and the annotated simulated traffic data is forwarded to a second type of processing device according to the second forwarding address.
[0096] In practical applications, when the network address of a physical machine is obtained, the traffic data can be forwarded to the corresponding machine.
[0097] For example, the marked simulated traffic data is forwarded to a first type of processing device according to a first forwarding address, and the marked simulated traffic data is forwarded to a second type of processing device according to a second forwarding address.
[0098] The embodiment of this specification forwards the traffic data to the corresponding machine according to the forwarding address, thereby reducing the load of the production machine.
[0099] In an achievable manner, forwarding the real traffic data to the first type of processing device includes:
[0100] In a case where the initial port information, the initial method information and the initial setting information of the initial flow data do not match the target port information, the target method information and the target setting information, determining that the initial flow data is the real flow data;
[0101] The real traffic data is forwarded to the first type of processing device according to a preset forwarding address.
[0102] The preset forwarding address may be a default forwarding address, and correspondingly, the forwarding address may be an IPV4 address or an IPV6 address.
[0103] In actual applications, if no matching information is found in the target port information, target method information and target setting information, the corresponding traffic data is determined to be real traffic data, and the real traffic data can be forwarded to the first type of processing device through the default forwarding address.
[0104] For example, when receiving initial traffic data A, the port number of initial traffic data A is determined to be 5555, the method information of initial traffic data A is determined to be method b, and the special parameter carried by initial traffic data A is determined to be parameter c. If the target port information is 1234, the target method information is method b, and the target setting information is parameter c, it can be determined that initial traffic data A is real traffic data. Determine the forwarding address of the real traffic data, and forward the real traffic data to the corresponding machine according to the forwarding address.
[0105] The embodiment of this specification adds a specific business tag to the verification traffic, while the production customer traffic does not need to be processed. In this way, when the client initiates traffic, the marked traffic is directed to the designated machine to achieve isolation from the production customer traffic. Among them, the verification traffic is the traffic sent to the pre-validation environment and the production environment for verification results, that is, the simulation traffic.
[0106] In an achievable manner, before determining the initial flow data, the method further includes:
[0107] Acquire user traffic data according to the level label, wherein the user traffic data carries the category label and the level label;
[0108] According to the category label, the user flow data is stored in a pre-established flow data list, wherein the flow data list includes a task table for the category label, and the task table manages the acquired user flow data;
[0109] Accordingly, the determining of the initial flow data includes:
[0110] The initial flow data is obtained from the task table.
[0111] Among them, the level label can be a label of the frequency level of the user traffic data. For example, the user requests to view the loan limit, and the frequency of user traffic data A is 100 times per minute, then the level label of user traffic data A is high-frequency traffic; user traffic data can be request data sent by the user to the server through the client, for example, the user requests to view the loan limit; the category label can be the business type to which the user traffic data belongs. For example, the user traffic data is the traffic data of the user requesting to view the loan limit, then the user traffic data is a loan business. The traffic data list can be the traffic pool in the above embodiment, and the implementation method can be a list-based data structure. The target traffic data can be part of the task table data, for example, 10 data are extracted from each task table.
[0112] In actual applications, there are user traffic data with high frequency and there are also user traffic data with low frequency. If the frequency of user traffic data is not considered and random sampling is performed, it may result in only high-frequency traffic being obtained. When processing user traffic data later, it may be impossible to obtain accurate processing results through comprehensive user traffic data. Therefore, the level labels of user traffic data are added to extract user traffic data with different level labels. The core high-frequency traffic, low-frequency traffic, and traffic artificially constructed in new scenarios are all uniformly stored through the traffic pool. Finally, after various products pass through the traffic pool, the output verification traffic remains stable. Obtaining user data traffic from the task table can be used for traffic simulation, that is, calling the production environment and the pre-effective environment in real time, and comparing the results returned by the two, so as to achieve the purpose of verifying the correctness of the change in advance.
[0113] For example, if the preset number is 100 and the preset time interval is ten minutes, user data traffic is collected every ten minutes, and the level labels include high-frequency traffic level labels, low-frequency traffic level labels, and new business level labels. Every ten minutes, 100 pieces of user traffic data are extracted from the high-frequency traffic level labels, low-frequency traffic level labels, and new business level labels. In a certain ten minutes, the preset frequency threshold is: a frequency of more than 100 times an hour is defined as high-frequency traffic data, and a frequency of one hour or less is defined as low-frequency traffic data. Within a period of time, the user traffic data corresponding to the user's balance acquisition occurs more than 100 times per hour, and the user traffic data corresponding to the user's credit acquisition occurs more than 100 times per hour, then the user traffic data corresponding to the user's balance acquisition and the user traffic data corresponding to the user's credit acquisition are high-frequency traffic data. The user traffic data corresponding to the user's balance and the user's credit limit are large in volume, for example, 1,000 pieces of user traffic data corresponding to the user's balance and 1,000 pieces of user traffic data corresponding to the user's credit limit every ten minutes. In this case, 100 pieces of user traffic data can be randomly selected from the user traffic data corresponding to the user's balance and the user traffic data corresponding to the user's credit limit. The user traffic data corresponding to the user's repayment date is less than 100 times per hour, and the user traffic data corresponding to the user's configuration items is less than 100 times per hour. In this case, the user traffic data corresponding to the user's repayment date and the user's configuration items are low-frequency traffic data. The user traffic data corresponding to the user's repayment date and the user's configuration items are small in volume, for example, 50 pieces of user traffic data corresponding to the user's repayment date and 10 pieces of user traffic data corresponding to the user's configuration items every ten minutes. In this case, the user traffic data corresponding to the user's repayment date and the user's configuration items can be obtained only when there are enough 100 pieces of user traffic data. The testing staff constructs 50 pieces of user traffic data corresponding to the new business N, and sends the 50 pieces of user traffic data corresponding to the new business N to the collection module, that is, receives the user traffic data corresponding to the preset number of the third-level tags.
[0114] 53 pieces of user traffic data corresponding to the balance obtained by users, 47 pieces of user traffic data corresponding to the credit limit obtained by users, 71 pieces of user traffic data corresponding to the repayment date obtained by users, 29 pieces of user traffic data corresponding to the configuration items obtained by users, and 50 pieces of user traffic data corresponding to the new business N are obtained. These 250 pieces of data are put into the traffic data list.
[0115] The task table in the traffic data list can store 100 traffic data, including the task table corresponding to the user's obtained credit limit and the task table corresponding to the user's obtained repayment date. Then, 50 user data traffic are obtained from each of the task tables corresponding to the user's obtained credit limit and the task table corresponding to the user's obtained repayment date.
[0116] The embodiments of the present specification obtain user flow data according to the level tags, so that the user flow data corresponding to different level tags can be obtained to improve the comprehensiveness of the user flow data. The user flow data is stored in a pre-established flow data list according to the category tags to ensure that the flow data list contains user flow data corresponding to all category tags, thereby improving the comprehensiveness of the data. By obtaining user data flow from the task table, since the task table includes user flow data of multiple level tags, the data is highly comprehensive, so the accuracy of the data obtained by data processing according to the target flow data is high.
[0117] The embodiments of this specification provide a data processing method and device, wherein the data processing method includes: determining initial traffic data, wherein the initial traffic data includes simulated traffic data and real traffic data; marking the simulated traffic data according to a preset marking rule, and forwarding the marked simulated traffic data to a first type of processing device and a second type of processing device according to a preset routing rule, and forwarding the real traffic data to the first type of processing device. By receiving the initial traffic data and marking the simulated traffic data therein, the simulated traffic can be forwarded to the first type of processing device and the second type of processing device according to the marking, so that the first type of processing device and the second type of processing device can process it, reducing the load of the first type of device that processes the real data.
[0118] See also Figure 3a , Figure 3a A flowchart showing another data processing method is provided. The embodiment of this specification provides a data processing method, which is applied to a client and includes:
[0119] Step 302a: determining initial traffic data for different servers, wherein the initial traffic data includes simulated traffic data and real traffic data;
[0120] Step 304a: Label the simulated traffic data according to preset labeling rules, and forward the labeled simulated traffic data to the first type server and the corresponding second type server according to preset routing rules, and forward the real traffic data to the corresponding first type server.
[0121] In practical applications, coloring of simulated traffic data can be performed on the client, that is, coloring can be performed on an intermediate machine.
[0122] For example, the user sends real traffic data to the client, and the client obtains simulated traffic data from the task table of the traffic pool. The client determines whether the traffic data is simulated traffic data or real traffic data based on the port number and other information of the obtained traffic data, and marks the simulated traffic data to forward the simulated traffic data to the verification machine and production machine A, and forward the real traffic data to production machine B. After the verification machine completes the processing and production machine A completes the processing, the processing results of the verification machine and production machine A can be compared to determine whether the configuration strategy of the verification machine is normal.
[0123] The embodiments of the present specification receive initial traffic data and mark the simulated traffic data therein, so that the simulated traffic can be forwarded to the first type of processing device and the second type of processing device according to the mark, so that the first type of processing device and the second type of processing device can process it, thereby reducing the load of the first type of device that processes the real data.
[0124] See also Figure 3b , Figure 3b A scenario diagram showing another data processing method is shown. The embodiment of this specification further provides a data processing method, which is applied to a first server, including:
[0125] Step 302b: determining initial traffic data of different clients, wherein the initial traffic data includes simulated traffic data and real traffic data;
[0126] Step 304b: Processing the real traffic data and labeling the simulated traffic data according to a preset labeling rule;
[0127] Step 306b: forwarding the marked simulated traffic data to the second server and the third server according to the preset routing rule.
[0128] In practical applications, the coloring of simulated traffic data can be performed on the server side, that is, the traffic data is colored on the production machine.
[0129] For example, see Figure 3c , Figure 3cA scenario diagram showing another data processing method is shown, in which a user sends real traffic data to a first server, and the first server obtains simulated traffic data from a task table of a traffic pool. The first server determines whether the traffic data is simulated traffic data or real traffic data based on the port number and other information of the obtained traffic data, and marks the simulated traffic data to forward the simulated traffic data to a second server and a third server, and process the real traffic data. After the second server completes the processing and the third server completes the processing, it can be determined whether the configuration strategy of the second server is normal by comparing the processing results of the second server and the third server.
[0130] The embodiments of the present specification receive initial traffic data and mark the simulated traffic data therein, so that the simulated traffic can be forwarded to the first type of processing device and the second type of processing device according to the mark, so that the first type of processing device and the second type of processing device can process it, thereby reducing the load of the first type of device that processes the real data.
[0131] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 4a FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 4a As shown, the device comprises:
[0132] The data determination module 402a is configured to determine initial flow data, wherein the initial flow data includes simulated flow data and real flow data;
[0133] The data forwarding module 404a is configured to label the simulated traffic data according to preset labeling rules, and forward the labeled simulated traffic data to the first type processing device and the second type processing device according to preset routing rules, and forward the real traffic data to the first type processing device.
[0134] In an implementable manner, the data determination module 402a is further configured to:
[0135] The traffic data sent to the first type of processing device is intercepted, and the traffic data sent to the first type of processing device is used as initial traffic data.
[0136] In an achievable manner, the data forwarding module 404a is further configured to:
[0137] Determine target parameter information according to a preset marking rule, and determine simulated flow data from the initial flow data according to the target parameter information;
[0138] The simulated traffic data is labeled.
[0139] In an achievable manner, the data forwarding module 404a is further configured to:
[0140] Accordingly, determining the simulated flow data from the initial flow data according to the parameter information includes:
[0141] Determining initial port information, initial method information and initial setting information of the initial flow data;
[0142] In a case where the initial port information, the initial method information and the initial setting information of the initial flow data match the target port information, the target method information and the target setting information, the initial flow data is determined to be the simulation flow data.
[0143] In an achievable manner, the data forwarding module 404a is further configured to:
[0144] Determine the application name and flow identifier of the labeled simulated flow data, and determine the forwarding address in the preset routing rule according to the application name and the flow identifier;
[0145] The marked simulation traffic data is forwarded to the first type processing device and the second type processing device according to the forwarding address.
[0146] In an achievable manner, the data forwarding module 404a is further configured to:
[0147] Determine the target application name and target identifier according to the preset routing rule;
[0148] When the application name and the identifier of the marked simulation traffic data match the target application name and the target identifier, a first forwarding address and a second forwarding address are determined.
[0149] In an achievable manner, the data forwarding module 404a is further configured to:
[0150] The annotated simulated traffic data is forwarded to a first type of processing device according to the first forwarding address, and the annotated simulated traffic data is forwarded to a second type of processing device according to the second forwarding address.
[0151] In an achievable manner, the data forwarding module 404a is further configured to:
[0152] In a case where the initial port information, the initial method information and the initial setting information of the initial flow data do not match the target port information, the target method information and the target setting information, determining that the initial flow data is the real flow data;
[0153] The real traffic data is forwarded to the first type of processing device according to a preset forwarding address.
[0154] In an achievable manner, the data forwarding module 404a is further configured to:
[0155] Acquire user traffic data according to the level label, wherein the user traffic data carries the category label and the level label;
[0156] According to the category label, the user flow data is stored in a pre-established flow data list, wherein the flow data list includes a task table for the category label, and the task table manages the acquired user flow data;
[0157] Accordingly, the determining of the initial flow data includes:
[0158] The initial flow data is obtained from the task table.
[0159] The embodiments of this specification provide a data processing method and device, wherein the data processing device includes: a data determination module, configured to determine initial traffic data, wherein the initial traffic data includes simulated traffic data and real traffic data; a data forwarding module, configured to label the simulated traffic data according to a preset labeling rule, and forward the labeled simulated traffic data to a first type of processing device and a second type of processing device according to a preset routing rule, and forward the real traffic data to the first type of processing device. By receiving the initial traffic data and labeling the simulated traffic data therein, the simulated traffic can be forwarded to the first type of processing device and the second type of processing device according to the labeling, so that the first type of processing device and the second type of processing device can process it, reducing the load of the first type of device that processes the real data.
[0160] This specification also provides a data processing device embodiment. Figure 4b FIG. 2 shows a schematic diagram of the structure of another data processing device provided by an embodiment of the present specification. Figure 4b As shown, the device comprises:
[0161] The data determination module 402b is configured to determine initial flow data for different service terminals, wherein the initial flow data includes simulated flow data and real flow data;
[0162] The data forwarding module 404b is configured to label the simulated traffic data according to preset labeling rules, and forward the labeled simulated traffic data to the first type server and the corresponding second type server according to preset routing rules, and forward the real traffic data to the corresponding first type server.
[0163] This specification also provides a data processing device embodiment. Figure 4c FIG. 2 shows a schematic diagram of the structure of another data processing device provided by an embodiment of the present specification. Figure 4c As shown, the device comprises:
[0164] The data determination module 402c is configured to determine initial traffic data of different clients, wherein the initial traffic data includes simulated traffic data and real traffic data;
[0165] The data marking module 404c is configured to process the real traffic data and mark the simulated traffic data according to a preset marking rule;
[0166] The data forwarding module 406c is configured to forward the marked simulation traffic data to the second server and the third server according to a preset routing rule.
[0167] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.
[0168] Figure 5 The structure block diagram of a computing device 500 provided according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.
[0169] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0170] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0171] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 500 may also be a mobile or stationary server.
[0172] The processor 520 is used to execute the following computer executable instructions, which implement the steps of the above data processing method when executed by the processor.
[0173] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.
[0174] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned data processing method when executed by a processor.
[0175] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.
[0176] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.
[0177] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above data processing method.
[0178] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0179] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0180] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.
[0181] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0182] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.
Claims
1. A data processing method, comprising: Determine initial traffic data, wherein the initial traffic data includes simulated traffic data and real traffic data, the simulated traffic data is traffic data obtained from a traffic pool and is used to simulate traffic data sent by a real user; The simulated traffic data is labeled according to preset labeling rules, and according to preset routing rules, the labeled simulated traffic data is forwarded to the first type of processing device and the second type of processing device, and the real traffic data is forwarded to the first type of processing device.
2. The method according to claim 1, wherein determining the initial flow data comprises: The traffic data sent to the first type of processing device is intercepted, and the traffic data sent to the first type of processing device is used as initial traffic data.
3. According to the method of claim 1, the step of labeling the simulated traffic data according to a preset labeling rule comprises: Determine target parameter information according to a preset marking rule, and determine simulated flow data from the initial flow data according to the target parameter information; The simulated traffic data is labeled.
4. The method according to claim 3, wherein the target parameter information includes target port information, target method information and target setting information; Accordingly, determining the simulated flow data from the initial flow data according to the parameter information includes: Determining initial port information, initial method information and initial setting information of the initial flow data; In a case where the initial port information, the initial method information and the initial setting information of the initial flow data match the target port information, the target method information and the target setting information, the initial flow data is determined to be the simulated flow data.
5. The method according to claim 1, wherein forwarding the annotated simulated traffic data to the first type processing device and the second type processing device according to a preset routing rule comprises: Determine the application name and flow identifier of the labeled simulated flow data, and determine the forwarding address in the preset routing rule according to the application name and the flow identifier; The marked simulation traffic data is forwarded to the first type processing device and the second type processing device according to the forwarding address.
6. The method according to claim 5, wherein determining the forwarding address in the preset routing rule according to the application name and the traffic identifier comprises: Determine the target application name and target identifier according to the preset routing rule; When the application name and the identifier of the marked simulation traffic data match the target application name and the target identifier, a first forwarding address and a second forwarding address are determined.
7. The method according to claim 6, wherein forwarding the annotated simulation traffic data to the first type processing device and the second type processing device according to the forwarding address comprises: The annotated simulated traffic data is forwarded to a first type of processing device according to the first forwarding address, and the annotated simulated traffic data is forwarded to a second type of processing device according to the second forwarding address.
8. The method according to claim 4, wherein forwarding the real traffic data to the first type of processing device comprises: In a case where the initial port information, the initial method information and the initial setting information of the initial flow data do not match the target port information, the target method information and the target setting information, determining that the initial flow data is the real flow data; The real traffic data is forwarded to the first type of processing device according to a preset forwarding address.
9. The method according to claim 1, before determining the initial flow data, further comprising: Acquire user traffic data according to the level label, wherein the user traffic data carries the category label and the level label; According to the category label, the user flow data is stored in a pre-established flow data list, wherein the flow data list includes a task table for the category label, and the task table manages the acquired user flow data; Accordingly, the determining of the initial flow data includes: The initial flow data is obtained from the task table.
10. A data processing method, applied to a client, comprising: Determine initial traffic data for different servers, wherein the initial traffic data includes simulated traffic data and real traffic data, and the simulated traffic data is traffic data obtained from a traffic pool and is used to simulate traffic data sent by real users; The simulated traffic data is labeled according to preset labeling rules, and according to preset routing rules, the labeled simulated traffic data is forwarded to the first type server and the corresponding second type server, and the real traffic data is forwarded to the corresponding first type server.
11. A data processing method, applied to a first server, comprising: Determine initial traffic data of different clients, wherein the initial traffic data includes simulated traffic data and real traffic data, and the simulated traffic data is traffic data obtained from a traffic pool and is used to simulate traffic data sent by real users; Processing the real traffic data and labeling the simulated traffic data according to a preset labeling rule; According to the preset routing rules, the marked simulation traffic data is forwarded to the second server and the third server.
12. A data processing device, comprising: The data determination module is configured to determine initial traffic data, wherein the initial traffic data includes simulated traffic data and real traffic data, and the simulated traffic data is traffic data obtained from a traffic pool and is used to simulate traffic data sent by a real user; The data forwarding module is configured to label the simulated traffic data according to preset labeling rules, and forward the labeled simulated traffic data to the first type processing device and the second type processing device according to preset routing rules, and forward the real traffic data to the first type processing device.
13. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data processing method described in any one of claims 1 to 9, claim 10 or claim 11 are implemented.
14. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method described in any one of claims 1 to 9, claim 10 or claim 11.
Citation Information
Patent Citations
Traffic dyeing method and system
CN113965510A
Distributed full-link gray release method and device
CN114615192A
