Data source call processing method, device, computer equipment and storage medium

Abstract

The present invention relates to the field of data processing technology, and discloses a data source call processing method, device, computer equipment and storage medium. The data source call processing method receives a data call request, obtains the data source configuration information of all initial data sources according to the data call request; obtains the historical call information of all initial data sources, and determines at least one candidate data source from all initial data sources according to the historical call information and the data source configuration information; determines the flow configuration ratio corresponding to each candidate data source according to the preset ratio configuration rule and the data source configuration information; generates a target random number according to a preset random rule, and determines the target data source corresponding to the data call request from all candidate data sources according to the target random number and the flow configuration ratio. The present invention realizes automatic monitoring of the data source, ensures the availability and reliability of the data source, reduces the risk of service interruption and data loss, and improves the rationality of the call.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data source call processing method, device, computer equipment and storage medium. Background Art

[0002] In today's digital age, with the rapid growth of data and the increase in business complexity, enterprises are increasingly relying on data sources from third-party suppliers to support their business operations and data management. The call of data sources involves important links such as data processing, storage and transmission. In the existing technology, enterprises use a fixed traffic ratio to call when accessing multiple data sources. When one of the data sources is abnormal, it will cause data loss and service interruption. At the same time, when the data source is unstable, it is necessary to manually modify the ratio configuration between the data sources, and it is impossible to call a reasonable data source in time, which will cause data quality problems.

[0003] Therefore, a new data source call processing solution is urgently needed to ensure data integrity and enterprise business continuity. Summary of the invention

[0004] Based on this, it is necessary to provide a data source call processing method, device, computer equipment and storage medium to address the above technical problems, so as to solve the problems of data loss and unreasonable calls that are prone to occur during the data source call process.

[0005] A data source call processing method, comprising: Receive a data call request, and obtain data source configuration information of all initial data sources according to the data call request; Acquire historical call information of all the initial data sources, and determine at least one candidate data source from all the initial data sources according to the historical call information and the data source configuration information; Determine the flow configuration ratio of each of the candidate data sources according to the preset ratio configuration rule and the data source configuration information, wherein the flow configuration ratio is used to characterize the distribution ratio of data flow direction and task load between each of the candidate data sources; A target random number is generated according to a preset random rule, and a target data source corresponding to the data call request is determined from all the candidate data sources according to the target random number and the traffic configuration ratio.

[0006] A data source call processing device, comprising: A data source configuration acquisition module, used to receive a data call request, and acquire data source configuration information of all initial data sources according to the data call request; A candidate data source determination module, used to obtain historical call information of all the initial data sources, and determine at least one candidate data source from all the initial data sources according to the historical call information and the data source configuration information; A flow ratio determination module, used to determine the flow configuration ratio of each of the candidate data sources according to a preset ratio configuration rule and the data source configuration information, wherein the flow configuration ratio is used to characterize the distribution ratio of data flow and task load between each of the candidate data sources; The target data source determination module is used to generate a target random number according to a preset random rule, and determine the target data source corresponding to the data call request from all the candidate data sources according to the target random number and the traffic configuration ratio.

[0007] A computer device comprises a memory, a processor and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data source call processing method when executing the computer-readable instructions.

[0008] A computer-readable storage medium stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the data source call processing method as described above.

[0009] In the above-mentioned data source call processing method, device, computer equipment and storage medium, the data source call processing method receives a data call request, obtains data source configuration information of all initial data sources according to the data call request; obtains historical call information of all initial data sources, and determines at least one candidate data source from all initial data sources according to the historical call information and data source configuration information; determines the flow configuration ratio of each candidate data source according to the preset ratio configuration rule and data source configuration information, and the flow configuration ratio is used to characterize the distribution ratio of data flow and task load between each candidate data source; generates a target random number according to a preset random rule, and determines the target data source corresponding to the data call request from all candidate data sources according to the target random number and the flow configuration ratio. The present invention determines at least one candidate data source for all initial data sources according to the historical call information and data source configuration information, realizes automatic monitoring and diagnosis of data sources, can timely identify abnormal data sources, helps to ensure the availability and reliability of data sources, and reduces the risk of service interruption and data loss. At the same time, the present invention determines the target data source corresponding to the data call request from all candidate data sources according to the target random number and the flow configuration ratio, can quickly and accurately determine the target data source that meets the data call requirements, and improves the rationality of the call. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0011] Figure 1 It is a flowchart of a data source calling processing method in one embodiment of the present invention; Figure 2 is a flow chart of step S10 in the data source calling processing method in one embodiment of the present invention; Figure 3 is a flow chart of step S20 in the data source calling processing method in one embodiment of the present invention; Figure 4 It is another flowchart diagram of step S20 in the data source calling processing method in one embodiment of the present invention; Figure 5 is a flow chart of step S30 in the data source calling processing method in one embodiment of the present invention; Figure 6 is a flow chart of step S40 in the data source calling processing method in one embodiment of the present invention; Figure 7 It is another flowchart diagram of step S40 in the data source calling processing method in one embodiment of the present invention; Figure 8 It is a structural diagram of a data source calling processing device in one embodiment of the present invention; Fig. 9 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] The data source call processing method provided in this embodiment can be applied to the application scenario where the server calls the data source to respond to the client, and the client establishes a communication connection with the server. The client includes but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. In actual applications, the client can be a business party, the server can be a server of a third-party supplier, and the server includes a data source routing of the logic layer and data sources of multiple third-party suppliers (such as API interfaces). Data source routing refers to the logic of deciding to call one of the data sources to obtain data according to actual business needs and data characteristics. In this embodiment, the server receives a data call request from the client, obtains data source configuration information of all initial data sources according to the data call request; obtains historical call information of all initial data sources, and determines at least one candidate data source from all initial data sources according to the historical call information and data source configuration information; determines the traffic configuration ratio of each candidate data source according to the preset ratio configuration rule and data source configuration information; generates a target random number according to a preset random rule, and determines the target data source corresponding to the data call request from all candidate data sources according to the target random number and the traffic configuration ratio. Finally, the server responds to the client's data call request by calling the target data source.

[0014] In one embodiment, if Figure 1 As shown, a data source call processing method is provided, including the following steps S10-S40. S10: Receive a data call request, and obtain data source configuration information of all initial data sources according to the data call request.

[0015] Understandably, when the client's business needs to call data, a data call request will be sent to the server. The data call request is a request information used to characterize the client's data call requirements. The server receives the data call request and obtains the data source configuration information of all initial data sources according to the data call request. The initial data source refers to a pre-configured data source used to meet the client's data call requirements. Considering the parallel processing capability and data source fault tolerance of the server, multiple initial data sources are set, and any data source can meet the client's data call request alone, that is, different initial data sources are backup data sources for each other. For example, when a data source fails, the server can automatically transfer the data call request to other available initial data sources to ensure the continuity and availability of the data call. Different data sources correspond to different data source configuration information, and the data source configuration information refers to the detailed parameters and call conditions set for each data source. For example, the data source configuration information includes the data source name corresponding to the data source, the preset call success threshold and the initial traffic ratio. The data source name is a unique identification name used to distinguish different data sources. In a specific embodiment, the server includes three initial data sources, and the data source names of the three initial data sources are data source A, data source B and data source C, respectively.

[0016] In one embodiment, if Figure 2 As shown, in step S10, that is, obtaining the data source configuration information of all initial data sources according to the data call request, includes: S101, querying the data source configuration information corresponding to each of the initial data sources in the preset data source configuration list according to the data call request, and determining whether the query of each of the initial data sources is successful; S102: If all the initial data sources are queried successfully, then the data source configuration information of all the initial data sources is obtained.

[0017] Understandably, the preset data source configuration list is a data table pre-set for recording data sources and data source configuration information. The data in the preset data source configuration list is structured data that stores data sources and data source configuration information according to the association relationship. One data source corresponds to one data source configuration information. For example, the data source configuration information corresponding to data source A includes data source A (indicating the data source name), 80% (indicating the preset call success threshold) and 20% (indicating the initial traffic ratio), etc. The server includes data sources from multiple third-party suppliers. After receiving the data call request, it determines whether each data source is in the open state, and determines the data source in the open state as the initial data source. The state of the data source includes the open state and the closed state. Only the data source in the open state can be called. After determining that all initial data sources are in the open state, the data source configuration information corresponding to each initial data source is queried in the preset data source configuration list, and it is determined whether each initial data source is successfully queried. The data source configuration information may be empty or missing in the preset data source configuration list. If all the data source configuration information corresponding to the initial data source can be queried in the preset data source configuration list, it indicates that the initial data source query is successful. If all data source configuration information corresponding to the initial data source is not found in the preset data source configuration list (eg, the preset call success threshold and the initial traffic ratio are missing), it indicates that the initial data source query has failed.

[0018] Each initial data source may be queried successfully or unsuccessfully. When all initial data sources are queried successfully, the server obtains the data source configuration information of all initial data sources. When at least one initial data source fails to query, a call alarm message is generated. The call alarm message refers to the prompt information used to characterize that the data source call is abnormal and the data source routing is terminated. After the server generates the call alarm message, the process of the data source call processing method is terminated and the subsequent steps are no longer performed. This is because when the data source configuration information of the initial data source is incomplete, it will lead to the inability to accurately determine the target data source corresponding to the data call request. For example, when the preset call success threshold is missing in the data source configuration information, the candidate data source cannot be accurately determined from the initial data source. For another example, when the initial traffic ratio is missing in the data source configuration information, the traffic configuration ratio of each candidate data source cannot be accurately determined. After the server generates the call alarm message, the call alarm message can be sent to the preset alarm receiver (such as the terminal device of the operator), so that the preset alarm receiver performs exception processing according to the received call alarm message.

[0019] This embodiment obtains the data source configuration information of all initial data sources only on the premise that all initial data sources are queried successfully, thereby ensuring the integrity of the data source configuration information and avoiding unreasonable data source calls due to missing data source configuration information.

[0020] S20: Obtain historical call information of all the initial data sources, and determine at least one candidate data source from all the initial data sources according to the historical call information and the data source configuration information.

[0021] Understandably, after obtaining the data source configuration information of each initial data source, the server further collects the historical call information of each initial data source. The historical call information refers to the call details of the data source in the past period of time, and the historical call information includes the response time, success rate, error rate and load of the initial data source during the historical call process. The data source configuration information includes a preset call success threshold corresponding to each initial data source, and the preset call success threshold is a preset minimum success rate critical value for determining whether the data source is allowed to be called. For each initial data source, the historical call information and the data source configuration information are compared to screen out the initial data source whose historical call success rate reaches the preset call success threshold, and the screened initial data source is determined as the candidate data source. The candidate data source is a candidate data source selected from the initial data source for responding to the data call request. After the screening is completed, one, two or more candidate data sources need to be determined from all the initial data sources to ensure that at least one data source can respond to the data call request.

[0022] In one embodiment, the data source configuration information includes a preset call success threshold corresponding to each of the initial data sources; Figure 3 As shown, in step S20, that is, determining at least one candidate data source from all the initial data sources according to the historical call information and the data source configuration information, includes: S201, obtaining historical call information within a preset historical time period, and determining the historical call success rate corresponding to each of the initial data sources according to the historical call information; S202, selecting an initial data source from all the initial data sources as a data source to be compared, and determining whether a historical call success rate corresponding to the data source to be compared reaches a preset call success threshold; S203: If the historical call success rate corresponding to the data source to be compared has not reached the preset call success threshold, the data source to be compared is determined as a data source to be retried, and the data source to be retried is changed from an open state to a closed state, and the process returns to the step of selecting an initial data source from all the initial data sources as the data source to be compared, until all the initial data sources are traversed; S204: If the historical call success rate corresponding to the data source to be compared reaches a preset call success threshold, the data source to be compared is determined as a selected data source, and the selected data source is kept in an open state, and the step of selecting an initial data source from all the initial data sources as the data source to be compared is returned until all the initial data sources are traversed.

[0023] Understandably, the server can monitor the communication status and data quality of each data source in real time, dynamically report the request response (including success, failure and timeout, etc.) during the data call request process, and make data summary statistics, and record the statistical data as historical call information. When obtaining the historical call information of all initial data sources, obtain the historical call information within the preset historical time period, and determine the historical call success rate corresponding to each initial data source based on the historical call information. The preset historical time period is a pre-set time span for limiting the time span when obtaining historical call information. The preset historical time period can be set to a default value, or it can be adjusted according to actual needs, for example, the default value of the preset historical time period is set to 5 minutes. The historical call success rate refers to the success rate of each initial data source being successfully called during the data call process within the preset historical time period, that is, the ratio of the number of successful calls to the total number of calls. For example, if data source A is called 10 times within 5 minutes and the number of successful calls is 7 times, then the historical call success rate of data source A is 70%.

[0024] After determining the historical call success rate corresponding to each initial data source, it is necessary to select an initial data source from all initial data sources as the data source to be compared, and compare the historical call success rate of the data source to be compared with the preset call success threshold. After completing one comparison, return to the step of selecting an initial data source that has not been selected from all initial data sources as the data source to be compared, until all initial data sources are compared. The data source to be compared is the data source selected from the initial data source as the current comparison object. When the historical call success rate is less than the preset call success threshold, the initial data source is likely to fail to respond, so it is not allowed to be used as a candidate data source, but the initial data source is determined as a data source to be retried, and the data source to be retried is changed from the open state to the closed state. The data source to be retried refers to a data source that is excluded in the process of determining the candidate data source from all initial data sources. The closed data source to be retried cannot be used as the initial data source for responding to the next data call request. When the historical call success rate is greater than or equal to the preset call success threshold, the initial data source is not likely to fail to respond, so it is allowed to be used as a candidate data source, and the candidate data source is kept in the open state.

[0025] In a specific embodiment, the server includes three initial data sources, namely, data source A, data source B, and data source C. Data source A is selected as the data source to be compared, and the historical call success rate of data source A is 70%, and the preset call success threshold of data source A is 80%. Since the historical call success rate of data source A "70%" has not reached the preset call success threshold "80%", data source A is determined as the data source to be retried, and the data source to be retried is changed from the open state to the closed state. Then, data source B is selected as the data source to be compared to complete the same comparison process until all comparisons of data sources A, data source B, and data source C are completed.

[0026] This embodiment obtains the historical call success rate by analyzing and processing the historical call information, and uses the preset call success threshold to evaluate the availability and stability of the initial data source, thereby achieving effective screening and classification of the initial data source and improving the reliability of data calls. The initial data source is divided into a data source to be retried and a data source to be selected. By changing the data source to be retried from an open state to a closed state, it is possible to avoid using an unstable data source in subsequent data calls and reduce the risk of call failures. At the same time, keeping the selected data source open can be used in subsequent data call processes to meet more data call requests.

[0027] In one embodiment, if Figure 4 As shown, in step S202, that is, after determining whether the historical call success rate corresponding to the data source to be compared reaches a preset call success threshold, it also includes: S2021. If all historical call success rates corresponding to the data sources to be compared have not reached a preset call success threshold, all the initial data sources are determined as data sources to be retried, and call alarm information is generated.

[0028] Understandably, the data sources to be compared are selected one by one from the initial data sources, and the historical call success rates of the data sources to be compared are compared with the preset call success threshold one by one. If all the historical call success rates corresponding to the data sources to be compared have not reached the preset call success threshold, it means that all the initial data sources are excluded, that is, it is impossible to determine at least one data source to be selected from all the initial data sources. At this time, all the initial data sources are determined as data sources to be retried, all the data sources to be retried are changed from the open state to the closed state, and a call alarm message is generated. After the server generates the call alarm message, the process of the data source call processing method is terminated, and the subsequent steps are no longer performed. This is because when all the initial data sources are data sources to be retried and are closed, the data call request cannot be responded to. After the server generates the call alarm message, the call alarm message can be sent to the preset alarm recipient (such as sending an alarm email to the terminal device of the operator).

[0029] When sending a call alarm message, the server will include the cause of the abnormal problem. That is, the call alarm message generated when all initial data sources are determined as data sources to be retried is different from the call alarm message generated when the query of the data source configuration information of the initial data source fails. For example, different abnormal code values ​​can be used to represent different causes of abnormal problems. When the preset alarm recipient receives the call alarm message, it directly locates the cause of the abnormal problem based on the abnormal code value. In addition, when sending the call alarm message, the server can also attach a monitoring icon link for the corresponding time period, which is convenient for operators to view information such as success rate and data value fluctuations, and analyze the cause of the abnormal problem through log analysis, system diagnosis, etc.

[0030] When determining that all initial data sources are data sources to be retried, this embodiment generates call alarm information in a timely manner and triggers an exception handling process, which helps to quickly locate the cause of the problem and take measures to solve it.

[0031] S30. Determine the flow configuration ratio of each of the candidate data sources according to a preset ratio configuration rule and the data source configuration information. The flow configuration ratio is used to characterize the distribution ratio of data flow and task load between each of the candidate data sources.

[0032] Understandably, the data source configuration information includes the initial traffic ratio corresponding to each initial data source, and the initial traffic ratio is the ratio used to balance the task load and distribute the data flow between multiple initial data sources. In a specific embodiment, the server includes three initial data sources, namely, data source A, data source B and data source C. Data source A, data source B and data source C each correspond to an initial traffic ratio, and the sum of the initial traffic ratios of the three is 100%. Among them, the initial traffic ratio of data source A is 20% (indicating that the server has a 20% probability of calling data source A to respond to the data call request), the initial traffic ratio of data source B is 30% (indicating that the server has a 30% probability of calling data source B to respond to the data call request), and the initial traffic ratio of data source C is 50% (indicating that the server has a 50% probability of calling data source C to respond to the data call request).

[0033] During the data source configuration phase, the initial traffic ratio of each initial data source can be specified to control the data flow direction and load distribution. Since each initial data source corresponds to its own initial traffic ratio, and the candidate data source is selected from the initial data source, the traffic ratio of the candidate data source may need to be redistributed. The preset ratio configuration rule is a pre-set rule for allocating the traffic ratio of each candidate data source. The traffic configuration ratio is used to characterize the distribution ratio of data flow direction and task load between each candidate data source. For the preset ratio configuration rules in different situations, the traffic configuration ratio of the candidate data source may be consistent with the initial traffic ratio of the initial data source, or it may be the result of redistributing the initial traffic ratio of the initial data source.

[0034] In one embodiment, the data source configuration information includes initial traffic ratios corresponding to each of the initial data sources; Figure 5 As shown, in step S30, that is, determining the flow configuration ratio of each of the candidate data sources according to the preset ratio configuration rule and the data source configuration information, includes: S301, determining whether there is a data source to be retried in the initial data source; S302: When there is no data source to be retried, determine the initial traffic ratio corresponding to each of the initial data sources as the traffic configuration ratio of the candidate data source corresponding to the same initial data source, and obtain the traffic configuration ratio of each of the candidate data sources; S303. When there is a data source to be retried, the initial traffic ratio corresponding to the data source to be retried is determined as the ratio to be allocated, and the initial traffic ratio corresponding to the selected data source is weightedly calculated according to the ratio to be allocated to obtain the traffic configuration ratio of each of the selected data sources.

[0035] Understandably, after determining at least one data source to be selected from all initial data sources, it is necessary to determine whether there is a data source to be retried in the initial data sources. When there is no data source to be retried in the initial data source, it indicates that all initial data sources are data sources to be selected. At this time, the initial traffic ratio corresponding to each initial data source is directly determined as the traffic configuration ratio of the data source to be selected corresponding to the same initial data source, and the traffic configuration ratio of each data source to be selected is obtained. In a specific embodiment, the server includes three initial data sources, namely, data source A, data source B, and data source C, and the corresponding initial traffic ratios are 20%, 30%, and 50%, respectively. After determining the data source to be selected from all initial data sources according to the historical call information and the data source configuration information, the server includes three data sources to be selected, namely, data source A, data source B, and data source C, and the corresponding traffic configuration ratios are 20%, 30%, and 50%, respectively.

[0036] When there is a data source to be retried in the initial data source, it indicates that part of the initial data source is a data source to be selected and part is a data source to be retried, that is, part of the initial data source is excluded. At this time, it is necessary to determine the initial traffic ratio corresponding to the data source to be retried as the ratio to be allocated, and perform weighted calculation processing on the initial traffic ratio corresponding to the data source to be selected according to the ratio to be allocated to obtain the traffic configuration ratio of each data source to be selected, wherein the weighted calculation processing is the process of redistributing the initial traffic ratio corresponding to each data source to be retried. The ratio to be allocated refers to the traffic ratio that needs to be redistributed. In another specific embodiment, the service end includes three initial data sources, namely, data source A, data source B and data source C, and the initial traffic ratios corresponding to each are 20%, 30% and 50% respectively. After determining the data source to be retried from all initial data sources according to the historical call information and data source configuration information, data source A is the data source to be retried, and data source B and data source C are the data sources to be selected. At this time, data source A is excluded, that is, the traffic ratio is 0. At this time, the initial traffic ratio of data source A, 20%, is determined as the ratio to be allocated, and the initial traffic ratios corresponding to data sources B and C are weighted and calculated according to the ratio to be allocated to obtain the traffic configuration ratios of data sources B and C. The traffic configuration ratio of data source B is the result of the weighted calculation of the initial traffic ratio 30% plus the initial traffic ratio 20% of the data source to be retried, and the calculation process is b1=b+a×[b / (b+c)]=30%+20%×(3 / 8)=37.5%. Among them, b1 represents the traffic configuration ratio of data source B, a represents the initial traffic ratio of data source A, b represents the initial traffic ratio of data source B, and c represents the initial traffic ratio of data source C. The traffic configuration ratio of data source C is the result of the weighted calculation of the initial traffic ratio 50% plus the initial traffic ratio 20% of the data source to be retried, and the calculation process is c1=c+a×[c / (b+c)]=50%+20%×(5 / 8)=62.5%. Among them, c1 represents the traffic configuration ratio of data source C.

[0037] Furthermore, since the data source to be selected is the data source that is turned on in the server at this time, the data source to be selected can be used as the initial data source for the next data source call processing flow. After obtaining the flow configuration ratio of each data source to be selected, the flow configuration ratio is stored in the preset data source configuration list to update the data source configuration information corresponding to each data source to be selected as the initial flow ratio for the next data source call processing flow.

[0038] This embodiment adopts different methods to determine the traffic configuration ratio of each candidate data source for two different situations of whether there is a data source to be retried in the initial data source, so as to facilitate dynamic adjustment of the traffic configuration ratio, which reflects the flexibility of the preset ratio configuration rule and improves the accuracy of the traffic configuration ratio.

[0039] S40: Generate a target random number according to a preset random rule, and determine a target data source corresponding to the data call request from all the candidate data sources according to the target random number and the traffic configuration ratio.

[0040] Understandably, after determining the traffic configuration ratio of each candidate data source, the server also needs to generate a target random number according to a preset random rule, and determine the target data source corresponding to the data call request from all candidate data sources based on the target random number and the traffic configuration ratio. The preset random rule is a pre-set rule for generating random numbers. The target random number is a random number generated after receiving the current data call request. The target data source is a data source for responding to the data call request sent by the client. The server calls the target data source to obtain data to respond to the data call request sent by the client. One data call request corresponds to one target random number and one target data source.

[0041] The preset random rule can be an online random number generator, which can set the range of the random number and whether it is unique. The uniquely generated random number will not be repeated, and the unique may generate repeated numbers. In a specific embodiment, it is necessary to set the minimum random number (such as 1) and the maximum random number (such as 100), and each time generate an integer from 1 to 100 as the target random number.

[0042] In one embodiment, if Figure 6 As shown, in step S40, that is, determining the target data source corresponding to the data call request from all the candidate data sources according to the target random number and the traffic configuration ratio, includes: S401, determining a flow ratio interval corresponding to each of the to-be-selected data sources according to the flow configuration ratio; S402: Determine a target traffic ratio interval that matches the target random number from all the traffic ratio intervals, and determine a candidate data source corresponding to the target traffic ratio interval as the target data source.

[0043] Understandably, since the sum of the traffic configuration ratios of each candidate data source is 100%, each candidate data source corresponds to exactly one traffic ratio interval. The traffic ratio interval refers to the ratio interval obtained by mapping each traffic configuration ratio in the range of 0 to 100% in order from small to large. The target traffic ratio interval refers to the traffic ratio interval in which the percentage corresponding to the target random number is located. When the target random number is an integer from 1 to 100, the target traffic ratio interval that matches the target random number can be determined from all traffic ratio intervals.

[0044] In a specific embodiment, the server includes three data sources to be selected, namely, data source A, data source B and data source C, and the corresponding traffic configuration ratios are 20%, 30% and 50% respectively. Then the traffic ratio interval corresponding to data source A is (0, 20%], the traffic ratio interval corresponding to data source B is (20%, 50%], and the traffic ratio interval corresponding to data source C is (50%, 100%]. When the target random number is 25, the percentage corresponding to the target random number is 25%. At this time, the target traffic ratio interval is the traffic ratio interval corresponding to data source B, and data source B is the target data source.

[0045] This embodiment determines the traffic ratio interval corresponding to each candidate data source according to the traffic configuration ratio, and determines the target data source based on matching the traffic ratio interval with the target random number. It can dynamically allocate the traffic of each data source according to the traffic configuration ratio, thereby balancing the load of each data source and improving the overall data source call processing performance and reliability.

[0046] In one embodiment, if Figure 7 As shown, in step S40, that is, after determining the target data source corresponding to the data call request from all the candidate data sources according to the target random number and the traffic configuration ratio, it includes: S403, receiving a new data call request, and determining whether there is a data source to be retried in the initial data source; S404: If there is a data source to be retried in the initial data source, obtain a retry time interval of the data source to be retried; S405: When it is confirmed that the retry time interval meets the preset retry time interval threshold, the data source to be retried is called through the new data call request, and it is determined whether the data source to be retried is called successfully; S406: If the data source to be retried is successfully called, the data source to be retried is changed from a closed state to an open state.

[0047] Understandably, during the operation, the client will continuously generate the need for data calls, and the server will continuously receive new data call requests. The new data call request is a new data call request generated after the data source call processing flow of the current data call request is completed. After receiving the new data call request, the server needs to first determine whether there is a data source to be retried in the initial data source. If there is no data source to be retried in the initial data source, the data source configuration information of all initial data sources is directly obtained according to the data call request, and the subsequent data source call processing flow is performed. If there is a data source to be retried in the initial data source, it is necessary to obtain the retry time interval of the data source to be retried, and determine whether the retry time interval meets the preset retry interval threshold. The retry time interval refers to the interval time from the last time the data source to be retried was called to the current time point. The preset retry interval threshold is a preset minimum critical value for determining whether the data source to be retried needs to be re-called. The preset retry interval threshold can be set to a default value, or it can be adjusted according to actual needs, for example, the default value of the preset retry interval threshold is set to 1 hour. When the data source to be retried is in a closed state, the operation personnel can analyze and repair the reason for the failure of calling the data source to be retried, so that the data source to be retried can succeed when it is called again.

[0048] When the retry time interval of the data source to be retried is greater than or equal to the preset retry time threshold, it indicates that the data source to be retried can be re-called. At this time, there is no need to determine the target data source corresponding to the newly added data call request. The data source to be retried is directly called through the newly added data call request, and it is determined whether the data source to be retried is successfully called, that is, it is determined whether the server can call the data source to be retried to obtain data. When the data source to be retried fails to be called, the data source to be retried remains in a closed state, and the data source configuration information of all the data sources except the data source to be retried in the initial data sources is obtained, and the subsequent data source call processing flow is performed. When the data source to be retried is called successfully, the data source call processing of the newly added data call request is completed, and the data source to be retried is changed from a closed state to an open state. Further, before receiving the next newly added data call request, the data source configuration information of the reopened data source to be retried needs to be updated (for example, the traffic ratio of each initial data source is re-determined) and stored in the preset data source configuration list. In addition, for the reopened data source to be retried, the historical call success rate can be considered to be 100%, that is, the reopened data source to be retried will become the selected data source in the next data source call processing flow.

[0049] When there is a data source to be retried in the initial data source, this embodiment ensures a reasonable retry strategy by judging the threshold of the retry time interval to avoid resource waste caused by infinite retry. At the same time, when the retry time interval meets the threshold condition, the newly added data call request is used in time to retry the call, which can detect and recover the availability problem of the data source, thereby improving the stability and reliability of the data source call.

[0050] The data source call processing method of this embodiment receives a data call request, obtains the data source configuration information of all initial data sources according to the data call request; obtains the historical call information of all initial data sources, and determines at least one candidate data source from all initial data sources according to the historical call information and the data source configuration information; determines the traffic configuration ratio of each candidate data source according to the preset ratio configuration rule and the data source configuration information; generates a target random number according to the preset random rule, and determines the target data source corresponding to the data call request from all candidate data sources according to the target random number and the traffic configuration ratio. This embodiment determines at least one candidate data source for all initial data sources according to the historical call information and the data source configuration information, realizes automatic monitoring and diagnosis of the data source, can timely identify the abnormal data source, helps to ensure the availability and reliability of the data source, and reduces the risk of service interruption and data loss. At the same time, this embodiment determines the target data source corresponding to the data call request from all candidate data sources according to the target random number and the traffic configuration ratio, can quickly and accurately determine the target data source that meets the data call requirements, and improves the rationality of the call.

[0051] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0052] In one embodiment, a data source call processing device is provided, and the data source call processing device corresponds one-to-one to the data source call processing method in the above embodiment. Figure 8 As shown, the data source call processing device includes a data source configuration acquisition module 10, a candidate data source determination module 20, a flow ratio determination module 30 and a target data source determination module 40. Each functional module is described in detail as follows: The data source configuration acquisition module 10 is used to receive a data call request and acquire data source configuration information of all initial data sources according to the data call request; A candidate data source determination module 20 is used to obtain historical call information of all the initial data sources, and determine at least one candidate data source from all the initial data sources according to the historical call information and the data source configuration information; A flow ratio determination module 30, configured to determine the flow configuration ratio of each of the candidate data sources according to a preset ratio configuration rule and the data source configuration information, wherein the flow configuration ratio is used to characterize the distribution ratio of data flow and task load between each of the candidate data sources; The target data source determination module 40 is used to generate a target random number according to a preset random rule, and determine the target data source corresponding to the data call request from all the candidate data sources according to the target random number and the traffic configuration ratio.

[0053] In one embodiment, the data source configuration acquisition module 10 includes: A data source configuration information query unit, configured to query the data source configuration information corresponding to each of the initial data sources in the preset data source configuration list according to the data call request, and determine whether the query of each of the initial data sources is successful; The data source configuration information acquisition unit is used to acquire the data source configuration information of all the initial data sources if all the initial data sources are queried successfully.

[0054] In one embodiment, the candidate data source determination module 20 includes: A historical call success rate determination unit, used to obtain historical call information within a preset historical time period, and determine the historical call success rate corresponding to each of the initial data sources according to the historical call information; A historical call success rate comparison unit, used to select an initial data source from all the initial data sources as a data source to be compared, and determine whether the historical call success rate corresponding to the data source to be compared reaches a preset call success threshold; a data source to be retried determining unit, configured to determine the data source to be compared as a data source to be retried, if the historical call success rate corresponding to the data source to be compared has not reached a preset call success threshold, and to change the data source to be retried from an open state to a closed state, and to return to the step of selecting an initial data source from all the initial data sources as the data source to be compared, until all the initial data sources are traversed; A unit for determining a data source to be selected is used to determine the data source to be compared as a data source to be selected if the historical call success rate corresponding to the data source to be compared reaches a preset call success threshold, and keep the data source to be selected open, and return to the step of selecting an initial data source from all the initial data sources as the data source to be compared, until all the initial data sources are traversed.

[0055] In one embodiment, the candidate data source determination module 20 further includes: The call warning information generating unit is used to determine all the initial data sources as data sources to be retried and generate call warning information if all historical call success rates corresponding to the data sources to be compared have not reached a preset call success threshold.

[0056] In one embodiment, the flow ratio determination module 30 includes: A to-be-retried data source determination unit, used to determine whether there is a to-be-retried data source in the initial data source; A first flow configuration ratio determination unit is used to determine the initial flow ratio corresponding to each of the initial data sources as the flow configuration ratio of the candidate data source corresponding to the same initial data source when there is no data source to be retried, so as to obtain the flow configuration ratio of each of the candidate data sources; The second traffic configuration ratio determination unit is used to determine the initial traffic ratio corresponding to the data source to be retried as the ratio to be allocated when there is a data source to be retried, and to perform weighted calculation processing on the initial traffic ratio corresponding to the data source to be selected according to the ratio to be allocated, so as to obtain the traffic configuration ratio of each of the data sources to be selected.

[0057] In one embodiment, the flow ratio determination module 30 further includes: A flow ratio interval determining unit, configured to determine a flow ratio interval corresponding to each of the to-be-selected data sources according to the flow configuration ratio; The target random number matching unit is used to determine a target traffic ratio interval that matches the target random number from all the traffic ratio intervals, and determine the candidate data source corresponding to the target traffic ratio interval as the target data source.

[0058] In one embodiment, the target data source determination module 40 includes: A new data call request receiving unit, used to receive the new data call request and determine whether there is a data source to be retried in the initial data source; a retry time interval acquiring unit, configured to acquire the retry time interval of the data source to be retried if there is a data source to be retried in the initial data source; A data source calling unit to be retried is used to call the data source to be retried through the new data calling request when confirming that the retry time interval meets the preset retry interval threshold, and determine whether the data source to be retried is successfully called; The data source to be retried recovery unit is used to change the data source to be retried from a closed state to an open state if the call of the data source to be retried is successful.

[0059] For the specific definition of the data source call processing device, please refer to the definition of the data source call processing method above, which will not be repeated here. Each module in the above-mentioned data source call processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0060] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating system, computer-readable instructions and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database of the computer device is used to store data involved in the data source call processing method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instructions are executed by the processor, a data source call processing method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0061] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor executes the computer-readable instructions to implement the following steps: Receive a data call request, and obtain data source configuration information of all initial data sources according to the data call request; Acquire historical call information of all the initial data sources, and determine at least one candidate data source from all the initial data sources according to the historical call information and the data source configuration information; Determine the flow configuration ratio of each of the candidate data sources according to the preset ratio configuration rule and the data source configuration information, wherein the flow configuration ratio is used to characterize the distribution ratio of data flow direction and task load between each of the candidate data sources; A target random number is generated according to a preset random rule, and a target data source corresponding to the data call request is determined from all the candidate data sources according to the target random number and the traffic configuration ratio.

[0062] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media store computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the following steps are implemented: Receive a data call request, and obtain data source configuration information of all initial data sources according to the data call request; Acquire historical call information of all the initial data sources, and determine at least one candidate data source from all the initial data sources according to the historical call information and the data source configuration information; Determine the flow configuration ratio of each of the candidate data sources according to the preset ratio configuration rule and the data source configuration information, wherein the flow configuration ratio is used to characterize the distribution ratio of data flow direction and task load between each of the candidate data sources; A target random number is generated according to a preset random rule, and a target data source corresponding to the data call request is determined from all the candidate data sources according to the target random number and the traffic configuration ratio.

[0063] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0064] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0065] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data source call processing method, characterized in that: include: Receive a data call request, and obtain data source configuration information of all initial data sources according to the data call request; Acquire historical call information of all the initial data sources, and determine at least one candidate data source from all the initial data sources according to the historical call information and the data source configuration information; Determine the flow configuration ratio of each of the candidate data sources according to the preset ratio configuration rule and the data source configuration information, wherein the flow configuration ratio is used to characterize the distribution ratio of data flow direction and task load between each of the candidate data sources; A target random number is generated according to a preset random rule, and a target data source corresponding to the data call request is determined from all the candidate data sources according to the target random number and the traffic configuration ratio.

2. The data source call processing method according to claim 1, characterized in that: The step of acquiring data source configuration information of all initial data sources according to the data call request includes: According to the data call request, query the data source configuration information corresponding to each of the initial data sources in the preset data source configuration list, and determine whether the query of each of the initial data sources is successful; If all the initial data sources are queried successfully, the data source configuration information of all the initial data sources is obtained.

3. The data source call processing method according to claim 1, characterized in that: The data source configuration information includes a preset call success threshold corresponding to each of the initial data sources; The step of determining at least one candidate data source from all the initial data sources according to the historical call information and the data source configuration information includes: Acquire historical call information within a preset historical time period, and determine the historical call success rate corresponding to each of the initial data sources according to the historical call information; Selecting an initial data source from all the initial data sources as the data source to be compared, and determining whether the historical call success rate corresponding to the data source to be compared reaches a preset call success threshold; If the historical call success rate corresponding to the data source to be compared has not reached the preset call success threshold, the data source to be compared is determined as the data source to be retried, and the data source to be retried is changed from an open state to a closed state, and the step of selecting an initial data source from all the initial data sources as the data source to be compared is returned until all the initial data sources are traversed; If the historical call success rate corresponding to the data source to be compared reaches a preset call success threshold, the data source to be compared is determined as a selected data source, and the selected data source is kept in an open state, and the step of selecting an initial data source from all the initial data sources as the data source to be compared is returned until all the initial data sources are traversed.

4. The data source calling processing method according to claim 3, characterized in that: After determining whether the historical call success rate corresponding to the data source to be compared reaches a preset call success threshold, the method further includes: If all historical call success rates corresponding to the data sources to be compared have not reached a preset call success threshold, all the initial data sources are determined as data sources to be retried, and call alarm information is generated.

5. The data source calling processing method according to claim 3, characterized in that: The data source configuration information includes initial traffic ratios corresponding to each of the initial data sources; The determining the flow configuration ratio of each of the to-be-selected data sources according to the preset ratio configuration rule and the data source configuration information includes: Determine whether there is a data source to be retried in the initial data source; When there is no data source to be retried, the initial flow ratio corresponding to each of the initial data sources is determined as the flow configuration ratio of the to-be-selected data source corresponding to the same initial data source, to obtain the flow configuration ratio of each of the to-be-selected data sources; When there is a data source to be retried, the initial traffic ratio corresponding to the data source to be retried is determined as the ratio to be allocated, and the initial traffic ratio corresponding to the selected data source is weightedly calculated according to the ratio to be allocated to obtain the traffic configuration ratio of each of the selected data sources.

6. The data source call processing method according to claim 1, characterized in that: The step of determining a target data source corresponding to the data call request from all the candidate data sources according to the target random number and the traffic configuration ratio includes: Determine the flow ratio interval corresponding to each of the to-be-selected data sources according to the flow configuration ratio; A target traffic ratio interval matching the target random number is determined from all the traffic ratio intervals, and a candidate data source corresponding to the target traffic ratio interval is determined as the target data source.

7. The data source calling processing method according to claim 3, characterized in that: After determining the target data source corresponding to the data call request from all the candidate data sources according to the target random number and the traffic configuration ratio, the method further comprises: Receive a new data call request, and determine whether there is a data source to be retried in the initial data source; If there is a data source to be retried in the initial data source, obtaining a retry time interval of the data source to be retried; When it is confirmed that the retry time interval meets the preset retry time interval threshold, calling the data source to be retried through the new data calling request, and determining whether the calling of the data source to be retried is successful; If the data source to be retried is successfully called, the data source to be retried is changed from a closed state to an open state.

8. A data source call processing device, characterized in that: include: A data source configuration acquisition module, used to receive a data call request, and acquire data source configuration information of all initial data sources according to the data call request; A candidate data source determination module, used to obtain historical call information of all the initial data sources, and determine at least one candidate data source from all the initial data sources according to the historical call information and the data source configuration information; A flow ratio determination module, used to determine the flow configuration ratio of each of the candidate data sources according to a preset ratio configuration rule and the data source configuration information, wherein the flow configuration ratio is used to characterize the distribution ratio of data flow and task load between each of the candidate data sources; The target data source determination module is used to generate a target random number according to a preset random rule, and determine the target data source corresponding to the data call request from all the candidate data sources according to the target random number and the traffic configuration ratio.

9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer-readable instructions, the data source call processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by one or more processors, the one or more processors execute the data source call processing method according to any one of claims 1 to 7.

Images