Reported data processing method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a reported data processing method and device, computer equipment and a storage medium. The method comprises the following steps: receiving a first data reporting request sent by a client, performing service calling processing according to the first data reporting request, and judging whether service calling abnormity occurs or not; if it is determined that the service calling is abnormal, receiving storage path information fed back by the client; writing target service data in the first data reporting request into a disk storage area according to the storage path information; and receiving a second data reporting request sent by the client, and when data reporting is successfully completed according to the second data reporting request, determining that the service calling exception has been eliminated, reading the target business data from the disk storage area and performing service calling processing again, so as to report the target business data when the recall does not have exception. According to the invention, delayed retry reporting can be realized, resource occupation is reduced, and the success rate and accuracy of data reporting are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a reported data processing method, device, computer equipment and storage medium. Background Art

[0002] Data reporting refers to the process in which users, merchants or third-party partners on a system or platform transmit their relevant data to the system for storage, processing and analysis. When the amount of reported data is large, in order to reduce resource usage costs and ensure data reporting quality, the existing technology uses remote procedure call (RPC) services to obtain data and message queue (MQ) asynchronous storage to complete data reporting.

[0003] The existing data reporting process includes three steps: calling the RPC reporting service, producing MQ messages, and consuming MQ data. Since this data reporting method involves both the RPC service and the MQ component, when the RPC service and the MQ component are unavailable, data loss will occur due to reporting exceptions. In order to avoid data loss, the existing method is to deploy the RPC service and the MQ component in a distributed manner and redundantly have enough nodes. When reporting an exception, different nodes are called through polling and retrying until success or all nodes are polled. This method has two disadvantages. On the one hand, the RPC service and the MQ component under the distributed deployment poll and retry when an exception occurs, which consumes a lot of resources. On the other hand, the polling and retrying are limited in the number of retries and time. When the RPC service and the MQ component are completely unavailable and the limit is reached, data loss will still occur, delayed retry cannot be achieved, and the retry success rate is low. Summary of the invention

[0004] Based on this, it is necessary to provide a reported data processing method, device, computer equipment and storage medium to address the above technical issues, so as to solve the problems of high resource consumption and low polling retry success rate when abnormalities occur in reported data.

[0005] A method for processing reported data, comprising:

[0006] Receive a first data reporting request sent by a client, perform service call processing according to the first data reporting request, and determine whether a service call exception occurs;

[0007] If it is determined that a service call exception occurs, sending a data storage request to the client, and receiving storage path information corresponding to the data storage request fed back by the client;

[0008] Writing the target service data in the first data reporting request into a disk storage area according to the storage path information;

[0009] Receive a second data reporting request sent by the client, and when data reporting is successfully completed according to the second data reporting request, determine that the service call exception has been eliminated, read the target business data from the disk storage area and re-perform service call processing, so as to perform reporting processing on the target business data when no service call exception occurs during the re-call.

[0010] A reporting data processing device, comprising:

[0011] A service calling module, used for receiving a first data reporting request sent by a client, performing service calling processing according to the first data reporting request and determining whether a service calling exception occurs;

[0012] A call exception handling module is used to send a data storage request to the client if it is determined that a service call exception occurs, and receive storage path information corresponding to the data storage request fed back by the client;

[0013] a disk storage module, configured to write the target service data in the first data reporting request into a disk storage area according to the storage path information;

[0014] The service recall module is used to receive a second data reporting request sent by the client, and when data reporting is successfully completed according to the second data reporting request, determine that the service call exception has been eliminated, read the target business data from the disk storage area and re-perform the service call processing, so as to perform reporting processing on the target business data when no service call exception occurs during the recall.

[0015] A computer device comprises a memory, a processor and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned reporting data processing method when executing the computer-readable instructions.

[0016] A computer-readable storage medium stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the above-mentioned reporting data processing method.

[0017] In the above-mentioned reported data processing method, device, computer equipment and storage medium, the reported data processing method stores the business data in a disk file when a service call exception occurs during service call processing according to the first data reporting request, thereby reducing resource usage, accurately judging the availability of the data reporting service, and preventing data loss. At the same time, when the service call is successfully completed according to the second data reporting request, it is judged that the service call has returned to normal, and the business data in the disk file is asynchronously read to re-process the service call. Based on the service exception and the disk cache, a retry reporting request is generated to perform asynchronous reporting of data, thereby realizing delayed retry of reporting data, avoiding read-write conflicts, and improving the success rate and accuracy of data reporting. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0019] Figure 1 It is a flowchart of a method for processing reported data in one embodiment of the present invention;

[0020] Figure 2 is a flow chart of step S10 in the method for processing reported data in one embodiment of the present invention;

[0021] Figure 3 is a flow chart of step S30 in the reported data processing method in one embodiment of the present invention;

[0022] Figure 4 is a flow chart of step S40 in the reported data processing method in one embodiment of the present invention;

[0023] Figure 5 is another flowchart of step S40 in the reported data processing method in one embodiment of the present invention;

[0024] Figure 6 is another flowchart of step S40 in the reported data processing method in one embodiment of the present invention;

[0025] Figure 7 is another flowchart diagram of step S40 in the reported data processing method in one embodiment of the present invention;

[0026] Figure 8 is a structural diagram of a reporting data processing device in one embodiment of the present invention;

[0027] Fig. 9is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] In the reported data processing method provided in this embodiment, a communication connection is pre-established between the client and the server. The client can exchange data with the server through the communication connection, and the server reports the client's data to the big data end by calling the data transmission service. The server can also send instructions or feedback information to the client through the communication connection, such as confirming the service call status, controlling the client to perform data operations, and feedback reporting processing results. Among them, the client includes but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0030] In one embodiment, if Figure 1 As shown, a method for processing reported data is provided, comprising the following steps S10-S40:

[0031] S10. Receive a first data reporting request sent by a client, perform service call processing according to the first data reporting request, and determine whether a service call exception occurs.

[0032] Understandably, the data reporting request is a request message used to report the business data in the client to the big data end. The client transmits the business data to the data warehouse of the big data end for processing, storage or analysis by sending a data reporting request to the server. After the server receives the data reporting request, it needs to call the data transmission service to realize the data reporting. The data transmission service is an important way for components in a distributed system to communicate, which can improve the availability, scalability and flexibility of the system. Service call processing refers to the process of calling the service or component used to perform data transmission operations, and the service call exception refers to the situation where the data transmission service cannot respond normally or is unavailable. The first data reporting request is a data reporting request for reporting the business data that currently needs to be reported.

[0033] In this embodiment, the data transmission service includes a distributed remote procedure call service and a message queue component. In order to respond to the data reporting request sent by the client, it is necessary to perform service call processing on the remote procedure call service and the message queue component, and determine whether a service call exception occurs (such as an overload or timeout of the remote procedure call service, or an abnormality of the message queue component). As long as any of the remote procedure call service and the message queue component cannot respond normally, it indicates that a service call exception occurs.

[0034] S20: If it is determined that a service call exception occurs, a data storage request is sent to the client, and storage path information corresponding to the data storage request fed back by the client is received.

[0035] Understandably, the data storage request is a request information for storing the business data that the client needs to report in the disk storage area. When a service call exception occurs, the client's business data cannot be transmitted to the data warehouse of the big data end. At this time, the business data is stored in the disk storage area. The disk storage area is a reliable and persistent data storage solution used to protect data when the system fails or anomalies occur. It can not only avoid data loss but also facilitate data access. In order to store business data in the disk storage area, the server needs to send a data storage request to the client, so that the client can determine the storage path information for storing the business data in the disk storage area after receiving the data storage request. The storage path information is the information used to identify the disk file when the business data is stored in the disk storage area, such as the relative path of the storage location of the disk file relative to the root directory of the disk partition and the file name of the disk file.

[0036] In this embodiment, after receiving the data storage request, the client displays a storage requirement input page on the client in response to the data storage request. The client user can input the specified requirement information through the input page according to the storage requirement, and the client generates the storage path information corresponding to the data storage request according to the input requirement information. The storage path information can be represented by the writeFile(String filePath,String fileName,Intger size) method, where String filePath represents the file path, String fileName represents the file name, and Intger size represents the upper limit of the file capacity. After the client generates the storage path information, it feeds back the storage path information to the server.

[0037] S30. Write the target business data in the first data reporting request into a disk storage area according to the storage path information.

[0038] Understandably, the server extracts the target business data from the first data reporting request, and the target business data is the business data generated by the client and currently needs to be reported to the data warehouse. The server searches for the disk file corresponding to the storage path information in the disk storage area according to the received storage path information, and writes the target business data into the disk file corresponding to the storage path information in the disk storage area.

[0039] S40. Receive a second data reporting request sent by the client, and when data reporting is successfully completed according to the second data reporting request, determine that the service call exception has been eliminated, read the target business data from the disk storage area and re-perform service call processing, so as to perform reporting processing on the target business data when no service call exception occurs during the re-call.

[0040] Understandably, after the target business data in the first data reporting request fails to be reported due to a service call exception, the server will continue to receive data reporting requests sent by the client to report the newly generated business data. The second data reporting request is a data reporting request for reporting the newly added business data after the first data reporting request, that is, the second data reporting request is a new data reporting request after the first data reporting request. The client will continuously generate new business data during use. Every time the client needs to report the new business data, the server will perform a service call process corresponding to the new business data, and determine whether a service call exception occurs. When it is determined that no service call exception occurs, the reporting of the new business data is completed. When the data reporting is successfully completed according to the second data reporting request, it indicates that the service call exception has been eliminated. After the service call returns to normal, the target business data is read from the disk storage area in a timely manner, and the service call process that was interrupted due to the exception is re-executed. For example, the readFileRetry method can be used when re-calling. After re-calling the service, the server will determine again whether an exception occurs. Only when no service call exception occurs during the re-call can the target business data be reported and sent to the data warehouse.

[0041] This embodiment stores the business data in a disk file when a service call exception occurs during service call processing according to the first data reporting request, thereby reducing resource usage, accurately judging the availability of the data reporting service, and preventing data loss. At the same time, this embodiment judges that the service call has returned to normal when the second data reporting request is successfully completed, asynchronously reads the business data in the disk file and re-processes the service call, generates a retry reporting request based on the service exception and the disk cache to perform asynchronous reporting of data, realizes delayed retry reporting of data, avoids read-write conflicts, and improves the success rate and accuracy of data reporting.

[0042] In a specific embodiment, in the scenario where the shopping platform client reports data, the shopping platform client needs to report the merchant's exposure, clicks and other settlement data to the data warehouse of the big data end. The existing method calls different nodes through polling retries when data reporting is abnormal until success or polling is completed. The existing node polling method and the reporting data processing method of the present invention are respectively applied to the application scenario of the shopping platform, and the statistical data results are sampled. When the number of service call exceptions on the server side within 5 minutes is 1000 times, the statistical result of the actual lost data in the existing method is 200 times, that is, the actual loss rate is 20%, while the statistical result of the actual lost data in the method of the present invention is 5 times, that is, the actual loss rate is 0.5%. It can be seen that compared with the existing method, the reporting data processing method of the present invention has a lower data loss rate in actual operation and a significantly better effect.

[0043] In one embodiment, the service call exception includes a remote service exception and a message queue exception. Figure 2 As shown, in step S10, that is, performing service call processing according to the first data reporting request and determining whether a service call exception occurs, includes:

[0044] S101, calling a remote procedure call service in response to the first data reporting request, and determining whether the remote procedure call service is successfully called;

[0045] S102: If the remote procedure call service fails, determining that a remote service exception occurs;

[0046] S103: If it is determined that the remote procedure call service is successfully called, a message queue component is called in response to the first data reporting request, and when it is determined that the message queue component fails to be called, it is determined that a message queue exception occurs.

[0047] Understandably, when the server performs service call processing according to the data reporting request, it needs to go through two calling links, first calling the remote procedure call service, and then calling the message queue component after the remote procedure call service is successfully called. Only after the message queue component is successfully called, it indicates that the service call is normal, that is, data reporting is achieved through the cooperation of the remote procedure call service and the message queue component. Service call exceptions include remote service exceptions and message queue exceptions. Remote service exceptions refer to the failure of the remote procedure call service to successfully respond to the data reporting request, and message queue exceptions refer to the failure of the message queue component to successfully respond to the data reporting request.

[0048] In a specific embodiment, after receiving the first data reporting request sent by the client, the server first calls the remote procedure call service in response to the first data reporting request, and determines whether the remote procedure call service is successfully called. Then, when it is determined that the remote procedure call service fails to call, it is determined that a remote service exception occurs. When it is determined that the remote procedure call service is successfully called, the message queue component is continued to be called and further determined whether the call is successful. When it is determined that the message queue component fails to call, it is determined that a message queue exception occurs.

[0049] This embodiment judges the two calling links separately in the service calling process in response to the first data reporting request, and can distinguish whether it is a remote service abnormality or a message queue abnormality, thereby achieving accurate judgment of the service calling abnormality.

[0050] In one embodiment, the storage path information includes file path information and file capacity information. Figure 3 As shown, in step S30, that is, writing the target service data in the first data reporting request into the disk storage area according to the storage path information, includes:

[0051] S301, searching in the disk storage area whether there is a target disk file corresponding to the file path information;

[0052] S302: if the target disk file does not exist, creating the target disk file in the disk storage area according to the storage path information and the file capacity information;

[0053] S303: If the target disk file exists, obtain the file capacity information of the target disk file and the target business data in the first data reporting request, and determine whether the target business data matches the file capacity information;

[0054] S304: When it is determined that the target business data matches the file capacity information, write the target business data into the target disk file.

[0055] Understandably, the storage path information includes file path information and file capacity information, wherein the file path information is information about the location of the disk file used to store the target business data in the disk storage area, and the file capacity information is storage space information about the disk file used to store the target business data in the disk storage area. The target disk file is a folder in the disk storage area used to store the target business data to be reported.

[0056] In a specific embodiment, in order to write the target business data into the disk storage area, first, the server extracts the file path information in the storage path information, and searches the disk storage area for a target disk file corresponding to the file path information specified by the user. When the target disk file cannot be found, the server creates a new target disk file corresponding to the file path information and the file capacity information in the disk storage area. When the target disk file can be found, it indicates that the user has specified an existing disk file as the target disk file. The server further obtains the file capacity information of the newly created target disk file (or the remaining capacity information of the existing disk file), and determines whether the target business data matches the file capacity information of the newly created target disk file (or the remaining capacity information of the existing disk file). If the data volume of the target business data is less than or equal to the file capacity information of the target disk file, it is determined that the target business data matches the file capacity information, and the target business data is written into the target disk file.

[0057] In another specific embodiment, after the target business data is written into the target disk file, a storage path directory list associated with the disk file storing the business data may be generated according to the service exception type, namely, a remote service exception directory list and a message queue exception directory list. Whenever a service call exception occurs and the current business data is stored in the target disk file in the disk storage area, if the type of the service call exception is a remote service exception, a storage path information corresponding to the target disk file is generated in the remote service exception directory list to facilitate the search for the target disk file.

[0058] This embodiment locates the target disk file through the file path information and the file capacity information, which can ensure that the target business data is correctly and completely written into the disk storage area, and effectively avoid data loss.

[0059] In one embodiment, if Figure 4 As shown, in step S40, that is, reading the target business data from the disk storage area, includes:

[0060] S401, generating a retry reporting request according to a preset time period, obtaining storage path information corresponding to the target business data, and determining whether the storage path information meets a preset retry condition according to the retry reporting request;

[0061] S402: If the preset retry condition is met, searching in the disk storage area whether there is a target disk file corresponding to the storage path information;

[0062] S403: When it is determined that there is a target disk file corresponding to the storage path information, parse the target disk file to obtain the target business data.

[0063] It is understandable that the preset time period is a fixed time range (the length of time between two retry operations) preset for performing periodic operations, such as the default setting of 1 minute. The retry report request is a request information used to re-call the data transmission service to report the business data stored in the disk storage area to the server. The preset retry condition is a condition information preset to determine whether the disk file corresponding to the storage path information is allowed to be read. Data reporting is continuous, and each service call exception will generate a disk file storing business data in the disk storage area. Each disk file can be stored as one or more data in the format of business data. When multiple business data reports are interrupted during a service call exception, there will be multiple disk files storing business data in the disk storage area at the same time.

[0064] In a specific embodiment, the number of disk files in the disk storage area is j, and there is a storage path directory list containing j storage path information, and the number of business data in each disk file is i. When it is detected that the service call exception has been eliminated, the server needs to first determine whether the number of disk files in the disk storage area is zero, that is, whether j is equal to zero. When j>0, it indicates that there is at least one storage path information in the storage path directory list. The server generates a retry report request according to a preset time period, selects one storage path information from the j storage path information according to the retry report request, and if the disk file corresponding to the currently selected storage path information is the target disk file, the business data stored in the disk file is the target business data. The server determines whether the storage path information meets the preset retry condition according to the retry report request, which is used to determine the availability of the storage path information. When the storage path information meets the preset retry condition, the server finds the target disk file corresponding to the storage path information in the disk storage area, parses the target disk file, and extracts the target business data. After the server reads the target business data from the target disk file, it deletes the target disk file from the disk storage area and deletes the storage path information corresponding to the target disk file in the storage path directory list to obtain an updated storage path directory list.

[0065] Furthermore, after reading the business data in a disk file, the server continues to determine whether the number of disk files in the disk storage area is zero, that is, whether j is equal to zero. When j=0, it indicates that there is no storage path information in the storage path directory list, and the generation of the retry report request is terminated. When j>0, it indicates that there is at least one storage path information in the storage path directory list, and the server continues to generate a new retry report request according to the preset time period and continues to read the business data.

[0066] This embodiment generates a retry report request according to a preset time period and reads business data from a disk storage area, thereby ensuring the availability and integrity of the data and improving the success rate of the retry.

[0067] In one embodiment, the storage path information includes file name information recording storage time information, and the retry report request includes retry time information. Figure 5 As shown, in step S401, that is, judging whether the storage path information meets the preset retry condition according to the retry reporting request, includes:

[0068] S4011, extracting storage time information from the storage path information, and determining a retry time interval according to the storage time information and the retry time information in the retry reporting request;

[0069] S4012: If the retry time interval is greater than a preset interval threshold, determine that the storage path information meets a preset retry condition.

[0070] Understandably, the storage time information refers to the time point when the target business data is stored in the disk storage area. The file name information is the name information of the disk file used to store the target business data in the disk storage area. When storing the target business data in the disk storage area, the file name needs to be determined according to the file naming rules. For example: if the report data is written to the disk storage area at 17:00 on December 28, 2023, the file name information seen under the disk file path is "-report_2023122817.txt". The retry time information refers to the time point when the retry report request is generated. The preset interval threshold is the minimum critical value of the time interval set in advance for determining whether the storage path information meets the preset retry condition. In order to avoid read-write conflicts, it is allowed to read the disk file that wrote the business data 1 hour ago from the disk storage area. For example: if the current system time is 18:00 on December 28, 2023, the disk file that wrote the business data between 17:00 and 18:00 on December 28, 2023 is not allowed to be read, that is, the storage path information does not meet the preset retry condition.

[0071] In a specific embodiment, after obtaining the storage path information corresponding to the target business data, it is necessary to extract the storage time information from the storage path information, calculate the difference between the storage time information and the retry time information in the retry report request, and obtain the retry time interval. When the retry time interval is greater than the preset interval threshold, it indicates that the disk file corresponding to the storage path information allows data to be read, and it is determined that the storage path information meets the preset retry condition.

[0072] This embodiment calculates and judges the storage time information and the retry time information in the retry reporting request to determine that, under the premise of meeting specific time conditions, the target business data corresponding to the storage path information can be read smoothly, thereby avoiding read and write conflicts in the disk storage area.

[0073] In one embodiment, if Figure 6 As shown, in step S40, that is, reading the target business data from the disk storage area and re-performing the service call processing, so as to perform reporting processing on the target business data when no service call exception occurs during the re-call, it also includes:

[0074] S404, re-calling the remote procedure call service in response to the retry report request, and determining whether the remote procedure call service is successfully called;

[0075] S405: If it is determined that the remote procedure call service is successfully called, a message queue component is called in response to the retry report request, and when the message queue component is successfully called, a data report queue corresponding to the target business data is generated through the message queue component;

[0076] S406: Send the target business data to a data warehouse according to the data reporting queue.

[0077] Understandably, after the server reads the target business data from the disk storage area, first, in response to the retry report request, the remote procedure call service is re-called, and it is determined whether the remote procedure call service is successfully called. Then, when it is determined that the remote procedure call service is successfully called, the message queue component is continued to be called and further determined whether the call is successful. When it is determined that the message queue component is successfully called, a data reporting queue corresponding to the target business data is generated by the message queue component. The message queue component is an intermediate component in a distributed system, which plays a buffering role in a highly concurrent distributed system. By storing messages in a queue, the sender and receiver of the message can operate independently of each other. The data reporting queue is a queue-type data that is output in sequence after the message queue component converts the target business data. The server sends the target business data to the data warehouse in the order of the data reporting queue, and finally completes the data reporting operation.

[0078] This embodiment performs a data reporting operation when it is determined that the re-service call process is normal, thereby ensuring reliable transmission and processing of business data and improving the stability and success rate of data reporting.

[0079] In one embodiment, if Figure 7 As shown, in step S40, that is, after reading the target service data from the disk storage area and re-performing the service call process, it also includes:

[0080] S407: If a service call exception occurs during the re-call, an alarm notification message is sent to a preset notification party.

[0081] Understandably, when the target business data is read from the disk storage area and the service call processing is re-performed, there is still a small probability that a service call exception (remote service exception or message queue exception) will occur, and the data reporting cannot be completed at this time. After the service call processing is re-performed, if the service call exception occurs again, the server will send an alarm notification message to the preset notification party. At the same time, since the target business data is deleted from the disk storage area after the target disk file is read from the target business data, the target business data will no longer be retried, and the preset notification party will perform data processing (such as data rollback) after receiving the alarm notification message. The way to send the alarm notification message includes but is not limited to email, SMS, instant messaging (such as DingTalk) and telephone. The preset notification party is a pre-set exception handling personnel for receiving notification information, such as system operation and maintenance personnel or developers. The alarm notification message is used to promptly notify the exception handling personnel to handle the problem when an exception occurs during the retry call. The alarm notification message contains details such as the exception type, error message, and occurrence time.

[0082] This embodiment helps operation and maintenance personnel to quickly respond to and handle abnormal situations, thereby ensuring the reliability and stability of the system by sending an alarm notification message when a service call exception occurs during re-service call processing.

[0083] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0084] In one embodiment, a reported data processing device is provided, and the reported data processing device corresponds one-to-one to the reported data processing method in the above embodiment. Figure 8 As shown, the reporting data processing device includes a service calling module 10, a calling exception processing module 20, a disk storage module 30 and a service recalling module 40. The detailed description of each functional module is as follows:

[0085] The service calling module 10 is used to receive a first data reporting request sent by a client, perform service calling processing according to the first data reporting request, and determine whether a service calling exception occurs;

[0086] The call exception processing module 20 is used to send a data storage request to the client if it is determined that a service call exception occurs, and receive storage path information corresponding to the data storage request fed back by the client;

[0087] The disk storage module 30 is used to write the target service data in the first data reporting request into the disk storage area according to the storage path information;

[0088] The service recall module 40 is used to receive a second data reporting request sent by the client, and when data reporting is successfully completed according to the second data reporting request, determine that the service call exception has been eliminated, read the target business data from the disk storage area and re-perform the service call processing, so as to perform reporting processing on the target business data when no service call exception occurs during the recall.

[0089] In one embodiment, the service calling module 10 includes:

[0090] A remote service calling unit, configured to call a remote procedure call service in response to the first data reporting request, and determine whether the remote procedure call service is successfully called;

[0091] A remote service exception determination unit, configured to determine that a remote service exception occurs if the remote procedure call service fails;

[0092] The message queue exception determination unit is used to call the message queue component in response to the first data reporting request if it is determined that the remote procedure call service call is successful, and determine that a message queue exception occurs when it is determined that the message queue component call fails.

[0093] In one embodiment, the disk storage module 30 includes:

[0094] A file path search unit, used for searching in the disk storage area whether there is a target disk file corresponding to the file path information;

[0095] a disk file creation unit, configured to create the target disk file in the disk storage area according to the storage path information and the file capacity information if the target disk file does not exist;

[0096] A file capacity determination unit, configured to obtain file capacity information of the target disk file and target service data in the first data reporting request if the target disk file exists, and determine whether the target service data matches the file capacity information;

[0097] The business data writing unit is used to write the target business data into the target disk file when it is determined that the target business data matches the file capacity information.

[0098] In one embodiment, the service recall module 40 includes:

[0099] a retry condition judgment unit, configured to generate a retry report request according to a preset time period, obtain storage path information corresponding to the target business data, and judge whether the storage path information meets the preset retry condition according to the retry report request;

[0100] a disk file search unit, configured to search the disk storage area for a target disk file corresponding to the storage path information if a preset retry condition is met;

[0101] The disk file parsing unit is used to parse the target disk file to obtain the target business data when it is determined that there is a target disk file corresponding to the storage path information.

[0102] In one embodiment, the service recall module 40 further includes:

[0103] a retry time interval determining unit, configured to extract storage time information from the storage path information, and determine a retry time interval according to the storage time information and the retry time information in the retry reporting request;

[0104] The retry time interval comparison unit is used to determine that the storage path information meets the preset retry condition if the retry time interval is greater than a preset interval threshold.

[0105] In one embodiment, the service recall module 40 further includes:

[0106] A remote service retry calling unit, configured to call the remote procedure call service again in response to the retry reporting request, and determine whether the remote procedure call service is successfully called;

[0107] a data reporting queue generating unit, configured to, if it is determined that the remote procedure call service is successfully called, call a message queue component in response to the retry reporting request, and generate a data reporting queue corresponding to the target business data through the message queue component when the message queue component is successfully called;

[0108] The business data reporting unit is used to send the target business data to the data warehouse according to the data reporting queue.

[0109] In one embodiment, the service recall module 40 further includes:

[0110] The notification information sending unit is used to send an alarm notification message to a preset notification party if a service call exception occurs during the re-call.

[0111] For the specific definition of the reporting data processing device, please refer to the definition of the reporting data processing method above, which will not be repeated here. Each module in the above-mentioned reporting data processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0112] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating system, computer-readable instructions and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database of the computer device is used to store data involved in the reported data processing method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instructions are executed by the processor, a reported data processing method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0113] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor executes the computer-readable instructions to implement the following steps:

[0114] Receive a first data reporting request sent by a client, perform service call processing according to the first data reporting request, and determine whether a service call exception occurs;

[0115] If it is determined that a service call exception occurs, sending a data storage request to the client, and receiving storage path information corresponding to the data storage request fed back by the client;

[0116] Writing the target service data in the first data reporting request into a disk storage area according to the storage path information;

[0117] Receive a second data reporting request sent by the client, and when data reporting is successfully completed according to the second data reporting request, determine that the service call exception has been eliminated, read the target business data from the disk storage area and re-perform service call processing, so as to perform reporting processing on the target business data when no service call exception occurs during the re-call.

[0118] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media store computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the following steps are implemented:

[0119] Receive a first data reporting request sent by a client, perform service call processing according to the first data reporting request, and determine whether a service call exception occurs;

[0120] If it is determined that a service call exception occurs, sending a data storage request to the client, and receiving storage path information corresponding to the data storage request fed back by the client;

[0121] Writing the target service data in the first data reporting request into a disk storage area according to the storage path information;

[0122] Receive a second data reporting request sent by the client, and when data reporting is successfully completed according to the second data reporting request, determine that the service call exception has been eliminated, read the target business data from the disk storage area and re-perform service call processing, so as to perform reporting processing on the target business data when no service call exception occurs during the re-call.

[0123] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0124] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0125] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for processing reported data, characterized in that: include: Receive a first data reporting request sent by a client, perform service call processing according to the first data reporting request, and determine whether a service call exception occurs; If it is determined that a service call exception occurs, sending a data storage request to the client, and receiving storage path information corresponding to the data storage request fed back by the client; Writing the target service data in the first data reporting request into a disk storage area according to the storage path information; Receive a second data reporting request sent by the client, and when data reporting is successfully completed according to the second data reporting request, determine that the service call exception has been eliminated, read the target business data from the disk storage area and re-perform service call processing, so as to perform reporting processing on the target business data when no service call exception occurs during the re-call.

2. The method for processing reported data according to claim 1, characterized in that: The service call exceptions include remote service exceptions and message queue exceptions; The performing service call processing according to the first data reporting request and determining whether a service call exception occurs includes: In response to the first data reporting request, calling a remote procedure call service, and determining whether the remote procedure call service is successfully called; If the remote procedure call service fails, determining that a remote service exception occurs; If it is determined that the remote procedure call service is successfully called, a message queue component is called in response to the first data reporting request, and when it is determined that the message queue component fails to be called, it is determined that a message queue exception occurs.

3. The method for processing reported data according to claim 1, characterized in that: The storage path information includes file path information and file capacity information; Writing the target service data in the first data reporting request into the disk storage area according to the storage path information includes: Searching the disk storage area for a target disk file corresponding to the file path information; If the target disk file does not exist, creating the target disk file in the disk storage area according to the storage path information and the file capacity information; If the target disk file exists, obtaining file capacity information of the target disk file and target service data in the first data reporting request, and determining whether the target service data matches the file capacity information; When it is determined that the target business data matches the file capacity information, the target business data is written into the target disk file.

4. The method for processing reported data according to claim 1, characterized in that: The reading the target business data from the disk storage area includes: Generate a retry report request according to a preset time period, obtain storage path information corresponding to the target business data, and determine whether the storage path information meets the preset retry condition according to the retry report request; If the preset retry condition is met, searching in the disk storage area whether there is a target disk file corresponding to the storage path information; When it is determined that there is a target disk file corresponding to the storage path information, the target disk file is parsed to obtain the target business data.

5. The method for processing reported data according to claim 4, characterized in that: The storage path information includes file name information recording storage time information, and the retry reporting request includes retry time information; The determining, according to the retry reporting request, whether the storage path information meets a preset retry condition includes: Extracting storage time information from the storage path information, and determining a retry time interval according to the storage time information and the retry time information in the retry reporting request; If the retry time interval is greater than a preset interval threshold, it is determined that the storage path information meets the preset retry condition.

6. The method for processing reported data according to claim 4, characterized in that: The step of reading the target service data from the disk storage area and re-performing the service call processing, so as to perform reporting processing on the target service data when no service call exception occurs during the re-call, further includes: Re-calling the remote procedure call service in response to the retry report request, and determining whether the remote procedure call service is successfully called; If it is determined that the remote procedure call service is successfully called, calling a message queue component in response to the retry report request, and when the message queue component is successfully called, generating a data report queue corresponding to the target business data through the message queue component; The target business data is sent to a data warehouse according to the data reporting queue.

7. The method for processing reported data according to claim 1, characterized in that: After reading the target service data from the disk storage area and re-performing the service call process, the method further includes: If a service call exception occurs during the re-call, an alarm notification message will be sent to the preset notification party.

8. A reporting data processing device, characterized in that: include: A service calling module, used for receiving a first data reporting request sent by a client, performing service calling processing according to the first data reporting request and determining whether a service calling exception occurs; A call exception handling module is used to send a data storage request to the client if it is determined that a service call exception occurs, and receive storage path information corresponding to the data storage request fed back by the client; a disk storage module, configured to write the target service data in the first data reporting request into a disk storage area according to the storage path information; The service recall module is used to receive a second data reporting request sent by the client, and when data reporting is successfully completed according to the second data reporting request, determine that the service call exception has been eliminated, read the target business data from the disk storage area and re-perform the service call processing, so as to perform reporting processing on the target business data when no service call exception occurs during the recall.

9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer-readable instructions, the reporting data processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by one or more processors, the one or more processors execute the reporting data processing method according to any one of claims 1 to 7.

Images