Data processing method, equipment and medium
By obtaining operator configuration information and using distributed message subscription systems, the development and maintenance problems during docking of different data platforms are solved, and efficient and secure messaging and system performance improvements are achieved.
Patent Information
- Application Number
- CN202410860718.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-12-29
- Filing Date
- 2024-06-28
- Publication Date
- 2025-07-08
AI Technical Summary
During the docking process of government and enterprise data platforms, different platforms use different custom protocols, the development and maintenance workload increases, the system efficiency decreases, and the docking complexity and cost increase, which cannot be managed in a unified way.
By obtaining the operator configuration information of the message to be sent, determining the sending processor and the result processor, using a distributed message subscription system for decoupling and asynchronous processing, setting a message sending lock and repush queue to achieve security, reliability and efficient delivery of messages.
Reduces development costs, improves messaging efficiency and system performance, ensures data consistency and security, reduces resource consumption, and enhances system reliability and scalability.
Smart Images
Figure CN120281644A_ABST
Abstract
Description
[0001] This application claims priority based on the invention patent application titled "A Protocol Configuration Method, Device, and Medium for Government-Enterprise Platforms" with the application number 202311867811.7 filed with the China National Intellectual Property Administration on December 29, 2023. The entire content of the above-mentioned Chinese patent application is incorporated herein by reference. Technical Field
[0002] This application relates to the fields of computer science and information technology, and specifically to a data processing method, device, and medium. Background Art
[0003] With the rapid development of today's society, both the government and enterprises are moving towards informatization and intelligentization, and the cooperation between the government and enterprises is becoming increasingly close, including cooperation in aspects such as information sharing, data interaction, and public services. Therefore, it is necessary to build a data platform to promote cooperation between the government and enterprises, improve information sharing and collaborative work efficiency, and provide more convenient public services for people.
[0004] Since data platforms usually handle sensitive information and important data, they have relatively high requirements for security. Customized protocols are used when building data platforms. Therefore, when a company connects to different data platforms, it needs to develop and process according to their specific protocols and cannot be managed in a unified way, which will lead to problems such as increased development and maintenance workload, reduced overall system efficiency, increased complexity and cost of connecting platforms. Summary of the Invention
[0005] To solve the above problems, this application proposes a data processing method, including:
[0006] Obtain a message to be sent, where the message to be sent represents a message that the enterprise side intends to send to the data platform;
[0007] Based on the content of the message to be sent, obtain the operator configuration information corresponding to the data platform, and determine a sending processor and a result processor based on the operator configuration information;
[0008] Through the sending processor, send an http request corresponding to the message to be sent, so that the data platform executes the http request and returns a processing result;
[0009] Through the result processor, parse the processing result, encapsulate it into a result object and return it. In one example, the content of the message to be sent is verified; if the verification passes, obtain the message sending lock corresponding to the message to be sent, and send the message based on the message sending lock.
[0010] In this technical solution, when it is determined that the message to be sent can be sent, the corresponding distributed lock is acquired, so that only one node or process can perform the sending operation each time, avoiding concurrent conflicts, ensuring data consistency, and being able to control the frequency and concurrency of message sending, avoiding excessive system load and resource waste caused by sending a large number of messages simultaneously, and also ensuring the sequentiality of message sending.
[0011] In one example, through a message sending listener, the content of the message to be sent is verified to determine whether the content of the message to be sent conforms to the preset rules corresponding to the data platform; if the content of the message to be sent conforms to the preset rules, the verification passes; if the content of the message to be sent does not conform to the preset rules, the verification fails; preferably, in the case of verification failure, exception handling is performed on the message to be sent.
[0012] In this technical solution, by using a message sending listener to capture the message to be sent and determine whether the message conforms to the rules, real-time inspection of the message can be carried out, messages that do not conform to the preset rules can be detected and processed in a timely manner, reducing the occurrence of errors, thereby improving the security and compliance of message sending.
[0013] In one example, according to the content of the message to be sent, the corresponding distributed lock is determined, and the distributed lock is determined as the message sending lock corresponding to the message to be sent.
[0014] In this technical solution, a distributed lock is determined according to the content of the message to be sent, and this distributed lock is used as the lock for message sending. The lock is acquired before sending the message to ensure that the same message will not be sent multiple times or to ensure the sequentiality and uniqueness of message sending. Ensure that the lock is held during message sending and released after the message is sent to avoid excessive holding of the lock and resource waste.
[0015] In one example, if it is determined based on the parsing result that the message to be sent is not processed successfully, the message to be sent is placed in the retransmission queue for subsequent re-attempted processing; and / or, if it is determined based on the parsing result that the message to be sent is processed successfully, the parsed result is encapsulated into a processing object, and the processing object is returned, and the message sending lock is released.
[0016] In this technical solution, a retransmission queue is set up, and the messages that fail to be sent are placed in the retransmission queue. The retransmission queue retrieves the messages from the queue again through the retry mechanism and processes them, which can ensure that the messages are not lost, enabling the system to process messages more reliably, and improving the reliability, sequentiality, flexibility, and scalability of message processing.
[0017] In one example, based on the operator configuration information, determine the types of the sending processor and the result processor; based on the types of the sending processor and the result processor, determine the corresponding sending processor and the result processor from the processor factory.
[0018] In this technical solution, according to the operator configuration information, determine the specific type of the message sending processor, and send the message sent by the enterprise through the corresponding sending processor to send an HTTP request that conforms to the data platform, without the need for additional processing, saving the development cost and improving the message transmission efficiency.
[0019] In one example, receive the HTTP request through the data platform, parse the request line and request headers of the HTTP request, obtain the content of the message to be sent, and execute the content of the message to be sent; generate an HTTP response according to the execution result and return the HTTP response; preferably, through the result processor, parse the processing result, encapsulate it into a result object and return it. Specifically, it includes: through the result processor, parse the HTTP response, encapsulate the parsing result into a result object and return it to the enterprise side; through the enterprise side, according to the result object, execute the operation corresponding to the message to be sent.
[0020] In this technical solution, encapsulating the parsing result into a result object can better encapsulate and abstract the response data for subsequent processing and transmission of the response data. In the subsequent processing process, when the structure of the HTTP response changes, only the definition of the result object and the relevant parsing logic need to be modified, without the need to modify the code of the calling party, reducing the code coupling, improving the maintainability and scalability of the code, and thus improving the development efficiency and code quality. By sending an HTTP request to the government-enterprise platform to achieve message sending, the security of message transmission can be improved, and through effective processing and optimization of the HTTP request, the performance of the system can be improved, the response time can be reduced, and the resource consumption can be reduced.
[0021] In one example, obtain the first message to be sent by the enterprise side, and through the distributed message subscription system, determine the second message consumed by the data platform; determine the second message as the message to be sent; preferably, the process of obtaining the first message to be sent by the enterprise and determining the second message consumed by the data platform through the distributed message subscription system specifically includes: obtaining the first message to be sent by the enterprise, decoupling the first message through the distributed message subscription system, and distributing the first message to the corresponding topics; according to the requirements of the enterprise, determine the consumption topic, and according to the consumption topic, obtain the second message consumed by the data platform.
[0022] In this technical solution, the decoupling of the message to be sent is achieved through a distributed message subscription system, enabling asynchronous processing of messages. After an enterprise sends a message to the distributed message subscription system, it can immediately return without waiting for the data platform to finish processing the message, improving the efficiency and throughput of message transmission. Moreover, the message can be saved in multiple topics, ensuring the reliability and persistence of the message.
[0023] On the other hand, this application also proposes a data processing method device, including:
[0024] At least one processor; and,
[0025] A memory communicatively connected to the at least one processor; wherein,
[0026] The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the data processing method as described in the above example.
[0027] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions, and the computer-executable instructions are set to: the data processing method as described in the above example.
[0028] The data processing method proposed by this application can bring the following beneficial effects:
[0029] According to the operator configuration information, the specific message sending processor is determined, and the message is sent as an HTTP request to the data platform through the specific sending processor. The data platform can directly receive it without the need for additional processing and development work due to different protocols, not only saving the development cost but also improving the efficiency of message transmission.
[0030] Moreover, through the decoupling of messages by the distributed message subscription system, asynchronous processing is achieved between the data platform and the enterprise side, and the messages are saved in multiple topics, not only improving the efficiency and throughput of message transmission but also ensuring the reliability and persistence of the messages. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The drawings described herein are used to provide a further understanding of this application and form a part of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:
[0032] Figure 1 is a schematic flowchart of a data processing method in an embodiment of this application;
[0033] Figure 2 is a schematic flowchart of the specific implementation process of a data processing method in an embodiment of this application;
[0034] Figure 3 It is a schematic diagram of a data processing device in an embodiment of the present application;
[0035] Figure 4 It is a schematic flowchart of a data processing method before claiming priority in an embodiment of the present application. Detailed implementation manners
[0036] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.
[0037] It should be noted that with the increasingly close cooperation between the government and enterprises, the number of data platforms established between governments and enterprises is also increasing. The construction of data platforms can realize cooperation such as information sharing, data interaction, and public services between the government and enterprises, help the government and enterprises achieve digital transformation, improve the management efficiency of the government and enterprises, optimize business processes, and thus provide more convenient public services and improve information sharing and collaborative work efficiency.
[0038] The following will detail the technical solutions provided by each embodiment of the present application in conjunction with the drawings.
[0039] As Figure 1 shown, an embodiment of the present application provides a data processing method, including:
[0040] S101: Obtain a message to be sent, where the message to be sent represents a message that the enterprise side is to send to the data platform.
[0041] Specifically, obtain the message to be sent by the enterprise side, which is a message that the enterprise side is to send to the data platform for the data platform to perform consumption operations. Among them, the data platform may be a government-enterprise platform.
[0042] Obtain the first message to be sent by the enterprise, and determine the second message consumed by the government-enterprise platform through a distributed message subscription system.
[0043] Obtain the first message to be sent by the enterprise side, and determine the second message consumed by the data platform through a distributed message subscription system; determine the second message as the message to be sent.
[0044] More specifically, obtain the first message to be sent by the enterprise, decouple the first message through a distributed message subscription system, and distribute the first message to the corresponding topic; according to the requirements of the enterprise, determine the consumption topic, and obtain the second message consumed by the data platform based on the consumption topic.
[0045] Obtain the first message to be sent by the enterprise, decouple the first message through a distributed message subscription system, and distribute the first message to the corresponding topic; according to the requirements of the enterprise, determine the consumption topic, and obtain the second message consumed by the government-enterprise platform based on the consumption topic.
[0046] It should be noted that when the enterprise side communicates with the data platform, since the data platform processes sensitive information and important data, it has relatively high requirements for security. A custom protocol is usually used when building the data platform. Therefore, when the enterprise docks with various data platforms, due to different data platforms using different http protocols, the client needs to process and develop separately for each data platform when sending http requests, resulting in problems of duplicate processing and development and being unable to be managed in a unified manner. This situation brings many inconveniences to the enterprise. For example, it increases the workload of developers for development and maintenance, reduces the overall efficiency of the system, and increases the complexity and cost of docking.
[0047] More specifically, through the system corresponding to the enterprise side, obtain the data or operations that the enterprise needs to dock with the data platform, convert these data or operations to be sent into messages, obtain the first message to be sent by the enterprise side, decouple the first message through a distributed message subscription system (such as the Kafka system), separate the message sender and receiver, so that they do not directly depend on each other.
[0048] Furthermore, distribute the first message to one or more topics, obtain the corresponding requirement document of the data platform, determine the second message that needs to be consumed by the data platform based on the requirement document, thereby determine the consumption topic, the data platform subscribes to the corresponding consumption topic, and obtains the second message within the consumption topic.
[0049] It should be noted that the Kafka distributed message subscription system allows multiple producers to publish messages to one or more topics and allows multiple consumers to subscribe to these topics and receive messages by using the publish / subscribe mode. As Figure 2 shown, decoupling the first message through the distributed message subscription system can enable asynchronous processing of messages. After the enterprise sends the message to the distributed message subscription system, it does not need to wait for the data platform to process the message before it can return immediately, improving the efficiency and throughput of message transmission, and being able to save the message in multiple topics, ensuring the reliability and persistence of the message.
[0050] Furthermore, verify the content of the message to be sent; if the verification passes, obtain the message sending lock corresponding to the message to be sent, and send the message based on the message sending lock.
[0051] Verify the content of the second message. After the verification passes, obtain the message sending lock.
[0052] Furthermore, determine the corresponding distributed lock according to the content of the message to be sent, and determine the distributed lock as the message sending lock corresponding to the message to be sent.
[0053] Specifically, through the message sending listener, verify the content of the message to be sent to determine whether the content of the message to be sent conforms to the preset rules corresponding to the data platform; if the content of the message to be sent conforms to the preset rules, the verification passes; if the content of the message to be sent does not conform to the preset rules, the verification fails; preferably, in the case of failed verification, perform exception handling on the message to be sent.
[0054] Through the message sending listener, verify the content of the second message to determine whether the content of the second message conforms to the preset rules; if the content of the second message conforms to the preset rules, the verification passes and obtain the message sending lock; if the content of the second message does not conform to the preset rules, the verification fails and perform exception handling on the second message.
[0055] If the content of the second message conforms to the preset rules, the verification passes and send the second message; determine the corresponding distributed lock according to the content of the second message, and call the distributed lock interface to obtain the message sending lock.
[0056] Specifically, after obtaining the message to be sent, the message sending listener is used to capture the message to be sent that is about to be sent in real time, and capture the content of the message to be sent. Based on the preset rules, verify the captured content of the message to be sent.
[0057] If the message to be sent does not conform to the preset rules, the message to be sent that does not conform to the preset rules is subjected to corresponding exception handling through the message sending listener, such as logging, rejecting the send, giving a prompt, etc.
[0058] If the message to be sent meets the preset rules, then according to the content of the message to be sent, determine the corresponding distributed lock, call the distributed lock interface through a function (such as the DistributedLocker function) to obtain the message sending lock. After obtaining the lock, perform the subsequent message sending operation. Among them, this process ensures that only one message sending thread can obtain the lock, avoiding concurrent conflicts, ensuring data consistency, and being able to control the frequency and concurrency of message sending, avoiding system overload and resource waste caused by sending a large number of messages simultaneously, and also ensuring the order of message sending.
[0059] Among them, set the preset rules according to the requirements document corresponding to the data platform. The purpose is to formulate verification logic to check whether the message content meets the specifications. The preset rules include rules for restricting the format, length, keywords, and legality of the message content. Through the verification of the message content based on the preset rules, it is possible to prevent illegal or non-conforming format messages from being transmitted to the data platform, improving data security. And, by capturing the message to be sent through the message sending listener and judging whether the message meets the rules, real-time inspection of the message can be carried out, timely discovering and processing messages that do not meet the preset rules, reducing errors, and thus improving the security and compliance of message sending.
[0060] S102: Based on the message content of the message to be sent, obtain the operator configuration information corresponding to the data platform, and determine the sending processor and the result processor based on the operator configuration information.
[0061] Specifically, based on the message content of the message to be sent, according to the requirements document corresponding to the data platform, obtain the operator configuration information corresponding to the data platform, and determine the sending processor and the result processor for the corresponding message according to the operator configuration information.
[0062] Obtain the operator configuration information according to the protocol corresponding to the government-enterprise platform, so as to determine the sending processor and the result processor.
[0063] According to the operator configuration information, determine the sending processor type and the result processor type; based on the sending processor type and the result processor type, determine the corresponding sending processor and result processor from the processor factory.
[0064] According to the protocol corresponding to the government-enterprise platform, obtain the operator configuration information according to the protocol; according to the operator configuration information, determine the sending processor type and the result processor type, and determine the specific sending processor and the specific result processor through the processor factory.
[0065] It should be noted that the operator configuration information is the docking method between different data platforms, such as message queue, direct database connection, file transfer, interface docking, etc. If the operator configuration information is not required in the corresponding requirement document in the data platform, the sending processor and result processor of the message to be sent are set to the default processor.
[0066] Furthermore, according to the type of the sending processor, a specific sending processor is obtained from the sending processor factory through a factory function (such as the SenderRequestHandlerFactory function). According to the type of the result processor, a specific result processor is obtained from the result processor factory. The sending processor is used to send the message sent by the enterprise through the processing operator function (such as the OperatorHandler function) and the send message function (such as the SendMessageService function), and the http request conforming to the data platform is sent through function processing without the need for additional processing, saving the development cost and improving the efficiency of message transmission.
[0067] S103: Send the http request corresponding to the message to be sent through the sending processor, so that the data platform executes the http request and returns a processing result.
[0068] Specifically, the data platform receives the http request, parses the request line and request header of the http request to obtain the content of the message to be sent, and executes the content of the message to be sent; an http response is generated according to the execution result and the http response is returned; more specifically, the result processor parses the http response, encapsulates the parsing result into a result object and returns it to the enterprise side; the enterprise side executes the operation corresponding to the message to be sent according to the result object.
[0069] Send the http request corresponding to the second message through the sending processor, execute the http request through the government-enterprise platform, and return a processing result.
[0070] The government-enterprise platform receives the http request, parses the request line and request header of the http request to obtain the content of the second message, and executes the content of the second message; an http response is generated according to the execution result and the http response is returned. The result processor parses the http response, encapsulates the parsing result into a result object and returns it to the enterprise; the enterprise executes the operation corresponding to the second message according to the result object.
[0071] Send the message to be sent to the data platform, and the specific sending processor sends the http request corresponding to the message to be sent. The http request includes parts such as a request line, request headers, request parameters, and a request body. The request line contains information such as the request method (such as the get method, post method, etc.), the URL path, and the protocol version.
[0072] Furthermore, after the data platform receives the http request sent by the enterprise side, it parses information such as the request line and request headers, obtains the request content of the enterprise side, and performs corresponding processing on the request content. After the processing is completed, the data platform generates an http response according to the processing result. The http response includes parts such as a response status code, response headers, and a response body. The response status code indicates the result of the request processing. For example, a response status code of 200 represents successful processing, and a response status code of 404 represents that the resource was not found, etc.
[0073] Even further, the data platform returns the generated http response to the enterprise side. By sending an http request to the data platform to implement message sending, it can improve the security of message transmission, and through effective processing and optimization of the http request, it can improve the system performance, reduce the response time, and reduce resource consumption.
[0074] S104: Through the result processor, parse the processing result, encapsulate it into a result object and return it.
[0075] Specifically, obtain the parsing result, and based on the parsing result, determine whether the second message was processed successfully.
[0076] After the enterprise side receives the http response, it parses and processes the response headers and response body of the http response through an inheritance function (such as the AbstractSenderRequestHandler function), and based on the parsing result, determines whether the http request initiated by the enterprise side was processed by the data platform.
[0077] If it is determined based on the parsing result that the message to be sent was not processed successfully, then put the message to be sent into the retransmission queue for subsequent re - attempt processing; and / or, if it is determined based on the parsing result that the message to be sent was processed successfully, encapsulate the parsed result into a processing object, return the processing object, and release the message sending lock.
[0078] If the second message was not processed successfully, put the second message into the retransmission queue for subsequent re - attempt processing; if the second message was processed successfully, encapsulate the parsing result into a processing object, return the processing object, and release the message sending lock.
[0079] If the processing is successful, perform corresponding operations according to specific requirements, such as rendering a page, extracting data, etc.
[0080] Further, according to the parsing result, if the processing is not successful and there is an exception in message sending, the second message is placed in the retry queue to be processed again later. The retry queue is a type of message queue used to store messages that failed to be sent and retry sending them, and the retry times or retry time intervals can be set. By using the retry queue, it can be ensured that messages will not be lost, enabling the system to process messages more reliably, improving the reliability, sequentiality, flexibility, and scalability of message processing.
[0081] Through the protocol used by the data platform, the operator configuration information is determined, thereby determining the specific message sending processor. The message is sent as an HTTP request to the data platform through the specific sending processor. The data platform can directly receive it without any additional processing and development work due to different protocols, not only saving the development cost but also improving the message delivery efficiency.
[0082] Moreover, the message is decoupled through the distributed message subscription system, enabling asynchronous processing between the data platform and the enterprise side. The message is saved in multiple topics, not only improving the message delivery efficiency and throughput but also ensuring the reliability and persistence of the message.
[0083] As Figure 3 shown, the embodiment of the present application also proposes a data processing device, including:
[0084] At least one processor; and,
[0085] A memory communicatively connected to the at least one processor; wherein,
[0086] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a data processing method as described in any of the above embodiments.
[0087] The embodiment of the present application also provides a non-volatile computer storage medium storing computer-executable instructions, and the computer-executable instructions are set as: the data processing method described in any of the above embodiments.
[0088] The embodiments in the present application are all described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments.
[0089] The device, medium, and method provided by the embodiments of this application correspond one by one. Therefore, the device and medium also have beneficial technical effects similar to those of their corresponding methods. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and medium will not be elaborated here.
[0090] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, system, or computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0091] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of flows and / or blocks in the flowchart and / or block diagram can also be implemented. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.
[0092] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.
[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.
[0094] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.
[0095] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash RAM. The memory is an example of computer-readable media.
[0096] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.
[0097] It should also be noted that the term "comprises", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but also other elements not expressly listed, or elements that are inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.
[0098] The above description is only for the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.
Claims
1. A data processing method, characterized in that, Including: Obtain the message to be sent, where the message to be sent represents a message that the enterprise side needs to send to the data platform; Based on the content of the message to be sent, obtain the operator configuration information corresponding to the data platform, and determine a sending processor and a result processor based on the operator configuration information; Through the sending processor, send an http request corresponding to the message to be sent, so that the data platform executes the http request and returns a processing result; Through the result processor, parse the processing result, encapsulate it into a result object and return it.
2. The data processing method according to claim 1, wherein After obtaining the message to be sent, the method further includes: Verify the content of the message to be sent; In the case of passing the verification, obtain a message sending lock corresponding to the message to be sent, and perform message sending based on the message sending lock.
3. The data processing method according to claim 2, wherein The verifying the content of the message to be sent includes: Through a message sending listener, verify the content of the message to be sent to determine whether the content of the message to be sent conforms to a preset rule corresponding to the data platform; If the content of the message to be sent conforms to the preset rule, the verification passes; If the content of the message to be sent does not conform to the preset rule, the verification fails; Preferably, in the case of the verification failing, perform exception handling on the message to be sent.
4. The data processing method according to claim 2, wherein The obtaining the message sending lock corresponding to the message to be sent includes: According to the content of the message to be sent, determine a corresponding distributed lock, and determine the distributed lock as the message sending lock corresponding to the message to be sent.
5. The data processing method according to claim 2, characterized in that, After parsing the processing result through the result processor, the method further includes: If it is determined based on the parsing result that the message to be sent is not processed successfully, put the message to be sent into a re-push queue for subsequent re-attempts at processing; And / or If it is determined based on the parsing result that the message to be sent is processed successfully, encapsulate the parsed result into a processing object, return the processing object, and release the message sending lock.
6. The data processing method according to claim 1, characterized in that The determining the sending processor and the result processor based on the operator configuration information specifically includes: According to the operator configuration information, determine the sending processor type and the result processor type; Based on the sending processor type and the result processor type, determine the corresponding sending processor and the result processor from a processor factory.
7. The data processing method according to claim 1, wherein The executing the http request through the data platform and returning a processing result specifically includes: Through the data platform, receive the http request, parse the request line and request headers of the http request, obtain the content of the message to be sent, and execute the content of the message to be sent; Generate an http response according to the execution result and return the http response; Preferably, the parsing the processing result through the result processor, encapsulating it into a result object and returning it specifically includes: Through the result processor, parse the http response, encapsulate the parsing result into a result object and return it to the enterprise side; Through the enterprise side, according to the result object, perform the operation corresponding to the message to be sent.
8. The data processing method according to claim 1, characterized in that The obtaining of the message to be sent includes: Obtain the first message to be sent by the enterprise side, and determine the second message consumed by the data platform through a distributed message subscription system; Determine the second message as the message to be sent; Preferably, the obtaining of the first message to be sent by the enterprise and determining the second message consumed by the data platform through a distributed message subscription system specifically includes: Obtain the first message to be sent by the enterprise, decouple the first message through a distributed message subscription system, and distribute the first message to corresponding topics; According to the requirements of the enterprise, determine the consumption topic, and according to the consumption topic, obtain the second message consumed by the data platform.
9. A data processing device, characterized in that, Includes: At least one processor; And, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method according to any one of claims 1 to 8.
10. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions can implement the data processing method according to any one of claims 1 to 8.