Data scheduling method, device and equipment and storage medium thereof
By performing feasibility verification, compression and decompression adaptation processing on message data, the problem of insufficient data detection between upstream and downstream processing ends is solved, and the task processing efficiency and data accuracy of business processes are improved.
Patent Information
- Application Number
- CN202510383560.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-28
- Publication Date
- 2025-07-11
AI Technical Summary
In the prior art, data reporting and issuance between upstream and downstream processing ends in business process management is insufficient in data reporting and issuance, resulting in unqualified or non-formatted task data being sent, affecting the task processing efficiency and data accuracy of the overall business process.
The preset verification mechanism is used to verify the transmission feasibility of message data, compress and store it in the cache middleware, and transmit it to the target receiver using the preset transmission channel, and decompress and adapt it according to the data format document of the target receiver, and store it in the form of the target receiver.
Through the data scheduling method, the adaptive scheduling and flow of data in the business process is realized, the task processing efficiency is improved, and the accuracy of data scheduling is ensured.
Smart Images
Figure CN120301940A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of data processing and is applied to the scenario of data transmission and transfer between upstream and downstream execution tasks of message data, and relates to a data scheduling method, device, equipment and its storage medium. Background Art
[0002] In the current information system, with the development of information technology, business process management has become increasingly complex. Business process management often involves data reporting and distribution between upstream and downstream processing ends. For example: common project declaration and approval systems, financial data audit systems, etc. There are generally problems such as insufficient data detection when reporting or distributing task data, or data detection relying heavily on manual detection.
[0003] These problems lead to unqualified or unformatted task data being sent to the downstream processing end, resulting in low task processing efficiency of the overall business process, difficult to guarantee data accuracy, and being unfavorable for the adaptive scheduling and transfer of data in the business process. Summary of the Invention
[0004] The purpose of the embodiments of this application is to propose a data scheduling method, device, equipment and its storage medium to achieve the adaptive scheduling and transfer of data in the business process, thereby improving the task processing efficiency of the overall business process and ensuring the accuracy of data transfer.
[0005] In the first aspect, the embodiments of this application provide a data scheduling method, which adopts the following technical solutions:
[0006] A data scheduling method includes the following steps:
[0007] Obtain the message data of the target sending end;
[0008] Perform a transmission feasibility check on the message data using a preset check mechanism to obtain a check record document;
[0009] If the check record document shows that the message data can be transmitted, compress the message data and send the check record document and the compressed message data to a preset cache middleware together;
[0010] Use a preset first transmission channel to transmit the check record document and the compressed message data from the cache middleware to the target receiving end together, where the preset first transmission channel is the data transmission channel between the cache middleware and the target receiving end;
[0011] Obtain a preset data format document of the target receiving end, where the data format document stipulates the storage format of the message data in the target receiving end;
[0012] Decompress the compressed message data according to the data format document to obtain message data adapted to the decompressed data format, and update the verification record document according to the adaptation result;
[0013] Store the message data adapted to the decompressed data format in a preset form of the target receiving end, and cache the updated verification record document in the file storage area of the target receiving end.
[0014] In a second aspect, an embodiment of the present application further provides a data scheduling device, which adopts the following technical solutions:
[0015] A data scheduling device includes:
[0016] A message data acquisition module, configured to acquire message data of a target sending end;
[0017] A transmission feasibility verification module, configured to perform transmission feasibility verification on the message data by using a preset verification mechanism to obtain a verification record document;
[0018] A first data processing module, configured to compress the message data if the verification record document indicates that the message data can be transmitted, and send the verification record document and the compressed message data to a preset cache middleware;
[0019] A second data processing module, configured to transmit the verification record document and the compressed message data from the cache middleware to a target receiving end by using a preset first transmission channel, where the preset first transmission channel is a data transmission channel between the cache middleware and the target receiving end;
[0020] A data format document acquisition module, configured to acquire a data format document preset by the target receiving end, where the data format document specifies the storage format of message data in the target receiving end;
[0021] A third data processing module, configured to decompress the compressed message data according to the data format document to obtain message data adapted to the decompressed data format, and update the verification record document according to the adaptation result;
[0022] A data storage execution module, configured to store the message data adapted to the decompressed data format in a preset form of the target receiving end, and cache the updated verification record document in the file storage area of the target receiving end.
[0023] In a third aspect, an embodiment of the present application further provides a computer device, which adopts the following technical solutions:
[0024] A computer device includes a memory and a processor. Computer-readable instructions are stored in the memory, and when the processor executes the computer-readable instructions, the steps of the data scheduling method described above are implemented.
[0025] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, adopting the following technical solution:
[0026] A computer-readable storage medium has computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor, the steps of the data scheduling method described above are implemented.
[0027] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects:
[0028] In the data scheduling method of the embodiment of the present application, by obtaining the message data of the target sending end; performing a transmission feasibility check on the message data by using a preset check mechanism to obtain a check record document; if the check record document shows that the message data can be transmitted, then compressing the message data and sending the check record document and the compressed message data to a preset cache middleware together; using a preset first transmission channel to transmit the check record document and the compressed message data from the cache middleware to the target receiving end together, where the preset first transmission channel is the data transmission channel between the cache middleware and the target receiving end; obtaining a preset data format document of the target receiving end, where the data format document stipulates the storage format of the message data in the target receiving end; decompressing the compressed message data according to the data format document to obtain message data adapted to the decompressed data format, and updating the check record document according to the adaptation result; storing the message data adapted to the decompressed data format in a preset form of the target receiving end, and caching the updated check record document in the file storage area of the target receiving end. Applying the data scheduling method to the message data processing scenario, by first performing a transmission feasibility check, then using the cache middleware to implement the actual transmission of the message data, and combining the data format document provided by the target receiving end to convert the message data into a compatible data format. It ensures the adaptive scheduling and circulation of data in the business process. Description of the Drawings
[0029] In order to more clearly illustrate the solutions in the present application, the following will briefly introduce the drawings required for the description of the embodiments of the present application. Obviously, the following drawings are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0030] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;
[0031] Figure 2 is a flowchart of an embodiment of a data scheduling method according to the present application;
[0032] Figure 3 is Figure 2 a flowchart of a specific embodiment of step 202 shown;
[0033] Figure 4 is Figure 2 a flowchart of another specific embodiment of step 202 shown;
[0034] Figure 5 is a flowchart of a specific embodiment of encrypting and updating data in the data scheduling method described in the present application;
[0035] Figure 6 is a flowchart of a specific embodiment of decrypting and updating data in the data scheduling method described in the present application;
[0036] Figure 7 is Figure 2 a flowchart of a specific embodiment of step 206 shown;
[0037] Figure 8 is a schematic structural diagram of an embodiment of a data scheduling device according to the present application;
[0038] Figure 9 is a schematic structural diagram of an embodiment of a computer device according to the present application. Detailed implementation manners
[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects and not to describe a specific order.
[0040] References to "embodiments" in this specification mean that specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.
[0041] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0042] As Figure 1 shown, the system architecture 100 may include a terminal device 101, a network 102, and a server 103. The terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.
[0043] A user may use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications may be installed on the terminal device 101, such as a web browser application, a shopping application, a search application, an instant messaging tool, an email client, a social platform software, etc.
[0044] The terminal device 101 may be various electronic devices having a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012, or the mobile phone 1013, the terminal device 101 may also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop portable computer, a desktop computer, etc.
[0045] The server 103 may be a server providing various services, such as a background server supporting the pages displayed on the terminal device 101.
[0046] It should be noted that a data scheduling method provided in the embodiments of the present application is generally executed by the server. Correspondingly, a data scheduling device is generally disposed in the server.
[0047] It should be understood, Figure 1The numbers of the terminal devices, networks, and servers in [it] are merely illustrative. According to the implementation requirements, there can be any number of terminal devices, networks, and servers.
[0048] Continuing to refer to Figure 2 , a flowchart of an embodiment of a data scheduling method according to the present application is shown. The data scheduling method includes the following steps:
[0049] Step 201, obtain the message data of the target sending end.
[0050] In this embodiment, the target sending end includes any processing end that executes node tasks in the target service execution process. The obtaining of the message data of the target sending end refers to obtaining the message data that the task execution node is about to send from any processing end that executes node tasks in the target service execution process.
[0051] Specifically, first, according to the distinctiveness of the business field, for the upstream and downstream processing tasks in the same business field, the execution processes of the upstream and downstream processing tasks are sorted out according to the execution logic sequence. For example, in the financial data processing process, it at least includes financial data review and approval processing tasks. In the review and reporting stage, the message data is the financial data, including the data in the financial documents to be reviewed, the data in the financial statements, etc. The target sending end refers to the reporting end that needs to review the financial data; after the review is completed and the review end gives the corresponding approval, it is necessary to feedback the approval reply to the reporting end. At this time, the target sending end refers to the review end, and the message data refers to the approval text data made by the review end for the above financial data.
[0052] In this embodiment, according to the result of the execution process sorting, the nodes with upstream and downstream execution relationships can be numbered, and according to the numbering result, an automated node processing scheduling scheme can be generated. That is, assuming that the above review processing node is numbered 5 and the approval processing node is numbered 6, the generated automated node processing scheduling scheme at this time includes first executing the task node numbered 5 and then executing the task node numbered 6. Through the automated node processing scheduling scheme, the automated scheduling of data flow is realized, which is more automated and intelligent.
[0053] Step 202, perform a transmission feasibility check on the message data by using a preset check mechanism to obtain a check record document.
[0054] Specifically, the performing of a transmission feasibility check on the message data by using a preset check mechanism is used to determine whether the message data can be normally transmitted; the check record document is used to record the modification information of each transmission configuration item and the data format conversion information, etc. during the process of checking the message data.
[0055] The message data is verified for transmission feasibility by using a preset verification mechanism to ensure that the message data can be normally transmitted to the target receiving end; and the verification record document is generated to facilitate subsequent data flow query.
[0056] Step 203: If the verification record document shows that the message data can be transmitted, the message data is compressed, and the verification record document and the compressed message data are sent to a preset cache middleware.
[0057] The message data is compressed to ensure lighter data transmission; the verification record document and the compressed message data are sent together to a preset cache middleware. By utilizing the data cache middleware, direct data transmission between the target sending end and the target receiving end is avoided, so that each execution node can focus more on specific business processing.
[0058] Step 204: transmit the verification record document and the compressed message data from the cache middleware to the target receiving end by using a preset first transmission channel, wherein the preset first transmission channel is a data transmission channel between the cache middleware and the target receiving end.
[0059] The verification record document and the compressed message data are transmitted from the cache middleware to the target receiving end through the preset first transmission channel. By utilizing the data cache middleware, direct data transmission between the target sending end and the target receiving end is avoided, so that each execution node can focus more on specific business processing.
[0060] Step 205: Obtain a data format document preset by the target receiving end, wherein the data format document specifies a storage format of message data in the target receiving end.
[0061] By obtaining in advance a data format document preset by the target receiving end, the message data can be adaptively processed in combination with the data format document.
[0062] Step 206, decompress the compressed message data according to the data format document to obtain message data with adapted data format after decompression, and update the verification record document according to the adaptation result.
[0063] The compressed message data is decompressed through the data format document to obtain the message data adapted to the decompressed data format, realizing the adaptation conversion of the message data directly in combination with the data format document during the decompression process, ensuring that the decompressed message data received by the target receiving end is adapted to the display or processing mode of the target receiving end; updating the verification record document according to the adaptation result is used to record the data format conversion, facilitating the subsequent reverse lookup or traceability of the message data.
[0064] Step 207: Store the message data adapted to the decompressed data format in the preset form of the target receiving end, and cache the updated verification record document in the file storage area of the target receiving end.
[0065] By storing the message data adapted to the decompressed data format in the preset form of the target receiving end and caching the updated verification record document in the file storage area of the target receiving end, it is convenient for the target receiving end to perform data archiving and invocation.
[0066] In this embodiment, the data scheduling method can be executed by an RPA robot. By pre-setting the steps for executing the data scheduling method in the RPA robot, as well as the corresponding execution conditions and configuration files for each step execution, the data scheduling method can be automatically executed. Among them, RPA (Robotic Process Automation) is robotic process automation.
[0067] In this embodiment, by obtaining the message data of the target sending end; performing a transmission feasibility check on the message data by using a preset check mechanism to obtain a check record document; if the check record document indicates that the message data can be transmitted, compressing the message data and sending the check record document and the compressed message data to a preset cache middleware together; using a preset first transmission channel to transmit the check record document and the compressed message data from the cache middleware to the target receiving end together, where the preset first transmission channel is the data transmission channel between the cache middleware and the target receiving end; obtaining a preset data format document of the target receiving end, where the data format document specifies the storage format of the message data in the target receiving end; decompressing the compressed message data according to the data format document to obtain message data adapted to the decompressed data format, and updating the check record document according to the adaptation result; storing the message data adapted to the decompressed data format in a preset form of the target receiving end, and caching the updated check record document in the file storage area of the target receiving end. Applying the above data scheduling method to the message data processing scenario, by first performing a transmission feasibility check, then using the cache middleware to implement the actual transmission of the message data, and combining the data format document provided by the target receiving end to convert the message data into a compatible data format. It ensures the adaptive scheduling and transfer of data in the business process.
[0068] In this embodiment, the target sending end includes a data reporting sending end and a data distribution sending end, and obtaining the message data of the target sending end includes: obtaining the message data to be reported sent by the data reporting sending end; and obtaining the message data to be distributed sent by the data distribution sending end.
[0069] Continue to refer to Figure 3 , Figure 3 Yes Figure 2 is a flowchart of a specific embodiment of step 202 shown in the figure, including the following steps:
[0070] Step 301, according to preset environment configuration items, detecting whether the operating environment during the transmission of the message data is adapted;
[0071] Step 302, according to preset task configuration items, detecting whether the message data has completed data initialization in the transmission task;
[0072] Step 303, according to a preset data uniqueness detection mechanism, identifying whether the primary key information between the initialized data to be transmitted and the message data is unique;
[0073] Step 304, if it is detected that the operating environment during the transmission of the message data is adapted, the message data has completed data initialization in the transmission task, and the primary key information between the to-be-transmitted data after the initialization is completed and the message data is unique, then the message data passes the transmission feasibility check, and the check record document is generated.
[0074] Continue to refer to Figure 4 , Figure 4 Yes Figure 2 is a flowchart of another specific embodiment of step 202 shown in the figure, including the following steps:
[0075] Step 401, if it is detected that the operating environment during the transmission of the message data is not adapted, a first processing message is sent to the target monitoring end, and after the first processing is performed at the target monitoring end, the detection of the operating environment during the transmission of the message data continues, where the first processing message includes a processing message for changing the operating environment;
[0076] Step 402, if it is detected that the message data has not completed data initialization in the transmission task, the data initialization is re-executed, and the result of the data initialization is detected;
[0077] Step 403, if it is detected that the primary key information between the to-be-transmitted data after the initialization is completed and the message data is not unique, a second processing message is sent to the target monitoring end, and after the second processing is performed at the target monitoring end, the uniqueness detection of the primary key information between the to-be-transmitted data after the initialization is completed and the message data continues, where the second processing message includes a processing message for manually adjusting the primary key information;
[0078] Step 404, until the operating environment during the transmission of the message data is adapted, the message data has completed data initialization in the transmission task, and the primary key information between the to-be-transmitted data after the initialization is completed and the message data is unique, the message data passes the transmission feasibility check, and the check record document is generated.
[0079] By detecting whether the operating environment during the transmission of the message data is adapted; detecting whether the message data has completed data initialization in the transmission task; identifying whether the primary key information between the to-be-transmitted data after the initialization is completed and the message data is unique, so as to ensure the transmission feasibility of the message data, and when a transmission obstacle problem occurs, timely adjust the transmission configuration.
[0080] Continue to refer to Figure 5 , in some alternative implementation manners, after the compression process of the message data, the method further includes a step of encrypting and updating the data, Figure 5It is a flowchart of a specific embodiment for encrypting and updating data in the data scheduling method described in this application, including the following steps:
[0081] Step 501, using a preset encryption policy, encrypt the compressed message data to obtain encrypted compressed data;
[0082] Specifically, the preset encryption policy can be to encrypt the compressed message data using a symmetric encryption algorithm or to encrypt the compressed message data using an asymmetric encryption algorithm.
[0083] Step 502, update and replace the encrypted compressed data with the compressed message data.
[0084] By encrypting the compressed message data and updating and replacing the encrypted compressed data with the compressed message data, the security during the transmission of message data using a cache middleware is ensured.
[0085] In this embodiment, after performing the step of using a preset encryption policy to encrypt the compressed message data to obtain encrypted compressed data, the method further includes: sending the decryption key generated by the encryption process to the target receiving end using a preset second transmission channel, where the preset second transmission channel is the data transmission channel between the target sending end and the target receiving end.
[0086] Transmitting the decryption key through the data transmission channel between the target sending end and the target receiving end realizes that the encrypted data and the decryption key are transmitted through different data transmission channels respectively; and only the decryption key is directly transmitted between the two specific service processing ends of the target sending end and the target receiving end, avoiding direct transmission on the actual service data. Especially when both the target sending end and the target receiving end, two specific service processing ends, are involved in large - volume data processing, the directly transmitted data volume is small, which can significantly reduce the risk of the service processing end crashing caused by a large amount of transmitted data.
[0087] Continue to refer to Figure 6 , in some optional implementation manners, before performing step 206, the method further includes a step of decrypting and updating the data, Figure 6 It is a flowchart of a specific embodiment for decrypting and updating data in the data scheduling method described in this application, including the following steps:
[0088] Step 601, using the decryption key, decrypt the compressed message data to obtain decrypted compressed data;
[0089] Step 602: Update and replace the decompressed compressed data with the compressed message data.
[0090] Perform a decryption process on the encrypted compressed data to facilitate subsequent decompression of the compressed data.
[0091] Continue to refer to Figure 7 , Figure 7 Yes Figure 2 is a flowchart of a specific embodiment of step 206 shown below, including the following steps:
[0092] Step 701: Parse the data format document to determine the data format of the decompressed message data;
[0093] Step 702: Decompress the compressed message data according to the data format to obtain decompressed message data adapted to the data format;
[0094] Specifically, the decompression process of the compressed message data according to the data format to obtain decompressed message data adapted to the data format realizes the adaptive conversion of the message data in combination with the data format when decompressing the compressed message data.
[0095] Step 703: Record the data adaptability change information during the data format adaptation of the message data in the verification record document to complete the update of the verification record document.
[0096] This application obtains the message data of the target sender; performs a transmission feasibility check on the message data using a preset check mechanism to obtain a check record document; if the check record document shows that the message data can be transmitted, compresses the message data, and sends the check record document and the compressed message data to a preset cache middleware together; uses a preset first transmission channel to transmit the check record document and the compressed message data from the cache middleware to the target receiver, where the preset first transmission channel is the data transmission channel between the cache middleware and the target receiver; obtains a preset data format document of the target receiver, where the data format document specifies the storage format of the message data in the target receiver; decompresses the compressed message data according to the data format document to obtain message data adapted to the decompressed data format, and updates the check record document according to the adaptation result; stores the message data adapted to the decompressed data format in a preset form of the target receiver, and caches the updated check record document in the file storage area of the target receiver. Applying the above data scheduling method to the message data processing scenario, by first performing a transmission feasibility check, then using the cache middleware to implement the actual transmission of the message data, and combining with the data format document provided by the target receiver, the message data is converted into a suitable data format. This ensures the adaptive scheduling and transfer of data in the business process.
[0097] Embodiments of this application can obtain and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is a theory, method, technology, and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.
[0098] Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0099] In the embodiments of the present application, by obtaining the message data of the target sending end; performing a transmission feasibility check on the message data by using a preset check mechanism to obtain a check record document; if the check record document indicates that the message data can be transmitted, compressing the message data and sending the check record document and the compressed message data to a preset cache middleware together; using a preset first transmission channel to transmit the check record document and the compressed message data from the cache middleware to the target receiving end together, where the preset first transmission channel is the data transmission channel between the cache middleware and the target receiving end; obtaining a preset data format document of the target receiving end, where the data format document stipulates the storage format of the message data in the target receiving end; decompressing the compressed message data according to the data format document to obtain message data adapted to the decompressed data format, and updating the check record document according to the adaptation result; storing the message data adapted to the decompressed data format in a preset form of the target receiving end, and caching the updated check record document in the file storage area of the target receiving end. Applying the above data scheduling method to the message data processing scenario, by first performing a transmission feasibility check, then using the cache middleware to implement the actual transmission of the message data, and combining the data format document provided by the target receiving end to convert the message data into a suitable data format. It ensures the adaptive scheduling and transfer of data in the business process.
[0100] Further referring to Figure 8 , as an implementation of the method shown above Figure 2 , the present application provides an embodiment of a data scheduling device. This device embodiment corresponds to Figure 2 the method embodiment shown, and this device can be specifically applied to various electronic devices.
[0101] As Figure 8 shown, the data scheduling device 800 in this embodiment includes: a message data acquisition module 801, a transmission feasibility check module 802, a first data processing module 803, a second data processing module 804, a data format document acquisition module 805, a third data processing module 806, and a data storage execution module 807. Among them:
[0102] The message data acquisition module 801 is used to obtain the message data of the target sending end;
[0103] The transmission feasibility check module 802 is used to perform a transmission feasibility check on the message data by using a preset check mechanism to obtain a check record document;
[0104] The first data processing module 803 is configured to compress the message data if the message data can be transmitted as shown in the verification record document, and send the verification record document and the compressed message data to a preset cache middleware together;
[0105] The second data processing module 804 is configured to transmit the verification record document and the compressed message data from the cache middleware to the target receiving end through a preset first transmission channel, where the preset first transmission channel is a data transmission channel between the cache middleware and the target receiving end;
[0106] The data format document acquisition module 805 is configured to acquire a data format document preset by the target receiving end, where the data format document specifies the storage format of the message data in the target receiving end;
[0107] The third data processing module 806 is configured to decompress the compressed message data according to the data format document to obtain message data adapted to the decompressed data format, and update the verification record document according to the adaptation result;
[0108] The data storage execution module 807 is configured to store the message data adapted to the decompressed data format into a preset form of the target receiving end, and cache the updated verification record document into the file storage area of the target receiving end.
[0109] This application obtains the message data of the target sending end; uses a preset verification mechanism to perform a transmission feasibility verification on the message data to obtain a verification record document; if the verification record document shows that the message data can be transmitted, compresses the message data, and sends the verification record document and the compressed message data to a preset cache middleware together; uses a preset first transmission channel to transmit the verification record document and the compressed message data from the cache middleware to the target receiving end together, where the preset first transmission channel is the data transmission channel between the cache middleware and the target receiving end; obtains a preset data format document of the target receiving end, where the data format document stipulates the storage format of the message data in the target receiving end; decompresses the compressed message data according to the data format document to obtain message data adapted to the decompressed data format, and updates the verification record document according to the adaptation result; stores the message data adapted to the decompressed data format in a preset form of the target receiving end, and caches the updated verification record document in the file storage area of the target receiving end. Applying the above data scheduling method to the message data processing scenario, by first performing a transmission feasibility verification, then using the cache middleware to achieve the actual transmission of the message data, and combining the data format document provided by the target receiving end, converting the message data into a mutually adapted data format. It ensures the adaptive scheduling and transfer of data in the business process.
[0110] In this embodiment, the transmission feasibility verification module 802 includes: a running environment adaptation detection unit, an initialization completion detection unit, a primary key information uniqueness detection unit, and a verification record document generation unit. Among them:
[0111] The running environment adaptation detection unit is used to detect whether the running environment during the transmission of the message data is adapted according to the preset environment configuration items;
[0112] The initialization completion detection unit is used to detect whether the message data has completed data initialization in the transmission task according to the preset task configuration items;
[0113] The primary key information uniqueness detection unit is used to identify whether the primary key information between the initialized data to be transmitted and the message data is unique according to the preset data uniqueness detection mechanism;
[0114] A verification record document generation unit, which is used to generate the verification record document if it is detected that the operating environment during the transmission of the message data is adapted, the message data has completed data initialization in the transmission task, and the primary key information between the to-be-transmitted data after the initialization is completed and the message data is unique; and is also used until the operating environment during the transmission of the message data is adapted, the message data has completed data initialization in the transmission task, and the primary key information between the to-be-transmitted data after the initialization is completed and the message data is unique, the message data passes the transmission feasibility verification, and the verification record document is generated.
[0115] In this embodiment, the transmission feasibility verification module 802 further includes: an operating environment adjustment and re-detection unit, a re-initialization and detection unit, and a primary key information uniqueness adjustment and re-detection unit. Among them:
[0116] The operating environment adjustment and re-detection unit is used to send a first processing message to the target monitoring end if it is detected that the operating environment during the transmission of the message data is not adapted, and continue to detect the operating environment during the transmission of the message data after the first processing on the target monitoring end, where the first processing message includes a processing message for changing the operating environment;
[0117] The re-initialization and detection unit is used to re-execute data initialization and detect the data initialization result if it is detected that the message data has not completed data initialization in the transmission task;
[0118] The primary key information uniqueness adjustment and re-detection unit is used to send a second processing message to the target monitoring end if it is detected that the primary key information between the to-be-transmitted data after the initialization is completed and the message data is not unique, and continue to perform uniqueness detection on the primary key information between the to-be-transmitted data after the initialization is completed and the message data after the second processing on the target monitoring end, where the second processing message includes a processing message for manually adjusting the primary key information.
[0119] In this embodiment, the data scheduling device 800 further includes a compressed data encryption and update module, and the compressed data encryption and update module is used to encrypt the compressed message data by using a preset encryption policy to obtain the encrypted compressed data; and is also used to update and replace the encrypted compressed data with the compressed message data.
[0120] In this embodiment, the data scheduling device 800 further includes a decryption key sending module, which is configured to send the decryption key generated by the encryption process to the target receiving end through a preset second transmission channel, where the preset second transmission channel is the data transmission channel between the target sending end and the target receiving end.
[0121] In this embodiment, the data scheduling device 800 further includes a decryption processing and updating module, which is configured to use the decryption key to decrypt the compressed message data to obtain the decrypted compressed data; and further update and replace the decrypted compressed data with the compressed message data.
[0122] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disc, a Read-Only Memory (ROM), etc., or a Random Access Memory (RAM), etc.
[0123] It should be understood that although the steps in the flowchart of the accompanying drawings are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless clearly stated in this article, the execution of these steps has no strict order limit, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.
[0124] To solve the above technical problems, an embodiment of the present application also provides a computer device. Specifically, please refer to Figure 9 , Figure 9 which is the basic structural block diagram of the computer device in this embodiment.
[0125] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c that are communicatively connected to each other through a system bus. It should be noted that Figure 9Only a computer device 9 with a component memory 9a, a processor 9b, and a network interface 9c is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Among them, those skilled in the art of this technology can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0126] The computer device can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device can interact with the user through a keyboard, a mouse, a remote control, a touchpad, or a voice control device, etc.
[0127] The memory 9a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 9a can be an internal storage unit of the computer device 9, such as the hard disk or memory of the computer device 9. In other embodiments, the memory 9a can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 9a can also include both the internal storage unit and the external storage device of the computer device 9. In this embodiment, the memory 9a is generally used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions of a data scheduling method, etc. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or will be output.
[0128] In some embodiments, the processor 9b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 9b is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to run the computer-readable instructions stored in the memory 9a or process data, such as running the computer-readable instructions of the data scheduling method.
[0129] The network interface 9c may include a wireless network interface or a wired network interface. The network interface 9c is generally used to establish a communication connection between the computer device 9 and other electronic devices.
[0130] The computer device proposed in this embodiment belongs to the field of data processing technology and is applied to the scenario of message data transmission and circulation between upstream and downstream execution tasks. This application obtains the message data of the target sending end; uses a preset verification mechanism to verify the transmission feasibility of the message data to obtain a verification record document; if the verification record document shows that the message data can be transmitted, compresses the message data, and sends the verification record document and the compressed message data to a preset cache middleware; uses a preset first transmission channel to transmit the verification record document and the compressed message data from the cache middleware to the target receiving end, where the preset first transmission channel is the data transmission channel between the cache middleware and the target receiving end; obtains a preset data format document of the target receiving end, where the data format document specifies the storage format of the message data in the target receiving end; decompresses the compressed message data according to the data format document to obtain message data adapted to the decompressed data format, and updates the verification record document according to the adaptation result; stores the message data adapted to the decompressed data format in a preset form of the target receiving end, and caches the updated verification record document in the file storage area of the target receiving end. Applying the data scheduling method to the message data processing scenario, by first performing transmission feasibility verification, then using the cache middleware to achieve the actual transmission of the message data, and combining the data format document provided by the target receiving end to convert the message data into a compatible data format. It ensures the adaptive scheduling and circulation of data in the business process.
[0131] This application also provides another implementation manner, that is, to provide a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to execute the steps of a data scheduling method as described above.
[0132] The computer-readable storage medium proposed in this embodiment belongs to the technical field of data processing and is applied to the scenario of message data transmission and transfer between upstream and downstream execution tasks. This application obtains the message data of the target sending end; performs a transmission feasibility check on the message data by using a preset check mechanism to obtain a check record document; if the check record document shows that the message data can be transmitted, compress the message data, and send the check record document and the compressed message data to a preset cache middleware; use a preset first transmission channel to transmit the check record document and the compressed message data from the cache middleware to the target receiving end, where the preset first transmission channel is the data transmission channel between the cache middleware and the target receiving end; obtain a preset data format document of the target receiving end, where the data format document specifies the storage format of the message data in the target receiving end; perform decompression processing on the compressed message data according to the data format document to obtain message data adapted to the decompressed data format, and update the check record document according to the adaptation result; store the message data adapted to the decompressed data format in a preset form of the target receiving end, and cache the updated check record document in the file storage area of the target receiving end. Applying the above data scheduling method to the message data processing scenario, by first performing a transmission feasibility check, then using the cache middleware to implement the actual transmission of the message data, and combining the data format document provided by the target receiving end, the message data is converted into a suitable data format. It ensures the adaptive scheduling and transfer of data in the business process.
[0133] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that makes a contribution to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of this application.
[0134] Obviously, the embodiments described above are only a part of the embodiments of this application, rather than all of them. The accompanying drawings show the preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosed content of this application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions recorded in the foregoing specific embodiments, or perform equivalent replacements on some of the technical features. Any equivalent structures made by using the content of this application's specification and drawings, directly or indirectly applied in other related technical fields, are similarly within the scope of patent protection of this application. The non-company software tools or components that appear in the embodiments of this application are only introduced by way of example and do not represent actual use.
Claims
1. A data scheduling method, characterized in that, Including the following steps: Obtain the message data of the target sending end; Perform a transmission feasibility check on the message data using a preset verification mechanism to obtain a verification record document; If the verification record document shows that the message data can be transmitted, compress the message data and send the verification record document and the compressed message data to a preset cache middleware together; Use a preset first transmission channel to transmit the verification record document and the compressed message data from the cache middleware to the target receiving end together, where the preset first transmission channel is the data transmission channel between the cache middleware and the target receiving end; Obtain a preset data format document of the target receiving end, where the data format document stipulates the storage format of the message data in the target receiving end; Decompress the compressed message data according to the data format document to obtain message data adapted to the decompressed data format, and update the verification record document according to the adaptation result; Store the message data adapted to the decompressed data format into a preset form of the target receiving end, and cache the updated verification record document into the file storage area of the target receiving end.
2. The data scheduling method according to claim 1, wherein The target sending end includes a data reporting sending end and a data distribution sending end, and the obtaining of the message data of the target sending end includes: Obtain the message data to be reported sent by the data reporting sending end; and Obtain the message data to be distributed sent by the data distribution sending end.
3. The data scheduling method according to claim 1, wherein The step of performing a transmission feasibility check on the message data using a preset verification mechanism to obtain a verification record document includes: Detect whether the operating environment during the transmission of the message data is adapted according to preset environment configuration items; Detect whether the message data has completed data initialization in the transmission task according to preset task configuration items; Identify whether the primary key information is unique between the initialized data to be transmitted and the message data according to a preset data uniqueness detection mechanism; If it is detected that the operating environment during the transmission of the message data is adapted, the message data has completed data initialization in the transmission task, and the primary key information is unique between the initialized data to be transmitted and the message data, then the message data passes the transmission feasibility check, and the verification record document is generated.
4. The data scheduling method according to claim 3, wherein The step of performing a transmission feasibility check on the message data using a preset verification mechanism to obtain a verification record document further includes: If it is detected that the operating environment during the transmission of the message data is not adapted, send a first processing message to the target monitoring end, and continue to detect the operating environment during the transmission of the message data after the first processing on the target monitoring end, where the first processing message includes a processing message for changing the operating environment; If it is detected that the message data has not completed data initialization in the transmission task, re-perform data initialization and detect the data initialization result; If the primary key information between the to-be-transmitted data with initialization completed and the message data is not unique, a second processing message is sent to the target monitoring end, and after the second processing is performed at the target monitoring end, the uniqueness of the primary key information between the to-be-transmitted data with initialization completed and the message data is continuously detected, where the second processing message includes a processing message for manually adjusting the primary key information; Until the operating environment adaptation during the transmission of the message data, the data initialization of the message data has been completed in the transmission task, and the primary key information between the to-be-transmitted data with initialization completed and the message data is unique, the message data passes the transmission feasibility check, and the check record document is generated.
5. The data scheduling method according to claim 1, wherein Before performing the step of sending the check record document and the compressed message data to a preset cache middleware together, the method further includes: Using a preset encryption policy, encrypting the compressed message data to obtain encrypted compressed data; Updating and replacing the compressed message data with the encrypted compressed data.
6. The data scheduling method according to claim 5, wherein After performing the step of using a preset encryption policy to encrypt the compressed message data to obtain encrypted compressed data, the method further includes: Sending the decryption key generated by the encryption process to the target receiving end through a preset second transmission channel, where the preset second transmission channel is the data transmission channel between the target sending end and the target receiving end.
7. The data scheduling method according to claim 6, wherein Before performing the step of decompressing the compressed message data according to the data format document to obtain message data with an adapted data format after decompression and updating the check record document according to the adaptation result, the method further includes: Using the decryption key to decrypt the compressed message data to obtain decrypted compressed data; Updating and replacing the compressed message data with the decrypted compressed data; The step of decompressing the compressed message data according to the data format document to obtain message data with an adapted data format after decompression and updating the check record document according to the adaptation result specifically includes: Parsing the data format document to determine the data format of the message data after decompression; Decompressing the compressed message data according to the data format to obtain decompressed message data with a matching data format; Recording the data adaptation change information during the data format adaptation of the message data in the check record document to complete the update of the check record document.
8. A data scheduling device, characterized in that, Including: A message data acquisition module for acquiring the message data of the target sending end; A transmission feasibility check module for performing a transmission feasibility check on the message data using a preset check mechanism to obtain a check record document; A data first processing module for compressing the message data if the check record document shows that the message data can be transmitted, and sending the check record document and the compressed message data to a preset cache middleware together; A data second processing module, configured to transmit the verification record document and the compressed message data together from the cache middleware to a target receiving end by using a preset first transmission channel, where the preset first transmission channel is a data transmission channel between the cache middleware and the target receiving end; A data format document acquisition module, configured to acquire a data format document preset by a target receiving end, where the storage format of message data in the target receiving end is specified in the data format document; A data third processing module, configured to decompress the compressed message data according to the data format document to obtain message data adapted to the decompressed data format, and update the verification record document according to the adaptation result; A data storage execution module, configured to store the message data adapted to the decompressed data format into a preset form in the target receiving end, and cache the updated verification record document into a file storage area of the target receiving end.
9. A computer device, characterized in that, It includes a memory and a processor. Computer-readable instructions are stored in the memory, and when the processor executes the computer-readable instructions, the steps of the data scheduling method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that, Computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, the steps of the data scheduling method according to any one of claims 1 to 7 are implemented.