Data monitoring method and device, storage medium and electronic equipment

Abstract

The invention provides a data monitoring method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining the incremental log of each business system in a current obtaining period, and achieving the obtaining of the waybill related data of each waybill, of which the state is changed, of each business system in the current obtaining period; according to the method, data barriers of all service systems are broken through, a monitoring system and all the service systems are decoupled, waybill related data are written into a wide table, and a final-stage node mark is configured for a waybill meeting a final-stage node strategy, so that only the waybill configured with the final-stage node mark in the wide table is monitored; according to the invention, monitoring of the return state of the receivable amount of the waybill is realized, and the influence on the performance of the monitoring system is reduced.

Description

Technical Field

[0001] The present application relates to the field of logistics technology, and in particular to a data monitoring method and device, a storage medium and an electronic device. Background Art

[0002] At present, in the logistics field, various business systems (such as waybill systems, distribution systems, and warehouse management systems) have different storage media and storage structures, which results in the data of various business systems being unable to communicate directly with each other, making it impossible to monitor the collection status of the receivables of the waybills. Summary of the invention

[0003] The present application provides a data monitoring method and device, a storage medium and an electronic device, the purpose of which is to solve the problem in the prior art that the data of various business systems cannot be directly communicated with each other, resulting in the inability to monitor the collection status of the receivable amount of the waybill.

[0004] In order to achieve the above objectives, this application provides the following technical solutions:

[0005] A data monitoring method, applied to a monitoring system, comprising:

[0006] According to a preset acquisition cycle, the incremental log of each business system in the current acquisition cycle is acquired; the incremental log includes the waybill-related data of each waybill whose status has changed in the current acquisition cycle;

[0007] For the waybill-related data of each waybill included in each incremental log, if the waybill-related data of the waybill satisfies the preset final node strategy corresponding to the waybill, the waybill-related data of the waybill is written into the preset wide table, and a final node mark is configured for the waybill in the wide table; otherwise, the waybill-related data of the waybill is directly written into the preset wide table;

[0008] For each waybill configured with the final node mark in the wide table, if the waybill-related data of the waybill satisfies the preset lower receivable strategy corresponding to the waybill, a lower receivable mark is configured for the waybill in the wide table, and the final node mark of the waybill is deleted; the lower receivable mark is used to mark that the receivable amount of the waybill has been collected.

[0009] The above method may optionally further include:

[0010] If the waybill-related data of the waybill does not satisfy the preset lower receivable strategy corresponding to the waybill, the last-level node mark of the waybill is deleted.

[0011] The above method may, optionally, further include: after configuring a receivable mark for the waybill in the wide table:

[0012] Generate a reason for the waybill to be collected, and write the reason for the waybill to the corresponding position in the wide table.

[0013] In the above method, optionally, obtaining incremental logs of each business system in the current acquisition cycle according to a preset acquisition cycle includes:

[0014] According to the preset acquisition cycle, the incremental logs of each business system in the current acquisition cycle are obtained through the data pipeline component.

[0015] In the above method, optionally, the step of writing the waybill-related data of the waybill into a preset wide table includes:

[0016] Based on the key fields in the preset wide table, the data items corresponding to each key field are extracted from the waybill-related data of the waybill, and the data items corresponding to each key field are written into the corresponding position of the waybill in the wide table; the multiple key fields include at least a waybill number field, a waybill type field, a packing time field and a receivable amount field.

[0017] The above method, optionally, after writing the waybill-related data of the waybill into a preset wide table, further comprises:

[0018] According to the preset summary cycle, according to the outlets to which each target waybill in the wide table belongs, the receivable amount of each target waybill is summarized to obtain the summary amount of each outlet; the target waybill is a waybill whose data item corresponding to the packaging time field is not empty and the receivable flag is not configured;

[0019] For each outlet, if the aggregate amount of the outlet is greater than a preset threshold, an early warning message is sent to the person in charge of the outlet through a preset communication method based on the aggregate amount of the outlet.

[0020] The above method, optionally, after writing the waybill-related data of the waybill into a preset wide table, further comprises:

[0021] For each target waybill in the wide table, if the packing time of the target waybill exceeds the preset time threshold, the preset accountability process is called to perform accountability processing on the target waybill to obtain an accountability result. If the accountability result is that the organization is responsible, a receivable message queue message is sent to the responsible organization, and a receivable flag is configured for the target waybill. If the accountability result is that the user is responsible, a loss return order is generated and pushed to the responsible user. After the user completes the payment of the loss return order, a receivable flag is configured for the target waybill. The target waybill is a waybill whose data item corresponding to the packing time field is not empty and for which a receivable flag is not configured.

[0022] A data monitoring device is applied to a monitoring system, and the method comprises:

[0023] An acquisition unit, used to acquire, according to a preset acquisition cycle, an incremental log of each business system in a current acquisition cycle; the incremental log includes waybill-related data of each waybill whose status has changed in the current acquisition cycle;

[0024] A writing unit, for writing the waybill-related data of each waybill included in each incremental log into a preset wide table if the waybill-related data of the waybill satisfies the preset final node strategy corresponding to the waybill, and configuring a final node mark for the waybill in the wide table; otherwise, directly writing the waybill-related data of the waybill into the preset wide table;

[0025] A configuration unit is used to configure a lower receivable mark for each waybill configured with the final node mark in the wide table, and delete the final node mark of the waybill if the waybill-related data of the waybill meets the preset lower receivable strategy corresponding to the waybill; the lower receivable mark is used to mark that the receivable amount of the waybill has been collected.

[0026] A storage medium stores an instruction set, wherein the instruction set implements the above-mentioned data monitoring method when executed by a processor.

[0027] An electronic device, comprising:

[0028] A memory for storing at least one set of instructions;

[0029] The processor is used to execute the instruction set stored in the memory, and implement the data monitoring method as described above by executing the instruction set.

[0030] Compared with the prior art, this application has the following advantages:

[0031] The present application provides a data monitoring method and device, a storage medium and an electronic device, which is applied to a monitoring system, and acquires the incremental log of each business system in the current acquisition cycle according to a preset acquisition cycle; the incremental log includes the waybill-related data of each waybill whose status changes in the current acquisition cycle; for the waybill-related data of each waybill included in each incremental log, if the waybill-related data of the waybill satisfies the preset final node strategy corresponding to the waybill, the waybill-related data of the waybill is written into a preset wide table, and a final node mark is configured for the waybill in the wide table; otherwise, the waybill-related data of the waybill is directly written into the preset wide table; for each waybill configured with the final node mark in the wide table, if the waybill-related data of the waybill satisfies the preset lower receivable strategy corresponding to the waybill, a lower receivable mark is configured for the waybill in the wide table, and the lower receivable mark of the final node mark of the waybill is deleted to mark that the receivable amount of the waybill has been collected. It can be seen that the present application scheme, by obtaining the incremental logs of each business system in the current acquisition cycle, realizes the acquisition of the waybill-related data of each waybill whose status has changed in each business system within the current acquisition cycle, breaks through the data barriers of each business system, and decouples the monitoring system and each business system, and writes the waybill-related data into a wide table, and configures the last-level node mark for the waybill that meets the last-level node strategy, so that only the waybill configured with the last-level node mark in the wide table is monitored, thereby realizing the monitoring of the collection status of the receivable amount of the waybill and reducing the performance impact on the monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0033] Figure 1 A method flow chart of a data monitoring method provided in this application;

[0034] Figure 2 An example diagram of a data monitoring method provided in this application;

[0035] Figure 3 Another method flow chart of a data monitoring method provided by the present application;

[0036] Figure 4 Another method flow chart of a data monitoring method provided by the present application;

[0037] Figure 5 Another example diagram of a data monitoring method provided by the present application;

[0038] Figure 6 A schematic diagram of the structure of a data monitoring device provided in this application;

[0039] Figure 7 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0042] It should be noted that the concepts such as "first" and "second" mentioned in the present application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0043] It should be noted that the modifications of "one" and "plurality" mentioned in the disclosure of this application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0044] The present application can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, etc.

[0045] The present application embodiment provides a data monitoring method, which can be applied to a monitoring system. The method flow chart of the method is as follows: Figure 1 As shown, specifically including:

[0046] S101. According to a preset acquisition cycle, the incremental logs of each business system in the current acquisition cycle are acquired.

[0047] In this embodiment, the incremental log of each business system in the current acquisition cycle is obtained according to the preset acquisition cycle; wherein the incremental log is a log after the format conversion of the binary log Binlog newly added in the current acquisition cycle; since the binary log of the business system is used to record all database table structure changes and table data modifications of the business system, by obtaining the incremental log of each business system in the current acquisition cycle, the waybill-related data of the waybill of each business system in the current acquisition cycle can be obtained, that is, the incremental log in the current acquisition cycle includes the waybill-related data of each waybill whose status has changed in the current acquisition cycle;

[0048] The acquisition period is a manually set period and can be changed according to needs.

[0049] In this embodiment, the process of obtaining the incremental logs of each business system in the current acquisition cycle according to the preset acquisition cycle specifically includes:

[0050] According to the preset acquisition cycle, the incremental logs of each business system in the current acquisition cycle are obtained through the data pipeline component.

[0051] In this embodiment, the data pipeline component simulates sending data pull requests from the database to the database of each business system through the interactive protocol according to the preset acquisition cycle. After receiving the data pull request, the database of each business system sends its own incremental log in the current acquisition cycle to the data pipeline component. After receiving the incremental log, the data pipeline component parses the incremental log, converts the format of the incremental log, and sends the incremental log after format conversion to the monitoring system. Specifically, the incremental log after format conversion is sent to the monitoring system through the message middleware.

[0052] Among them, message middleware includes but is not limited to Kafka message middleware, MQ message middleware and distributed search and analysis engine ElasticSearch message middleware.

[0053] It should be noted that the process of converting the format of the incremental log specifically includes: converting the format of the incremental log according to the message format of the message middleware.

[0054] It should be noted that the data pipeline component can be a data pipeline cluster component, that is, it includes multiple business processors, and the incremental logs are processed by multiple business processing machines, that is, the incremental logs are parsed and the format of the incremental logs is converted, thereby improving the processing efficiency of the incremental logs.

[0055] In this embodiment, the monitoring system receives the incremental log after format conversion sent by the data pipeline component.

[0056] See also Figure 2 The process of obtaining the incremental logs of each business system in the current acquisition cycle is illustrated as follows:

[0057] The data pipeline component Canal sends a dump request (i.e., data pull request) to the MySQL database of the business system according to the acquisition cycle. After receiving the dump request, the MySQL database sends the binary log Binlog added during the acquisition cycle to the data pipeline component Canal. The data pipeline component parses and converts the format of the binary log Binlog that will be added during the acquisition cycle, and sends the format-converted Binlog to the Kafka message middleware, MQ message middleware, or ElasticSearch message middleware, providing incremental data subscription and consumption for the monitoring system, so that the monitoring system can obtain the format-converted binary log Binlog added during the acquisition cycle through the message middleware.

[0058] Among them, MySQL is a relational database management system, Binlog is a binary log, dump request is a data export request, Canal is a data pipeline component, Kafka is a high-throughput distributed publish-subscribe messaging system, MQ (MessageQueue), and ElasticSearch is a distributed search and analysis engine.

[0059] S102. For the waybill-related data of each waybill included in each incremental log, determine whether the waybill-related data satisfies the preset terminal node strategy corresponding to the waybill. If so, execute S103; if not, execute S104.

[0060] In this embodiment, the final node strategies corresponding to different waybill types are preset. It should be noted that the final node strategies corresponding to different waybill types may be the same or different.

[0061] It should be noted that the final node strategy includes at least one of the preset multiple conditions; illustratively, the multiple conditions include successful delivery of waybills, completion of orders, reverse order exchange, completion of reconciliation, completion of loss reporting and completion of return to warehouse.

[0062] In this embodiment, for the waybill-related data of each waybill included in each incremental log, it is determined whether the waybill-related data satisfies the preset final node strategy corresponding to the waybill. Specifically, it is determined whether the waybill-related data satisfies any one condition in the preset final node strategy corresponding to the waybill. If the waybill-related data satisfies any one condition in the final node strategy, it is determined that the waybill-related data satisfies the final node strategy corresponding to the waybill. If none of the waybill-related data satisfies any one condition in the final node strategy, it is determined that the waybill-related data does not satisfy the final node strategy corresponding to the waybill.

[0063] The final node strategy corresponding to the waybill is the final node strategy corresponding to the waybill type of the waybill.

[0064] S103: Write the waybill-related data of the waybill into a preset wide table, and configure a final node mark for the waybill in the wide table.

[0065] In this embodiment, for each waybill-related data included in each incremental log, if the waybill-related data satisfies the preset final node strategy corresponding to the waybill, the waybill-related data is written into the preset wide table, and a final node flag is configured for the waybill in the wide table. Exemplarily, the final node flag may be update_flag.

[0066] In this embodiment, the process of writing the waybill-related data into the preset wide table specifically includes the following steps:

[0067] Based on the key fields in the preset wide table, the data items corresponding to each key field are extracted from the waybill-related data of the waybill, and the data items corresponding to each key field are written into the corresponding position of the waybill in the wide table; the multiple key fields include at least a waybill number field, a waybill type field, a packing time field and a receivable amount field.

[0068] In this embodiment, the preset wide table includes multiple key fields, and the multiple key fields include at least a waybill number field, a waybill type field, a packing time field, and a receivable amount field.

[0069] Optionally, the multiple key fields may also include a full tracking message field, a fulfillment time field, a waybill status field, a current responsibility unit field, a receivable or not field, a receivable reason field and a final node mark field.

[0070] In this embodiment, based on the key fields in the preset wide table, the data items corresponding to each key field are extracted from the waybill-related data, and the data items corresponding to each key field are written to the corresponding position of the waybill in the wide table, that is, the data items corresponding to each key field are written to the corresponding position of the waybill number field of the waybill in the wide table.

[0071] It should be noted that the incremental log obtained each time includes only data items corresponding to some key fields in the waybill-related data of the waybill, that is, the data items corresponding to each key field are extracted from the waybill-related data, and some of the extracted data items may be empty.

[0072] In this embodiment, the data items corresponding to each key field are extracted from the waybill-related data, that is, the waybill number, waybill type, packing time, fulfillment time, waybill status, current responsible unit, receivable amount, full tracking information, etc. of the waybill are extracted from the waybill-related data.

[0073] It should be noted that the preset wide table may be a wide table of ElasticSearch.

[0074] It should be noted that if the warehouse management system completes the packing of the goods, the incremental log of the received warehouse management system includes the packing time of a certain waybill, and the data item corresponding to the packing time field written to the wide table is not empty, that is, the packing time of the waybill in the wide table is not empty. At this time, the waybill is in a suspended receivables status; the receivable amount of the waybill in the suspended receivables status has not been collected.

[0075] In this embodiment, the full tracking information records the waybill number and the operation time and operation unit of the package in each link of the distribution, and the full tracking information of the consumer package. First, by querying the final node policy configuration, determine whether the current state is the final node. If it is not the final node, query the basic data information through the operation department, and update the region and unit name of the operation department as the current responsible area and responsible unit to the wide table of ElasticSearch. As the package delivery link continues to advance, the responsible unit is updated in real time until the package status in the full tracking information is changed to delivered or other final nodes. Once the final node message is identified, it is marked with update_flag.

[0076] S104. Directly write the waybill-related data of the waybill into a preset wide table.

[0077] In this embodiment, for the waybill-related data of each waybill included in each incremental log, if the waybill-related data does not satisfy the preset final-level node strategy corresponding to the waybill, the waybill-related data of the waybill is directly written into the preset wide table.

[0078] In this embodiment, for the process of writing the waybill-related data of the waybill into the preset wide table, please refer to the specific process of writing the waybill-related data of the waybill into the preset wide table mentioned in the above step S103, which will not be repeated here.

[0079] S105. For each waybill with the final node mark in the wide table, determine whether the waybill-related data of the waybill satisfies the preset lower receivables strategy corresponding to the waybill. If so, execute S106; if not, execute S107.

[0080] In this embodiment, receivables placement strategies for different waybill types are preset. It should be noted that the receivables placement strategies corresponding to different waybill types may be the same or different.

[0081] It should be noted that the receivables policy includes at least one of the preset multiple conditions; illustratively, the multiple conditions include successful delivery of the waybill, completion of the order, reverse order exchange, completion of reconciliation, completion of loss reporting and completion of return to warehouse.

[0082] For example, the receivables policy corresponding to waybill type A is waybill delivery, and the receivables policy corresponding to waybill type B needs to satisfy the conditions of waybill delivery, order completion, and reconciliation completion at the same time.

[0083] It should be noted that the various conditions included in the accounts receivable strategy often rely on different messages from different business systems.

[0084] In this embodiment, for each waybill configured with a final node mark in the wide table, it is determined whether the waybill-related data of the waybill satisfies the preset lower receivable policy corresponding to the waybill. Specifically, it is determined whether each data item of the waybill satisfies all conditions in the preset lower receivable policy corresponding to the waybill type of the waybill. If each data item of the waybill satisfies all conditions in the preset lower receivable policy corresponding to the waybill type of the waybill, it is determined that the waybill-related data of the waybill satisfies the preset lower receivable policy corresponding to the waybill. If each data item of the waybill does not satisfy all conditions in the preset lower receivable policy corresponding to the waybill type of the waybill, it is determined that the waybill-related data of the waybill does not satisfy the preset lower receivable policy corresponding to the waybill.

[0085] It should be noted that step S105 may be executed through a scheduled task, that is, step S105 may be executed periodically.

[0086] S106. Configure a receivables mark for the waybill in the wide table and delete the last node mark of the waybill; the receivables mark is used to mark that the receivable amount of the waybill has been collected.

[0087] In this embodiment, for each waybill configured with the final node mark in the wide table, if the waybill-related data of the waybill satisfies the preset lower receivable strategy corresponding to the waybill, it means that the receivable amount of the waybill has been paid, and the lower receivable mark is configured for the waybill in the wide table, and the final node mark of the waybill is deleted to avoid repeated scanning of the waybill. Among them, the lower receivable mark is used to mark that the receivable amount of the waybill has been paid.

[0088] It should be noted that the process of configuring the receivable mark for the waybill in the wide table specifically includes: setting the data item corresponding to the receivable field in each data item of the waybill in the wide table to yes.

[0089] In this embodiment, after configuring the receivable mark for the waybill in the wide table, the following steps may also be included:

[0090] Generate the reason for the waybill to be receivable and write the reason for the waybill to the corresponding position in the wide table.

[0091] In this embodiment, after configuring the receivable mark for the waybill in the wide table, the receivable reason for the waybill can also be generated. Specifically, the receivable reason is generated based on a preset receivable reason generation strategy; wherein the receivable reason is used to explain the reason for setting the receivable mark.

[0092] In this embodiment, after generating the reason for the next receivable, the reason for the next receivable can also be written into the corresponding position of the waybill in the wide table, that is, the reason for the next receivable is written into the corresponding position of the waybill and the reason for the next receivable field in the wide table.

[0093] S107: Delete the last node mark of the waybill.

[0094] In this embodiment, for each waybill configured with a final node mark in the wide table, if the waybill-related data of the waybill does not satisfy the preset receivables strategy corresponding to the waybill, it means that the receivable amount of the waybill has not been collected, and the final node mark of the waybill is deleted, that is, the final node mark of the waybill in the wide table is deleted.

[0095] The present application provides a data monitoring method, which obtains the incremental logs of each business system in the current acquisition cycle, and realizes the acquisition of the waybill-related data of each waybill whose status changes in each business system in the current acquisition cycle, breaks through the data barriers of each business system, and decouples the monitoring system and each business system, and writes the waybill-related data into a wide table, and configures the last-level node mark for the waybill that meets the last-level node strategy, so that only the waybill configured with the last-level node mark in the wide table is monitored, thereby realizing the monitoring of the collection status of the receivable amount of the waybill and reducing the performance impact on the monitoring system.

[0096] See also Figure 3 In the data monitoring method provided in the above embodiment of the present application, after writing the waybill-related data of the waybill into the preset wide table mentioned in steps S103 and S104, the following steps may also be included:

[0097] S301. According to a preset summary cycle, based on the outlets to which each target waybill in the wide table belongs, the receivable amounts of each target waybill are summarized to obtain a summary amount for each outlet.

[0098] In this embodiment, according to the preset summary cycle, the receivable amount of each target waybill in the wide table is summarized according to the outlet to which each target waybill belongs, that is, the receivable amount of the target waybill belonging to the same outlet is summed up to obtain the summary amount of each outlet. Among them, the target waybill is a waybill whose data item corresponding to the packaging field is not empty and the receivable flag is not configured.

[0099] The summary period can be modified according to the needs. Preferably, the summary period can be 3 am every day.

[0100] Optionally, the total amount of each outlet can be stored in a database, so that a real-time page can be generated based on the total amount in the database, and the user can query the total amount of each outlet through the real-time page. Exemplarily, the database can be a Mysql database. Optionally, a statistical report can be generated based on the total amount in the database, and the statistical report can be displayed in the real-time page.

[0101] S302: for each outlet, determine whether the total amount of the outlet is greater than a preset threshold, if so, execute S303, if not, directly end.

[0102] In this embodiment, for each outlet, the total amount of the outlet is compared with a preset threshold to determine whether the total amount of the outlet is greater than the preset threshold.

[0103] In this embodiment, for each outlet, if the total amount of the outlet is greater than the preset threshold, step S303 is executed; if the total amount of the outlet is not greater than the preset threshold, the operation process for the outlet is terminated.

[0104] S303. Based on the aggregate amount of the outlets, a warning message is sent to the person in charge of the outlets through a preset communication method.

[0105] In this embodiment, for each outlet, if the total amount of the outlet is greater than a preset threshold, a warning message is sent to the person in charge of the outlet based on the total amount of the outlet through a preset communication method. The preset communication method includes but is not limited to SMS or email.

[0106] In the data monitoring method provided in the embodiment of the present application, the receivable amount of the target waybill is summarized according to the summary period, and when the summarized amount is greater than a preset threshold, an early warning message is sent to the person in charge of the branch to remind the person in charge of the branch to collect the payment for the target waybill as soon as possible.

[0107] See also Figure 4In the data monitoring method provided in the above embodiment of the present application, after writing the waybill-related data of the waybill into the preset wide table mentioned in steps S103 and S104, the following steps may also be included:

[0108] S401. For each target waybill in the wide table, determine whether the packing time of the target waybill exceeds a preset time threshold. If so, execute S402; if not, return to execute S401.

[0109] In this embodiment, for each target waybill in the wide table, it is determined whether the packing time of the target waybill exceeds the preset time threshold, wherein the target waybill is a waybill whose data item corresponding to the packing time field is not empty and is a waybill with a receivable mark under the configuration, and the packing time of the target waybill is the data item corresponding to the packing time field in each data item of the target waybill.

[0110] Preferably, the preset time threshold is 30 days.

[0111] Optionally, step S401 may be executed through a scheduled task, that is, step S401 is executed regularly.

[0112] S402: Call the preset accountability process, perform accountability processing on the target waybill, and obtain an accountability result.

[0113] In this embodiment, for each target waybill in the wide table, if the packaging time of the target waybill exceeds the preset time threshold, the preset accountability process is called to perform accountability processing on the target waybill to obtain an accountability result, wherein the accountability result includes organizational accountability or individual accountability.

[0114] S403. Determine whether the organization is responsible for the judgment result. If so, execute S404; if not, execute S405.

[0115] In this embodiment, for each target waybill in the wide table, it is determined whether the accountability result indicates that the organization is responsible.

[0116] S404: Send a receivable message queue message to the responsible organization, and configure a receivable flag for the target waybill.

[0117] In this embodiment, for each target waybill in the wide table, if the judgment result is that the organization bears the responsibility, a receivable message queue message is sent to the responsible organization, and a receivable flag is configured for the target waybill, that is, a receivable flag is configured for the target waybill in the wide table.

[0118] Optionally, after configuring the receivable flag for the target waybill, a receivable reason for the target waybill may be generated and entered into the wide table.

[0119] S405: Generate a loss return order, and push the loss return order to the user who bears the responsibility. After the user completes the payment of the loss return order, configure a receivable mark for the target.

[0120] In this embodiment, for each target waybill in the wide table, if the judgment result is that the individual is responsible, a loss return order is generated, and the loss return order is pushed to the user who is responsible, that is, pushed to the client of the user who is responsible, so that the user pays for the loss return order. Optionally, a payment QR code can also be generated, and the loss return order and the payment QR code are sent together to the user who is responsible, so that the user who is responsible can scan and pay by the payment QR code.

[0121] In this embodiment, after the user completes the payment of the loss return order, a receivable indicator is configured for the target waybill.

[0122] Optionally, after configuring the receivable flag for the target waybill, a receivable reason for the target waybill may be generated and entered into the wide table.

[0123] In the data monitoring method provided in the embodiment of the present application, by monitoring the waybills that are not configured with the receivable mark, the waybills that exceed the preset time threshold are mined, and the waybills are judged and returned for loss, so as to achieve loss recovery.

[0124] See also Figure 5 The specific implementation process of the data monitoring method provided in the above embodiment of the present application is illustrated as follows:

[0125] The monitoring system mentioned above also includes data pipeline components, accountability systems, loss return systems, and automatic account suspension and delisting monitoring systems.

[0126] The data pipeline component sends data pull requests to each upstream business system according to the preset acquisition cycle. Each business system sends the Binlog (binary log) in the current acquisition cycle to the data pipeline component. The data pipeline component Binlog parses and converts the format, and sends the converted Binlog to the message middleware Kafka. Among them, the upstream business systems include but are not limited to the order system, waybill system, warehouse management system WMS, distribution system and other systems.

[0127] The automatic account suspension and debit monitoring system subscribes to Kafka messages for consumption, that is, obtains the format-converted Binlog in the message middleware Kafka, and then processes the format-converted Binlog into the wide table of ElasticSearch according to the waybill dimension through the Flink real-time processing program. Specifically, the format-converted Binlog is parsed to obtain the waybill-related data of each waybill. If the waybill-related data of the waybill meets the preset end-level node strategy corresponding to the waybill, the end-level node mark update_flag is configured for the waybill in the wide table, and based on each key field in the preset wide table, the data item corresponding to each key field is extracted from the waybill-related data of the waybill, and the data item corresponding to each key field is written to the corresponding position of the waybill in the wide table; if the waybill-related data of the waybill meets the preset end-level node strategy corresponding to the waybill, based on each key field in the preset wide table, the data item corresponding to each key field is extracted from the waybill-related data of the waybill, and the data item corresponding to each key field is written to the corresponding position of the waybill in the wide table.

[0128] A first distributed scheduled task, a second distributed scheduled task and a third distributed scheduled task are preset, and the first scheduled task, the second scheduled task and the third scheduled task are multi-threaded.

[0129] The first distributed scheduled task is used to scan the wide table and monitor whether there is a waybill in the receivable monitoring report that meets its corresponding receivable strategy. If so, configure the receivable mark for the waybill that meets its corresponding receivable strategy and delete the last node mark of the waybill. The receivable monitoring report is generated based on each waybill with the last node mark configured in the wide table.

[0130] The second distributed timing task is used to scan each waybill included in the wide table to determine the waybill whose data item corresponding to the packaging time field is not empty and has not been configured with the receivable identifier, and to summarize the receivable amounts for the determined waybills by outlets to obtain the summarized amount for each outlet. When the summarized amount of the outlet is greater than the preset threshold, an early warning message will be sent to the person in charge of the outlet based on the summarized amount of the outlet through a preset communication method. That is, based on the summarized amount of the outlet, an early warning notification task is generated and pushed to the person in charge of the outlet.

[0131] The third distributed scheduled task is used to scan the wide table, obtain the waybills whose packaging time exceeds the preset time threshold and are not configured with the receivables mark, and generate overdue tasks based on the obtained waybills. Based on the overdue tasks, they are sent to the accountability system through the message queue MQ. The accountability system monitors MQ and starts the accountability process after obtaining the overdue tasks. If the accountability process determines that it is not an individual's responsibility, it shall be borne by the organization, and the receivables MQ message shall be sent to the automatic account suspension and account release monitoring system. The automatic account suspension and account release monitoring system consumes the receivables MQ message through the JMQ consumer program, thereby configuring the receivables mark for the waybill corresponding to the receivables MQ message. If the liability assessment process determines that it is personal responsibility, the liability assessment system will send an automatic loss return MQ message to the loss return system. After listening to the automatic loss return MQ message, the loss return system will start the loss return process, generate a loss return order and a payment QR code, and push the loss return order and the payment QR code to the responsible person's mobile terminal. After the responsible person completes the payment, the loss return system will send a receivables MQ message to the automatic account suspension and account release monitoring system. The automatic account suspension and account release monitoring system will consume the receivables MQ message through the JMQ consumer program, thereby configuring the receivables flag for the waybill corresponding to the receivables MQ message.

[0132] The data monitoring method provided in the embodiment of the present application converts the business system Binlog to Kafka real-time messages, uses Flink data processing technology to achieve real-time waybill monitoring, and decouples the monitoring system from the business system, and automatically places and places receivables based on the waybill tracking messages throughout the process, thereby achieving automatic monitoring. Different strategies are configured according to different order types, and receivables are placed by different strategies by using the method of asynchronous scanning based on the first-reaching standard, and by monitoring the receivables data that have not been placed, excavating overdue abnormal documents, and opening up the judgment and loss return through MQ messages to achieve an automatic loss recovery closed loop.

[0133] It should be noted that although the operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order. In certain circumstances, multitasking and parallel processing may be advantageous.

[0134] It should be understood that the various steps described in the method embodiments disclosed in the present application can be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0135] and Figure 1 Corresponding to the method described above, the embodiment of the present application also provides a data monitoring device for Figure 1 The specific implementation of the method is shown in the following diagram: Figure 6 As shown, specifically including:

[0136] The acquisition unit 601 is used to acquire the incremental log of each business system in the current acquisition cycle according to a preset acquisition cycle; the incremental log includes the waybill-related data of each waybill whose status has changed in the current acquisition cycle;

[0137] The writing unit 602 is used for writing the waybill-related data of each waybill included in each incremental log into a preset wide table if the waybill-related data of the waybill satisfies the preset last-level node strategy corresponding to the waybill, and configuring a last-level node mark for the waybill in the wide table; otherwise, directly writing the waybill-related data of the waybill into the preset wide table;

[0138] Configuration unit 603 is used to configure a lower receivable mark for each waybill configured with the final node mark in the wide table, and delete the final node mark of the waybill if the waybill-related data of the waybill meets the preset lower receivable strategy corresponding to the waybill; the lower receivable mark is used to mark that the receivable amount of the waybill has been collected.

[0139] The data monitoring device provided in the embodiment of the present application obtains the incremental logs of each business system in the current acquisition cycle to obtain the waybill-related data of each waybill whose status has changed in each business system within the current acquisition cycle, thereby breaking through the data barriers of each business system, decoupling the monitoring system and each business system, and writing the waybill-related data into a wide table. By configuring the last-level node mark for the waybill that meets the last-level node strategy, only the waybill configured with the last-level node mark in the wide table is monitored, thereby realizing the monitoring of the collection status of the receivable amount of the waybill and reducing the performance impact on the monitoring system.

[0140] In one embodiment of the present application, based on the above solution, it can also be configured as follows:

[0141] A deleting unit is used to delete the last-level node mark of the waybill if the waybill-related data of the waybill does not satisfy the preset lower receivable strategy corresponding to the waybill.

[0142] In one embodiment of the present application, based on the above solution, it can also be configured as follows:

[0143] A generating unit is used to generate a reason for the waybill to be collected, and write the reason for the waybill to a corresponding position in the wide table.

[0144] In one embodiment of the present application, based on the above solution, the acquisition unit 601 is specifically configured to:

[0145] According to the preset acquisition cycle, the incremental logs of each business system in the current acquisition cycle are obtained through the data pipeline component.

[0146] In one embodiment of the present application, based on the above solution, when the writing unit 602 writes the waybill-related data of the waybill into the preset wide table, it is specifically used to:

[0147] Based on the key fields in the preset wide table, the data items corresponding to each key field are extracted from the waybill-related data of the waybill, and the data items corresponding to each key field are written into the corresponding position of the waybill in the wide table; the multiple key fields include at least a waybill number field, a waybill type field, a packing time field and a receivable amount field.

[0148] In one embodiment of the present application, based on the above solution, it can also be configured as follows:

[0149] A summarizing unit, used to summarize the receivable amount of each target waybill according to the outlets to which each target waybill in the wide table belongs according to a preset summarizing period, so as to obtain the summary amount of each outlet; the target waybill is a waybill whose data item corresponding to the packaging time field is not empty and no receivable mark is configured;

[0150] The sending unit is used to send a warning message to the person in charge of each outlet through a preset communication method based on the aggregate amount of the outlet if the aggregate amount of the outlet is greater than a preset threshold.

[0151] In one embodiment of the present application, based on the above solution, it can also be configured as follows:

[0152] The loss return unit is used for each target waybill in the wide table. If the packing time of the target waybill exceeds the preset time threshold, the preset accountability process is called to perform accountability processing on the target waybill to obtain the accountability result. If the accountability result is that the organization is responsible, a receivable message queue message is sent to the responsible organization, and a receivable flag is configured for the target waybill. If the accountability result is that the user is responsible, a loss return order is generated and pushed to the responsible user. After the user completes the payment of the loss return order, a receivable flag is configured for the target waybill. The target waybill is a waybill whose data item corresponding to the packing time field is not empty and for which the receivable flag is not configured.

[0153] An embodiment of the present application further provides a storage medium, wherein the storage medium stores an instruction set, wherein when the instruction set is executed, the data monitoring method disclosed in any of the above embodiments is executed.

[0154] The present application also provides an electronic device, whose structural diagram is as follows: Figure 7As shown, it specifically includes a memory 701 for storing at least one set of instruction sets; a processor 702 for executing the instruction set stored in the memory, and implementing the data monitoring method disclosed in any of the above embodiments by executing the instruction set.

[0155] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0156] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope disclosed in the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0157] The above description is only a preferred embodiment disclosed in this application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the disclosure of this application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the disclosure of this application (but not limited to) to form a technical solution.

Claims

1. A data monitoring method, characterized in that: Applied to a monitoring system, the method comprises: According to a preset acquisition cycle, the incremental log of each business system in the current acquisition cycle is acquired; the incremental log includes the waybill-related data of each waybill whose status has changed in the current acquisition cycle; For the waybill-related data of each waybill included in each incremental log, if the waybill-related data of the waybill satisfies the preset final node strategy corresponding to the waybill, the waybill-related data of the waybill is written into the preset wide table, and a final node mark is configured for the waybill in the wide table; otherwise, the waybill-related data of the waybill is directly written into the preset wide table; For each waybill configured with the final node mark in the wide table, if the waybill-related data of the waybill satisfies the preset lower receivable strategy corresponding to the waybill, a lower receivable mark is configured for the waybill in the wide table, and the final node mark of the waybill is deleted; the lower receivable mark is used to mark that the receivable amount of the waybill has been collected.

2. The method according to claim 1, characterized in that: Also includes: If the waybill-related data of the waybill does not satisfy the preset lower receivable strategy corresponding to the waybill, the last-level node mark of the waybill is deleted.

3. The method according to claim 1, characterized in that After configuring the receivable mark for the waybill in the wide table, the method further includes: Generate a reason for the waybill to be collected, and write the reason for the waybill to the corresponding position in the wide table.

4. The method according to claim 1, characterized in that: The step of obtaining incremental logs of each business system in the current acquisition cycle according to a preset acquisition cycle includes: According to the preset acquisition cycle, the incremental logs of each business system in the current acquisition cycle are obtained through the data pipeline component.

5. The method according to claim 1, characterized in that The step of writing the waybill-related data of the waybill into a preset wide table includes: Based on the key fields in the preset wide table, the data items corresponding to each key field are extracted from the waybill-related data of the waybill, and the data items corresponding to each key field are written into the corresponding position of the waybill in the wide table; the multiple key fields include at least a waybill number field, a waybill type field, a packing time field and a receivable amount field.

6. The method according to claim 5, characterized in that After the waybill-related data of the waybill is written into the preset wide table, the method further includes: According to the preset summary cycle, according to the outlets to which each target waybill in the wide table belongs, the receivable amount of each target waybill is summarized to obtain the summary amount of each outlet; the target waybill is a waybill whose data item corresponding to the packaging time field is not empty and the receivable flag is not configured; For each outlet, if the aggregate amount of the outlet is greater than a preset threshold, an early warning message is sent to the person in charge of the outlet through a preset communication method based on the aggregate amount of the outlet.

7. The method according to claim 5, characterized in that After the waybill-related data of the waybill is written into the preset wide table, the method further includes: For each target waybill in the wide table, if the packing time of the target waybill exceeds the preset time threshold, the preset accountability process is called to perform accountability processing on the target waybill to obtain an accountability result. If the accountability result is that the organization is responsible, a receivable message queue message is sent to the responsible organization, and a receivable flag is configured for the target waybill. If the accountability result is that the user is responsible, a loss return order is generated and pushed to the responsible user. After the user completes the payment of the loss return order, a receivable flag is configured for the target waybill. The target waybill is a waybill whose data item corresponding to the packing time field is not empty and for which a receivable flag is not configured.

8. A data monitoring device, characterized in that: Applied to a monitoring system, the method comprises: An acquisition unit, used to acquire, according to a preset acquisition cycle, an incremental log of each business system in a current acquisition cycle; the incremental log includes waybill-related data of each waybill whose status has changed in the current acquisition cycle; A writing unit, for writing the waybill-related data of each waybill included in each incremental log into a preset wide table if the waybill-related data of the waybill satisfies the preset final node strategy corresponding to the waybill, and configuring a final node mark for the waybill in the wide table; otherwise, directly writing the waybill-related data of the waybill into the preset wide table; A configuration unit is used to configure a lower receivable mark for each waybill configured with the final node mark in the wide table, and delete the final node mark of the waybill if the waybill-related data of the waybill meets the preset lower receivable strategy corresponding to the waybill; the lower receivable mark is used to mark that the receivable amount of the waybill has been collected.

9. A storage medium, characterized in that: The storage medium stores an instruction set, wherein the instruction set, when executed by a processor, implements the data monitoring method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory for storing at least one set of instructions; A processor is used to execute an instruction set stored in the memory, and implement the data monitoring method as described in any one of claims 1 to 7 by executing the instruction set.

Images