Data processing method, device, apparatus and medium

Abstract

The embodiment of the application provides a kind of data processing method, device and electronic equipment, the method comprises: receiving current time from data source interface to be sent data;In the case where the first data in current time to be sent data is determined to send failure, first data is stored to database, first data is at least part of current to be sent data;Database is also used to receive data from data source interface after storing first data;In the case where the data volume of the unsent data stored in database is greater than or equal to preset data volume, the data of preset data volume is read from the unsent data stored in database, and the data of preset data volume is sent to target equipment;In the case where the data volume of the unsent data stored in database is less than preset data volume, the unsent data stored in database is read, and the unsent data is sent to target equipment.

Description

Technical Field

[0001] This invention relates to a data processing method, and more particularly to a data processing method, apparatus, electronic device, and computer storage medium. Background Technology

[0002] In recent years, with the increase in business application scenarios, accurate and efficient data has become a bottleneck in the construction of information technology projects, placing higher demands on data transmission at the edge. In weak network environments, timely resumption of data transmission after interruption is a critical issue. Currently, the logic of breakpoint resumption technology is relatively simple: after a network failure, data is first stored locally, and then uploaded again after the network is restored. However, when the amount of data to be transmitted is large, if the traditional breakpoint resumption logic is followed, a large amount of data needs to be retransmitted when the network recovers, causing network congestion and affecting the operation of other applications, resulting in a high network load. Summary of the Invention

[0003] The embodiments of the present invention mainly provide a data processing method, apparatus, electronic device, and computer storage medium.

[0004] This invention provides a data processing method applied in an electronic device, the method comprising:

[0005] Receive the data to be sent at the current moment from the data source interface;

[0006] If it is determined that the first data in the data to be sent at the current moment has failed to be sent, the first data is stored in the database, wherein the first data is at least a portion of the data in the current data to be sent; the database is also used to receive data from the data source interface after storing the first data;

[0007] If the amount of unsent data stored in the database is greater than or equal to a preset amount of data, the preset amount of data is read from the unsent data stored in the database and the read preset amount of data is sent to the target device.

[0008] If the amount of unsent data stored in the database is less than a preset amount, the unsent data stored in the database is read and sent to the target device.

[0009] In the above scheme, the method further includes: after sending the read preset amount of data to the target device, determining the sending timestamp of the preset amount of data sent this time, adding the sending timestamp of the preset amount of data sent this time to the corresponding data in the database; and determining the unsent data in the database according to the data with added sending timestamps in the database.

[0010] As can be seen, by recording the sending timestamp of the preset amount of data sent this time, it is possible to effectively identify unsent data in the database based on the data with added sending timestamps, thereby improving the efficiency of data interruption resumption.

[0011] In the above scheme, when the amount of unsent data stored in the database is greater than or equal to a preset amount of data, reading the preset amount of data from the unsent data stored in the database includes: determining the last sent data in the database, and reading the preset amount of data adjacent to the last sent data from the database; the last sent data represents the data in the database with the latest sent timestamp; when the amount of unsent data stored in the database is less than the preset amount of data, reading the unsent data stored in the database includes: determining the last sent data in the database, and reading the unsent data adjacent to the last sent data from the database; the last sent data represents the data in the database with the latest sent timestamp.

[0012] As can be seen, based on the last data sent in the database, the data currently being read from the database can be determined, as well as the unsent data in the database, which can effectively improve the efficiency of resuming data transmission after a breakpoint.

[0013] In the above scheme, the method further includes: if it is determined that the data to be sent at the current moment has been successfully sent according to a preset first data sending method, determining the connection state between the electronic device and the target device as a first connection state, and the data sending method corresponding to the first connection state as the first data sending method; if it is determined that the first data in the data to be sent at the current moment has failed to be sent, determining the connection state between the electronic device and the target device as a second connection state, repeating the steps of receiving data from the data source interface and storing the data received from the data source interface into the database; the second connection state is different from the first connection state; after sending the unsent data to the target device, re-determining the connection state between the electronic device and the target device as the first connection state.

[0014] It can be seen that by determining whether the data to be sent at the current moment has been successfully sent according to the preset first data sending method, the connection status between the electronic device and the target device can be determined. Based on the connection status between the electronic device and the target device, the data sending method can be determined, which can effectively improve the efficiency of data interruption resumption.

[0015] In the above scheme, the method further includes: when the amount of unsent data stored in the database is less than a preset amount of data, determining the state of the database to be locked, the locked state indicating that data from the data source interface is not allowed; after sending the unsent data to the target device, switching the state of the database from locked to unlocked, the unlocked state indicating that data from the data source interface is allowed.

[0016] As can be seen, the state of the database can be determined based on the amount of unsent data stored in the database, thereby enabling the determination of the data sending method and effectively improving the efficiency of data interruption resumption.

[0017] In the above scheme, at least one of the data source interface, data storage interface, and data output interface is a pre-configured interface, wherein the data storage interface represents an interface for configuring the database, and the data output interface represents an interface for configuring the target device.

[0018] As can be seen, the data source interface, data storage interface, and data output interface can be flexibly configured according to actual needs, making data processing methods easy to implement.

[0019] This invention also provides a data processing apparatus for use in electronic devices, the apparatus comprising:

[0020] The receiving module is used to receive the data to be sent at the current time from the data source interface;

[0021] A first processing module is configured to store the first data in a database when it is determined that the transmission of the first data in the data to be sent at the current moment has failed, wherein the first data is at least a portion of the data in the current data to be sent; the database is further configured to receive data from the data source interface after storing the first data;

[0022] The second processing module is configured to, when the amount of unsent data stored in the database is greater than or equal to a preset amount of data, read a preset amount of data from the unsent data stored in the database and send the read preset amount of data to the target device; and when the amount of unsent data stored in the database is less than the preset amount of data, read the unsent data stored in the database and send the unsent data to the target device.

[0023] In one implementation, the device further includes: a third processing module, which is configured to, after sending the read preset data amount to the target device, determine the sending timestamp of the preset data amount sent this time, and add the sending timestamp of the preset data amount sent this time to the corresponding data in the database;

[0024] Based on the data with added sending timestamps in the database, determine the unsent data in the database.

[0025] In one implementation, the third processing module is configured to read a preset amount of data from the unsent data stored in the database when the amount of unsent data stored in the database is greater than or equal to a preset amount of data, including:

[0026] Determine the last transmitted data in the database, and read a preset amount of data adjacent to the last transmitted data from the database; the last transmitted data represents the data in the database with the latest transmission timestamp;

[0027] When the amount of unsent data stored in the database is less than a preset amount, reading the unsent data stored in the database includes:

[0028] Determine the last transmitted data in the database, and read the untransmitted data adjacent to the last transmitted data from the database; the last transmitted data represents the data in the database with the latest transmission timestamp.

[0029] In one implementation, the device further includes: a fourth processing module; the fourth processing module is used to determine, when it is determined that the data to be sent at the current time has been successfully sent according to a preset first data sending method, the connection state between the electronic device and the target device is a first connection state, and the data sending method corresponding to the first connection state is the first data sending method;

[0030] If it is determined that the first data in the data to be sent at the current moment has failed to be sent, the connection state between the electronic device and the target device is determined to be a second connection state, and the steps of receiving data from the data source interface and storing the data received from the data source interface into the database are repeated; the second connection state is different from the first connection state.

[0031] After the unsent data is sent to the target device, the connection status between the electronic device and the target device is redefined as the first connection status.

[0032] In one implementation, the device further includes: a fifth processing module; the fifth processing module is used to determine the state of the database as a locked state when the amount of unsent data stored in the database is less than a preset amount of data, the locked state indicating that data from the data source interface is not allowed to be received;

[0033] After sending the unsent data to the target device, the database state is switched from locked to unlocked, indicating that the database is allowed to receive data from the data source interface.

[0034] In one implementation, at least one of the data source interface, data storage interface, and data output interface is a pre-configured interface, wherein the data storage interface represents an interface for configuring the database, and the data output interface represents an interface for configuring the target device.

[0035] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described data processing methods.

[0036] This invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements any of the above-described data processing methods.

[0037] According to embodiments of the present invention, a data processing method, apparatus, electronic device, and computer storage medium are provided. The method includes: receiving data to be sent at a current time from a data source interface; if it is determined that the transmission of first data in the data to be sent at the current time has failed, storing the first data in a database, wherein the first data is at least a portion of the data to be sent at the current time; the database is further configured to receive data from the data source interface after storing the first data; if the amount of unsent data stored in the database is greater than or equal to a preset amount of data, reading the preset amount of data from the unsent data stored in the database and sending the read preset amount of data to a target device; if the amount of unsent data stored in the database is less than the preset amount of data, reading the unsent data stored in the database and sending the unsent data to the target device.

[0038] As can be seen, in this embodiment of the invention, if the first data in the data to be sent at the current moment fails to be sent, the first data is stored in the database, and the data received after the first data from the data source interface is also stored in the database. By determining the amount of unsent data stored in the database, if the amount of unsent data stored in the database is greater than or equal to a preset amount, the preset amount of data is read and sent to the target device; if the amount of unsent data stored in the database is less than the preset amount, all unsent data is read and sent to the target device. By determining the amount of unsent data stored in the database and reading and sending data according to the size of the unsent data stored in the database, the network bandwidth occupied during data interruption resumption can be reduced, the possibility of network congestion can be reduced, and the network load can be reduced.

[0039] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0040] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a data processing architecture provided in an embodiment of the present invention;

[0042] Figure 3 A schematic diagram illustrating the working principle of a data processing architecture provided in an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the data flow in the first data processing method provided in the embodiments of the present invention;

[0044] Figure 5 This is a schematic diagram of the data flow in the second data processing method provided in this embodiment of the invention;

[0045] Figure 6 A flowchart illustrating a specific implementation of a data processing method provided in an embodiment of the present invention;

[0046] Figure 7 A schematic diagram of a data processing apparatus provided in an embodiment of the present invention;

[0047] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the embodiments provided herein are merely illustrative and not intended to limit the scope of the invention. Furthermore, the embodiments provided below are partial embodiments for implementing the present invention, not all embodiments for implementing the present invention. Unless otherwise specified, the technical solutions described in the embodiments of the present invention can be implemented in any combination.

[0049] It should be noted that, in the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly stated, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus, such as portions of circuitry, processors, programs, or software, etc.) in the method or apparatus that includes that element.

[0050] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0051] For example, a data processing method provided in an embodiment of the present invention includes a series of steps, but the data processing method provided in an embodiment of the present invention is not limited to the steps described herein. Similarly, a data processing apparatus provided in an embodiment of the present invention includes a series of modules, but the data processing apparatus provided in an embodiment of the present invention is not limited to the modules explicitly described, but may also include modules required for obtaining relevant information or processing based on information.

[0052] In this embodiment of the invention, the entity executing the data processing method can be an electronic device. For example, the electronic device can be a terminal, a server, or other processing device. Here, the terminal can be a thin client, a thick client, a handheld or laptop device, a microprocessor-based system, a set-top box, a programmable consumer electronics product, a network personal computer, a minicomputer system, and so on. The server can be a minicomputer system, a mainframe computer system, or a distributed cloud computing environment that includes any of the above systems, and so on.

[0053] Servers and other electronic devices may include program modules that execute computer instructions. Typically, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked via communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0054] This invention provides a data processing method. Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the process may include:

[0055] Step 101: Receive the data to be sent at the current time from the data source interface.

[0056] In this embodiment of the invention, the source interface refers to an interface that can interact with different types of data sources and databases; that is, data can be received through the source interface. Here, the data source can be data conforming to the Message Queuing Telemetry Transport (MQTT) protocol or data conforming to the Hyper Text Transfer Protocol (HTTP) protocol; the database can be the Kafka data storage system.

[0057] In this embodiment of the invention, the data to be sent refers to the data that needs to be sent to the target device, and the data to be sent at the current moment refers to the data to be sent received by the data source interface at the current moment and before the current moment. That is, the data to be sent at the current moment changes with time.

[0058] Step 102: If it is determined that the first data in the data to be sent at the current moment has failed to be sent, the first data is stored in the database. The first data is at least a portion of the data to be sent at the current time. The database is also used to receive data from the data source interface after storing the first data.

[0059] In this embodiment of the invention, the first data represents the data that failed to be sent in the data to be sent at the current moment. Since packet loss and delay occur during data transmission when the network quality is poor, resulting in data transmission failure, the first data in the data to be sent at the current moment, i.e. the data that failed to be sent, is stored in the database. When the network quality is good, the first data is then sent to the target device.

[0060] In this embodiment of the invention, the database can be a time-series database, also known as a time series database. A time-series database is mainly used to process data with time tags (data that changes in chronological order, i.e., time-series data), and data with time tags is also called time series data.

[0061] In this embodiment of the invention, after determining that the first data in the data to be sent at the current moment has failed to be transmitted, the first data can be stored in the database based on the data storage interface. Furthermore, since the failure to transmit the first data in the data to be sent at the current moment indicates poor network quality, preventing normal data transmission, it is necessary to also store the data received after the first data from the data source interface in the database.

[0062] In this embodiment of the invention, when the network quality is good, the data to be sent from the previous moment can be sent to the target device through the data output interface (Sink Interface).

[0063] Step 103: If the amount of unsent data stored in the database is greater than or equal to the preset amount of data, read the preset amount of data from the unsent data stored in the database and send the read preset amount of data to the target device; if the amount of unsent data stored in the database is less than the preset amount of data, read the unsent data stored in the database and send the unsent data to the target device.

[0064] In this embodiment of the invention, since a large amount of data needs to be retransmitted when the network is restored, which will cause network congestion, in order to reduce the network bandwidth occupied when data is resumed from interruption and not affect the operation of other services in the network, it is necessary to control the amount of data sent each time the unsent data is sent.

[0065] In this embodiment of the invention, when the amount of unsent data stored in the database is greater than or equal to a preset amount of data, it indicates that the amount of unsent data stored in the current database is large. If all the unsent data stored in the current database is sent to the target device at the same time, it may cause network congestion. Therefore, the preset amount of data can be read from the unsent data stored in the database through the Resume Break Point Interface (RBP Interface), and the read preset amount of data can be sent to the target device.

[0066] In this embodiment of the invention, the preset data volume can be pre-set based on existing experience. Sending the preset data volume consumes less network bandwidth and will not affect the operation of other services on the network. For example, the preset data volume can be 5000 or 8000 data entries, and the invention does not limit this to any particular value.

[0067] In this embodiment of the invention, when the amount of unsent data stored in the database is less than the preset amount of data, it indicates that the amount of unsent data currently stored in the database is small. All unsent data currently stored in the database can be sent to the target device at the same time without causing network congestion. Therefore, all unsent data can be read from the unsent data stored in the database through the breakpoint resume interface and sent to the target device.

[0068] As can be seen, in this embodiment of the invention, if the first data in the data to be sent at the current moment fails to be sent, the first data is stored in the database, and the data received after the first data from the data source interface is also stored in the database. By determining the amount of unsent data stored in the database, if the amount of unsent data stored in the database is greater than or equal to a preset amount, the preset amount of data is read and sent to the target device; if the amount of unsent data stored in the database is less than the preset amount, all unsent data is read and sent to the target device. By determining the amount of unsent data stored in the database and reading and sending data according to the size of the unsent data stored in the database, the network bandwidth occupied during data interruption resumption can be reduced, the possibility of network congestion can be reduced, and the network load can be reduced.

[0069] In some embodiments, the method further includes:

[0070] After sending the preset amount of data to the target device, the sending timestamp of the preset amount of data sent this time is determined, and the sending timestamp of the preset amount of data sent this time is added to the corresponding data in the database;

[0071] Based on the data with added sending timestamps in the database, determine the unsent data in the database.

[0072] In some embodiments, the data transmission timestamp represents the moment the data is sent to the target device. After sending a preset amount of data to the target device, the transmission timestamp of the preset amount of data sent can be determined and added to the corresponding data in the database.

[0073] For example, the sending timestamp of a preset amount of data in a certain transmission could be: January 1, 2023, 12:30.

[0074] In some embodiments, since the database is a time-series database and the data in the database is time-series data with time tags, the data in the database with added sending timestamps can be determined based on the sending timestamps of the preset amount of data sent this time. Thus, the data that has not been sent in the database can be determined, that is, the data that has not been sent in the database is other data in the database besides the data of the preset amount of data sent this time.

[0075] As can be seen, by recording the sending timestamp of the preset amount of data sent this time, it is possible to effectively identify unsent data in the database based on the data with added sending timestamps, thereby improving the efficiency of data interruption resumption.

[0076] In some implementations, when the amount of unsent data stored in the database is greater than or equal to a preset amount, reading the preset amount of data from the unsent data stored in the database includes:

[0077] Determine the last transmitted data in the database, and read a preset amount of data adjacent to the last transmitted data from the database; the last transmitted data represents the data in the database with the latest transmission timestamp;

[0078] When the amount of unsent data stored in the database is less than a preset amount, reading the unsent data stored in the database includes:

[0079] Determine the last transmitted data in the database, and read the untransmitted data adjacent to the last transmitted data from the database; the last transmitted data represents the data in the database with the latest transmission timestamp.

[0080] In some embodiments, when the amount of unsent data stored in the database is greater than or equal to a preset amount of data, the preset amount of data is read from the unsent data stored in the database and sent each time. The preset amount of data currently read from the unsent data stored in the database can be based on the last sent data in the database, where the last sent data refers to the data in the database with the latest sent timestamp, and the preset amount of data adjacent to the last sent data is read from the database.

[0081] In some embodiments, since the database is a time-series database and the data in the database is time-series data with time tags, the timestamp of the last sent data being written to the database can be determined based on the last sent data in the database, and the adjacent data can be determined, thereby determining a preset amount of data adjacent to the last sent data.

[0082] In some embodiments, if the amount of unsent data stored in the database is less than a preset amount of data, all unsent data adjacent to the last sent data can be read from the database and sent to the target device.

[0083] As can be seen, based on the last data sent in the database, the data currently being read from the database can be determined, as well as the unsent data in the database, which can effectively improve the efficiency of resuming data transmission after a breakpoint.

[0084] In some embodiments, the method further includes:

[0085] If it is determined that the data to be sent at the current moment has been successfully sent according to the preset first data sending method, the connection state between the electronic device and the target device is determined to be the first connection state, and the data sending method corresponding to the first connection state is the first data sending method.

[0086] If it is determined that the first data in the data to be sent at the current moment has failed to be sent, the connection state between the electronic device and the target device is determined to be a second connection state, and the steps of receiving data from the data source interface and storing the data received from the data source interface into the database are repeated; the second connection state is different from the first connection state.

[0087] After the unsent data is sent to the target device, the connection status between the electronic device and the target device is redefined as the first connection status.

[0088] In some embodiments, the first connection state indicates that the network status of the link between the electronic device and the target device is good, that is, the data to be sent at the current moment can be sent to the target device through the data output interface; the second connection state indicates that the network status of the link between the electronic device and the target device is poor, that is, the data to be sent at the current moment cannot be sent to the target device through the data output interface.

[0089] In some embodiments, if it is determined that the data to be sent at the current moment has been successfully sent in accordance with the preset first data sending method, it indicates that the current network quality is good, and the data to be sent at the current moment can be sent to the target device through the data output interface. That is, the connection state between the electronic device and the target device is the first connection state, and the method of sending the data to be sent at the current moment through the data output interface is recorded as the first data sending method.

[0090] In some embodiments, if it is determined that the first data in the data to be sent at the current moment has failed to be sent, it indicates that the current network quality is poor and the data to be sent at the current moment cannot be sent to the target device through the data output interface. That is, the connection state between the electronic device and the target device is the second connection state. The data to be sent at the current moment is stored in the database, and the unsent data stored in the database is sent through the breakpoint resume interface.

[0091] In some embodiments, if it is determined that the first data in the data to be sent at the current moment has failed to be sent, the electronic device receives data from the data source interface and stores the data received from the data source interface into the database.

[0092] In some embodiments, when unsent data is sent to the target device, it indicates that the link for the electronic device to send data to the target device is in a connected state, and the connection state between the electronic device and the target device can be redefined as the first connection state.

[0093] It can be seen that by determining whether the data to be sent at the current moment has been successfully sent according to the preset first data sending method, the connection status between the electronic device and the target device can be determined. Based on the connection status between the electronic device and the target device, the data sending method can be determined, which can effectively improve the efficiency of data interruption resumption.

[0094] In some embodiments, the method further includes:

[0095] If the amount of unsent data stored in the database is less than a preset amount, the database is set to a locked state, which means that data from the data source interface is not allowed to be received.

[0096] After sending the unsent data to the target device, the database state is switched from locked to unlocked, indicating that the database is allowed to receive data from the data source interface.

[0097] In some embodiments, if the amount of unsent data stored in the database is less than a preset amount, it indicates that all the unsent data currently stored in the database is about to be sent to the target device, and the network quality is already good enough to send data to the target device normally. Therefore, the database is set to a locked state, meaning that the database is not allowed to receive data from the data source interface. Then, the subsequent data to be sent can be sent directly to the target device through the data output interface.

[0098] In some embodiments, after sending unsent data to the target device, switching the database state from locked to unlocked allows the database to receive data from the data source interface.

[0099] As can be seen, the state of the database can be determined based on the amount of unsent data stored in the database, thereby enabling the determination of the data sending method and effectively improving the efficiency of data interruption resumption.

[0100] In some implementations, at least one of the data source interface, data storage interface, and data output interface is a pre-configured interface, wherein the data storage interface represents an interface for configuring the database, and the data output interface represents an interface for configuring the target device.

[0101] In some embodiments, the data source interface represents an interface for configuring a database. The data source interface can be configured with different types of data sources and databases. When it is necessary to obtain different types of data sources or obtain data from different databases, only the data source interface needs to be configured.

[0102] In some embodiments, the data output interface represents an interface for configuring target devices. The data output interface can be configured with different target devices. When data needs to be sent to different target devices, only the data source interface needs to be configured.

[0103] In some embodiments, the data storage interface represents an interface for configuring a database. The data storage interface can be configured with different databases. When it is necessary to store data in different databases, only the data storage interface needs to be configured.

[0104] As can be seen, the data source interface, data storage interface, and data output interface can be flexibly configured according to actual needs, making data processing methods easy to implement.

[0105] Figure 2A schematic diagram of a data processing architecture provided in an embodiment of the present invention is shown below. Figure 2 The data processing architecture includes: RESTful Application Programming Interface (RESTful API), Manager Interface, ConfigClient Interface, Rule Interface, Data Source Interface, Data Output Interface, Data Storage Interface, and Resume-on-Breakpoint Interface.

[0106] In some embodiments, a RESTful API is a program access point provided to external devices. A RESTful API includes: a data source API, a data output API, and a rule API. The data source API represents the program access point for the data source interface; the data output API represents the program access point for the data output interface; and the rule API represents the program access point for the rule interface.

[0107] In some embodiments, the management interface is used to manage various interfaces in the data processing architecture, including loading and configuring each interface. The methods for managing the interface are shown in Table 1.

[0108] Table 1. Description of the methods in the management interface

[0109]

[0110] In some embodiments, the configuration storage interface is used to store configuration information related to data processing, such as data source, target device, data preprocessing, etc. The methods for configuring the storage interface are shown in Table 2.

[0111] Table 2. Explanation of methods for configuring storage interfaces

[0112]

[0113] In some embodiments, the rule interface is used to preprocess the data to be sent, such as data extraction, data deletion, and data modification. The methods of the rule interface are shown in Table 3.

[0114] Table 3. Description of the methods in the rule interface

[0115]

[0116] In some embodiments, the data to be sent needs to be parsed before preprocessing to facilitate preprocessing. Taking JSON (JavaScript Object Notation) data as an example, a JSON parsing library needs to be selected first to parse the JSON data. Commonly used JSON parsing libraries include: the standard library JSON Unmarshal, valyala / fastjson, tidwall / gjson, and json-iterator / go. The performance and resource consumption of these four JSON parsing libraries are shown in Table 4.

[0117] Table 4 shows the performance and resource consumption of four JSON parsing libraries.

[0118]

[0119] Note: ns / op represents the time (nanoseconds) required to perform one operation; B / op represents the memory (bytes) required to perform one operation; allocs / op represents the number of times memory is allocated in one operation.

[0120] It can be seen that the tidwall / gjson parsing library has the best performance and resource consumption, but converting it to a map requires separate extraction and reconstruction. In contrast, the json-iterator / go parsing library can do this in one step. Considering both performance and ease of use, the json-iterator / go parsing library is chosen.

[0121] In some embodiments, the data source interface is used to receive data to be sent at the current moment. The methods of the data source interface are shown in Table 5.

[0122] Table 5. Description of methods for the data source interface

[0123]

[0124] In some embodiments, the data output interface is used to send data to a target device. Methods for using the data output interface are shown in Table 6.

[0125] Table 6. Description of the data output interface methods

[0126]

[0127] In some embodiments, the data storage interface is used to store the first data in the case that the transmission of the first data in the data to be sent at the current moment fails. The methods of the data storage interface are shown in Table 7.

[0128] Table 7 Description of Data Storage Interface Methods

[0129]

[0130]

[0131] In some embodiments, the breakpoint resume interface is used to send unsent data stored in the database to the target device. The methods of the breakpoint resume interface are shown in Table 8.

[0132] Table 8. Description of the methods for the breakpoint resume interface.

[0133]

[0134] Figure 3 A schematic diagram illustrating the working principle of a data processing architecture provided in an embodiment of the present invention is shown below. Figure 3 The RESTful API configures the rule interface and data source interface by calling the management interface based on the configuration information received from the configuration storage interface. After receiving the data to be sent at the current time, the rule interface preprocesses the data and then sends it to the target device through the data output interface. If the data output interface fails to send the data, the first data in the data to be sent at the current time is stored in the database through the data storage interface, and then the first data stored in the database is sent to the target device through the breakpoint resume interface.

[0135] Figure 4 This is a schematic diagram of the data flow in the first data processing method provided in this embodiment of the invention. (See attached diagram.) Figure 4 The source data refers to the data to be sent at the current moment, received from the data source interface. If the data to be sent at the current moment fails to be sent, the source data will be sent to the target device through the breakpoint resume interface.

[0136] Figure 5 This is a data flow diagram illustrating the second data processing method provided in this embodiment of the invention. (See attached diagram.) Figure 5 The system receives the data to be sent from the data source interface, parses it to obtain parsed data, preprocesses it using the rules interface to obtain processed data, and then sends the processed data to the target device via the data output interface. If the data output interface fails to send the processed data, the system stores the processed data in a database via the data storage interface, then reads the unsent data from the database and sends it to the target device via the breakpoint resume interface.

[0137] Figure 6The following is a flowchart illustrating a specific implementation of a data processing method provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the process may include:

[0138] Step 601: Receive the data to be sent at the current time from the data source interface.

[0139] Step 602: Determine whether the connection status between the electronic device and the target device is the first connection status. If yes, proceed to step 603; otherwise, proceed to step 605.

[0140] Step 603: Send the data to be sent at the current moment to the target device through the data output interface.

[0141] Step 604: Determine if the transmission failed. If yes, proceed to step 605; otherwise, end the process.

[0142] Step 605: Determine whether a breakpoint resume interface is configured. If yes, proceed to step 606; otherwise, end the process.

[0143] Step 606: Determine the connection status between the electronic device and the target device as the second connection status.

[0144] Step 607: Store the first data in the data to be sent at the current moment into the database.

[0145] In this embodiment of the invention, the database is a time-series database, such as influx Database (influxDB).

[0146] Step 608: Determine whether the first data was successfully stored. If yes, proceed to step 609; otherwise, end the process.

[0147] Step 609: Attempt to send the unsent data stored in the database to the target device via the breakpoint resume interface.

[0148] Step 610: Determine whether the transmission was successful. If yes, proceed to step 611; otherwise, proceed to step 609.

[0149] Step 611: Determine if "is need lock" is marked as True. If yes, proceed to step 612; otherwise, proceed to step 613.

[0150] In this embodiment of the invention, "is need lock" indicates whether the database needs to be in a locked state. If "is need lock" is marked as True, the database needs to be in a locked state. If "is need lock" is marked as False, the database needs to be in an unlocked state.

[0151] Step 612: Determine the database status as locked, and proceed to step 613.

[0152] Step 613: Read 5000 data entries from the unsent data stored in the database.

[0153] In some embodiments, the preset data volume may be 5000 records, but this invention does not limit it.

[0154] Step 614: Send the 5000 read data entries to the target device.

[0155] Step 615: Determine if the transmission was successful. If yes, proceed to step 616; otherwise, proceed to step 609.

[0156] Step 616: Determine the sending timestamps of the 5000 data items sent this time and add them to the corresponding data in the database.

[0157] Step 617: Determine whether the amount of unsent data stored in the database is less than 5000. If yes, proceed to step 618; otherwise, proceed to step 611.

[0158] Step 618: Determine if "is need lock" is marked as True. If yes, proceed to step 620; otherwise, proceed to step 619.

[0159] Step 619: Set "is need lock" to True and proceed to step 611.

[0160] Step 620: Determine the connection status between the electronic device and the target device as the first connection status.

[0161] Step 621: Switch the database state from locked to unlocked.

[0162] Based on the same technical concept as the foregoing embodiments, see Figure 7 The present invention provides a data processing apparatus, the apparatus comprising:

[0163] The receiving module 701 is used to receive the data to be sent at the current time from the data source interface;

[0164] The first processing module 702 is configured to store the first data in a database when it is determined that the first data in the data to be sent at the current moment has failed to be sent, wherein the first data is at least a portion of the data in the current data to be sent; the database is further configured to receive data from the data source interface after storing the first data;

[0165] The second processing module 703 is configured to, when the amount of unsent data stored in the database is greater than or equal to a preset amount of data, read a preset amount of data from the unsent data stored in the database and send the read preset amount of data to the target device; and when the amount of unsent data stored in the database is less than the preset amount of data, read the unsent data stored in the database and send the unsent data to the target device.

[0166] In one implementation, the device further includes: a third processing module, which is configured to, after sending the read preset data amount to the target device, determine the sending timestamp of the preset data amount sent this time, and add the sending timestamp of the preset data amount sent this time to the corresponding data in the database;

[0167] Based on the data with added sending timestamps in the database, determine the unsent data in the database.

[0168] In one implementation, the third processing module is configured to read a preset amount of data from the unsent data stored in the database when the amount of unsent data stored in the database is greater than or equal to a preset amount of data, including:

[0169] Determine the last transmitted data in the database, and read a preset amount of data adjacent to the last transmitted data from the database; the last transmitted data represents the data in the database with the latest transmission timestamp;

[0170] When the amount of unsent data stored in the database is less than a preset amount, reading the unsent data stored in the database includes:

[0171] Determine the last transmitted data in the database, and read the untransmitted data adjacent to the last transmitted data from the database; the last transmitted data represents the data in the database with the latest transmission timestamp.

[0172] In one implementation, the device further includes: a fourth processing module; the fourth processing module is used to determine, when it is determined that the data to be sent at the current time has been successfully sent according to a preset first data sending method, the connection state between the electronic device and the target device is a first connection state, and the data sending method corresponding to the first connection state is the first data sending method;

[0173] If it is determined that the first data in the data to be sent at the current moment has failed to be sent, the connection state between the electronic device and the target device is determined to be a second connection state, and the steps of receiving data from the data source interface and storing the data received from the data source interface into the database are repeated; the second connection state is different from the first connection state.

[0174] After the unsent data is sent to the target device, the connection status between the electronic device and the target device is redefined as the first connection status.

[0175] In one implementation, the device further includes: a fifth processing module; the fifth processing module is used to determine the state of the database as a locked state when the amount of unsent data stored in the database is less than a preset amount of data, the locked state indicating that data from the data source interface is not allowed to be received;

[0176] After sending the unsent data to the target device, the database state is switched from locked to unlocked, indicating that the database is allowed to receive data from the data source interface.

[0177] In one implementation, at least one of the data source interface, data storage interface, and data output interface is a pre-configured interface, wherein the data storage interface represents an interface for configuring the database, and the data output interface represents an interface for configuring the target device.

[0178] In practical applications, the receiving module 701, the first processing module 702, and the second processing module 703 can all be implemented using a processor of an electronic device. The processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. This embodiment of the invention does not limit this.

[0179] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0180] It should be noted that, in the embodiments of the present invention, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a terminal, server, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0181] Correspondingly, this embodiment of the invention also provides a computer program product, which includes computer-executable instructions for implementing any of the data processing methods provided in this embodiment of the invention.

[0182] Accordingly, this embodiment of the invention further provides a computer storage medium storing computer-executable instructions, which are used to implement any of the data processing methods provided in the above embodiments.

[0183] In some embodiments, the functions or modules of the apparatus provided in the present invention can be used to execute the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0184] Based on the same technical concept as the foregoing embodiments, see Figure 8 The electronic device 800 provided in this embodiment of the invention may include: a memory 810 and a processor 820; wherein,

[0185] Memory 810 is used to store computer programs and data;

[0186] The processor 820 is configured to execute a computer program stored in the memory to implement any of the data processing methods described in the foregoing embodiments.

[0187] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities can be referred to each other. For the sake of brevity, they will not be repeated here.

[0188] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0189] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0190] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0191] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative and exemplary. The division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0192] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple grid units. Depending on the actual situation, some or all of the units may be selected to achieve the purpose of this embodiment.

[0193] In addition, each functional unit in the various embodiments of this application can be integrated into one processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0194] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium, and when executed, the program performs the steps of the above method embodiments. The above descriptions are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, When applied to electronic devices, the method includes: Receive the data to be sent at the current moment from the data source interface; If it is determined that the first data in the data to be sent at the current moment has failed to be sent, the first data is stored in the database, wherein the first data is at least a portion of the data in the current data to be sent; the database is also used to receive data from the data source interface after storing the first data; If the amount of unsent data stored in the database is greater than or equal to a preset amount of data, the preset amount of data is read from the unsent data stored in the database and the read preset amount of data is sent to the target device. If the amount of unsent data stored in the database is less than a preset amount, the unsent data stored in the database is read and sent to the target device. If the amount of unsent data stored in the database is less than a preset amount, and the electronic device is able to send data to the target device, the database is set to a locked state, which indicates that data from the data source interface is not allowed to be received. After sending the unsent data to the target device, the database state is switched from locked to unlocked, indicating that the database is allowed to receive data from the data source interface.

2. The method according to claim 1, characterized in that, The method further includes: After sending the preset amount of data to the target device, the sending timestamp of the preset amount of data sent this time is determined, and the sending timestamp of the preset amount of data sent this time is added to the corresponding data in the database; Based on the data with added sending timestamps in the database, determine the unsent data in the database.

3. The method according to claim 2, characterized in that, When the amount of unsent data stored in the database is greater than or equal to a preset amount of data, reading the preset amount of data from the unsent data stored in the database includes: Determine the last transmitted data in the database, and read a preset amount of data adjacent to the last transmitted data from the database; the last transmitted data represents the data in the database with the latest transmission timestamp; When the amount of unsent data stored in the database is less than a preset amount, reading the unsent data stored in the database includes: Determine the last transmitted data in the database, and read the untransmitted data adjacent to the last transmitted data from the database; the last transmitted data represents the data in the database with the latest transmission timestamp.

4. The method according to claim 1, characterized in that, The method further includes: If it is determined that the data to be sent at the current moment has been successfully sent according to the preset first data sending method, the connection state between the electronic device and the target device is determined to be the first connection state, and the data sending method corresponding to the first connection state is the first data sending method. If it is determined that the first data in the data to be sent at the current moment has failed to be sent, the connection state between the electronic device and the target device is determined to be a second connection state, and the steps of receiving data from the data source interface and storing the data received from the data source interface into the database are repeated; the second connection state is different from the first connection state. After the unsent data is sent to the target device, the connection status between the electronic device and the target device is redefined as the first connection status.

5. The method according to any one of claims 1 to 4, characterized in that, At least one of the data source interface, data storage interface, and data output interface is a pre-configured interface, wherein the data storage interface represents an interface for configuring the database, and the data output interface represents an interface for configuring the target device.

6. A data processing apparatus, characterized in that, The device, used in electronic devices, includes: The receiving module is used to receive the data to be sent at the current time from the data source interface; A first processing module is configured to store the first data in a database when it is determined that the transmission of the first data in the data to be sent at the current moment has failed, wherein the first data is at least a portion of the data in the current data to be sent; the database is further configured to receive data from the data source interface after storing the first data; The second processing module is used to read a preset amount of data from the unsent data stored in the database when the amount of unsent data stored in the database is greater than or equal to a preset amount of data, and then send the read preset amount of data to the target device. The sending module is used to read the unsent data stored in the database and send the unsent data to the target device when the amount of unsent data stored in the database is less than a preset amount of data. The fifth processing module is used to determine the state of the database as locked when the amount of unsent data stored in the database is less than a preset amount of data and the electronic device is able to send data to the target device. The locked state indicates that data from the data source interface is not allowed to be received. After the unsent data is sent to the target device, the state of the database is switched from locked to unlocked. The unlocked state indicates that data from the data source interface is allowed to be received.

7. The apparatus according to claim 6, characterized in that, The device further includes a third processing module, which is used to determine the sending timestamp of the preset data amount sent this time after sending the read preset data amount to the target device, and add the sending timestamp of the preset data amount sent this time to the corresponding data in the database. Based on the data with added sending timestamps in the database, determine the unsent data in the database.

8. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the data processing method according to any one of claims 1-5.

9. A computer storage medium storing a computer program; characterized in that, When the computer program is executed, it can implement the data processing method according to any one of claims 1-5.

Images