Information transmission method, device, computer system and computer readable storage medium

Abstract

The present disclosure provides an information transmission method, an information transmission device, a computer system and a computer-readable storage medium. The information transmission method includes: in the case of information loss during information transmission, determining an information field corresponding to the lost information, wherein the lost information includes information that has not been successfully sent to the business party, and the information field records the storage location of the data source corresponding to the lost information; creating a supplementary task for the information field, wherein the supplementary task includes: obtaining the data source corresponding to the lost information according to the storage location recorded in the information field, and processing the data source to obtain supplementary information identical to the lost information; and sending the supplementary information to the business party to achieve information transmission.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to an information transmission method, an information transmission device, a computer system, and a computer-readable storage medium. Background Art

[0002] With the rapid development of computers and the Internet, more and more interactions and services are realized through information transmission. Due to the instability of computer equipment, network environment or application programs, information loss is inevitable during the information transmission process, which affects interactions and services. Therefore, it is particularly important to ensure the reliable transmission of information.

[0003] In the process of implementing the concept of the present disclosure, the inventors discovered that there are at least the following problems in the related art: the reliability of information transmission is low. Summary of the invention

[0004] In view of this, the present disclosure provides an information transmission method, an information transmission device, a computer system, and a computer-readable storage medium.

[0005] One aspect of the present disclosure provides an information transmission method, comprising: in the event that information is lost during information transmission, determining an information field corresponding to the lost information, wherein the lost information includes information that has not been successfully sent to a business party, and the information field records a storage location of a data source corresponding to the lost information; creating a supplementary task for the information field, wherein the supplementary task comprises: acquiring a data source corresponding to the lost information according to the storage location recorded in the information field, and processing the data source to obtain supplementary information identical to the lost information; and sending the supplementary information to the business party to achieve information transmission.

[0006] Another aspect of the present disclosure provides an information transmission device, including: a determination module, used to determine an information field corresponding to the lost information when information is lost during the information transmission process, wherein the lost information includes information that has not been successfully sent to a business party, and the information field records the storage location of a data source corresponding to the lost information; a creation module, used to create a supplementary task for the information field, wherein the supplementary task includes: obtaining a data source corresponding to the lost information according to the storage location recorded in the information field, and processing the data source to obtain supplementary information identical to the lost information; and a sending module, used to send the supplementary information to the business party to achieve information transmission.

[0007] Another aspect of the present disclosure provides a computer system, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the information transmission method as described above.

[0008] Another aspect of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, wherein the instructions are used to implement the information transmission method as described above when executed.

[0009] Another aspect of the present disclosure provides a computer program, which includes computer executable instructions, and when the instructions are executed, are used to implement the information transmission method as described above.

[0010] According to an embodiment of the present disclosure, in the event that information is lost during information transmission, an information field corresponding to the lost information is determined, wherein the lost information includes information that is not successfully sent to the business party, and the information field records the storage location of the data source corresponding to the lost information; a supplementary task is created for the information field, wherein the supplementary task includes: obtaining the data source corresponding to the lost information according to the storage location recorded in the information field, and processing the data source to obtain supplementary information identical to the lost information; and sending the supplementary information to the business party to achieve a technical means of information transmission. Since the storage location of the data source corresponding to the lost information can be determined, the lost information can be retrieved according to the data source and sent to the business party in the form of supplementary information, thereby improving the reliability of information transmission and enabling the lost information to be recorded, so as to at least partially overcome the technical problem of unstable information transmission, thereby achieving the technical effect of achieving reliable transmission of each piece of information. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0012] Figure 1 An exemplary system architecture to which the information transmission method according to an embodiment of the present disclosure can be applied is schematically shown;

[0013] Figure 2 The flowchart of the information transmission method according to the embodiment of the present disclosure is schematically shown;

[0014] Figure 3 The information transmission flow chart of the Kafka system according to the embodiment of the present disclosure is schematically shown;

[0015] Figure 4Schematically shows an information transmission flow chart of the data source processing stage according to an embodiment of the present disclosure;

[0016] Figure 5 The information transmission flow chart of the information sending stage according to the embodiment of the present disclosure is schematically shown;

[0017] Figure 6 The structure diagram of the information field according to the embodiment of the present disclosure is schematically shown;

[0018] Figure 7 Schematically shows a flow chart of creating a supplementary task for an information field according to an embodiment of the present disclosure;

[0019] Figure 8 A block diagram schematically shows an information transmission device according to an embodiment of the present disclosure; and

[0020] Fig. 9 A block diagram of a computer system suitable for implementing the information transmission method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0022] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0023] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0024] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0025] Take the Kafka system as an example. Kafka is a high-throughput distributed publish-subscribe messaging system that can process large amounts of data in real time to meet various demand scenarios. As a commonly used real-time messaging system, Kafka works with various real-time data processing frameworks to serve businesses. However, in the data processing process of real-time tasks, message loss is still inevitable, which affects downstream businesses. The current solutions to this problem include: Storm message confirmation mechanism, Spark Streaming micro-batch processing mechanism (Spark Streaming is an extension of Spark core API that can achieve high-throughput, fault-tolerant real-time micro-batch data processing. Spark is a fast and general computing engine designed by Apache for large-scale data processing. Apache is a group that specializes in operating open source software projects and has many popular open source projects), and Flink distributed snapshot mechanism (Flink is an open source stream processing framework developed by the Apache Software Foundation).

[0026] However, in the process of implementing the concept disclosed in the present invention, the inventors found that the Storm message confirmation mechanism requires users to manage the status of messages themselves, which is not automated enough. In addition, the tracking, confirmation and processing measures of messages in the Storm message confirmation mechanism will increase the task load, resulting in increased latency, reduced throughput, etc., affecting task performance.

[0027] The inventors also found that the Spark Streaming micro-batch processing mechanism, because it is changed from stream processing to batch processing, means that users can no longer window data at arbitrary time but only at multiples of checkpoint intervals, which limits the programming model. Moreover, micro-batch processing obviously limits the delay of the job to the delay of micro-batch processing. Although sub-second batch processing delay is acceptable for simple applications, applications with multiple network shuffles (network exchanges) can easily extend the delay time to several seconds. Therefore, the micro-batch processing mechanism also causes certain delay problems.

[0028] The inventors also found that Flink's distributed snapshot mechanism, as an emerging mechanism, is incompatible with many historical businesses based on other real-time frameworks, requires high migration costs, and Flink cannot solve the message loss that occurs on Kafka or other nodes.

[0029] The embodiments of the present disclosure provide an information transmission method, an information transmission device, a computer system, and a computer-readable storage medium. The method includes: in the case of information loss during information transmission, determining an information field corresponding to the lost information, wherein the lost information includes information that has not been successfully sent to the business party, and the information field records the storage location of the data source corresponding to the lost information; creating a supplementary task for the information field, wherein the supplementary task includes: obtaining the data source corresponding to the lost information according to the storage location recorded in the information field, and processing the data source to obtain supplementary information identical to the lost information; and sending the supplementary information to the business party to achieve information transmission.

[0030] Figure 1 The exemplary system architecture 100 to which the information transmission method according to the embodiment of the present disclosure can be applied is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0031] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0032] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc.

[0033] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0034] The server 105 may be a server that provides various services, such as a background management server that provides support for messages subscribed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0035] It should be noted that the information transmission method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the information transmission device provided in the embodiment of the present disclosure can generally be set in the server 105. The information transmission method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the information transmission device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Alternatively, the information transmission method provided in the embodiment of the present disclosure can also be executed by the terminal devices 101, 102, or 103, or can also be executed by other terminal devices different from the terminal devices 101, 102, or 103. Accordingly, the information transmission device provided in the embodiment of the present disclosure can also be set in the terminal devices 101, 102, or 103, or can be set in other terminal devices different from the terminal devices 101, 102, or 103.

[0036] For example, the information field corresponding to the lost information may be originally stored in any one of the terminal devices 101, 102, or 103 (for example, the terminal device 101, but not limited thereto), or stored on an external storage device and may be imported into the terminal device 101. Then, the terminal device 101 may locally execute the information transmission method provided by the embodiment of the present disclosure, or send the information field corresponding to the lost information to other terminal devices, servers, or server clusters, and the other terminal devices, servers, or server clusters that receive the information field corresponding to the lost information may execute the information transmission method provided by the embodiment of the present disclosure.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0038] Figure 2 The flowchart of the information transmission method according to the embodiment of the present disclosure is schematically shown.

[0039] like Figure 2 As shown, the method includes operations S201 to S203.

[0040] In operation S201, when information loss occurs during information transmission, an information field corresponding to the lost information is determined, wherein the lost information includes information that is not successfully sent to the business party, and the information field records the storage location of the data source corresponding to the lost information.

[0041] In operation S202, a supplementary task is created for the information field, wherein the supplementary task includes: acquiring a data source corresponding to the lost information according to a storage location recorded in the information field, and processing the data source to obtain supplementary information identical to the lost information.

[0042] In operation S203, supplementary information is sent to the business party to realize information transmission.

[0043] Through the above-mentioned embodiments of the present disclosure, since supplementary information for the lost information can be obtained again according to the data source corresponding to the lost information, the problem of incomplete information transmission caused by information loss during the information transmission process can be effectively solved by sending supplementary information to the business party, thereby achieving the technical effect of realizing reliable and complete transmission of information.

[0044] The following is combined with specific embodiments and reference Figure 3 to Figure 6 ,right Figure 2 The method shown is further explained.

[0045] A specific embodiment of the present disclosure is to implement the automatic complement mechanism of the Kafka system (wherein complement means supplementary data, and the automatic complement mechanism means generating corresponding supplementary information for information lost during information transmission and sending it to the client) as an example.

[0046] Figure 3 The information transmission flow chart of the Kafka system according to an embodiment of the present disclosure is schematically shown.

[0047] See also Figure 3 As shown, the general information transmission process in the Kafka system is, for example: the upstream Kafka is used to store the data source. When there is information that needs to be transmitted, the data source is taken out from the upstream Kafka (i.e., consumption) and a real-time task is assigned to it. The real-time task processes the taken out data source into a new message (i.e., production). The produced new message is stored in the downstream Kafka, so that the business party can obtain the corresponding message from the downstream Kafka (i.e., consumption), thereby completing the normal information transmission process (without information loss) in the Kafka system.

[0048] In the case of abnormal (information loss) information transmission process in the Kafka system, combined with Figure 2 In the embodiment shown, the present invention saves the Offset information of the upstream Kafka message (i.e., the above-mentioned data source) in the entire information transmission process of the Kafka system. The Offset information can record the storage location of the data source corresponding to each piece of information in the information transmission process, so that the data source corresponding to the information lost in the subsequent information transmission process can be determined according to the Offset information, providing a basis for the subsequent re-production of supplementary information for the lost information.

[0049] according to Figure 3 In the embodiment shown, the information transmission process of the Kafka system may include, for example, a data source processing stage (for example, corresponding to Figure 3 The upstream Kafka to real-time task stage in the process) and the information sending stage (for example, corresponding to Figure 3The information in the process reaches the downstream Kafka and waits to be consumed by the business party). The method disclosed in the present invention for processing the problem of information loss that may occur during information transmission is, for example, manifested as: on the basis of the original real-time flow (upstream Kafka→consumption→real-time task→production→downstream Kafka→consumption→business party), two processes of real-time reporting of failed Offset information (real-time task→reporting failed Offset→complement module) and offline reporting of short Offset information (downstream Kafka→consumption→offline statistics of short offset→reporting short Offset→complement module) are added. The Offset information corresponding to the data source that failed to be processed in the data source processing stage is obtained through the process of real-time reporting of failed Offset information. The lost information (including upstream Kafka→consumption→real-time task→production→downstream Kafka) corresponding to the offset information of the data source, and then dynamically read these offset information through the complement module, and create corresponding complement tasks (i.e. the above-mentioned supplementary tasks) for these offset information, and execute these complement tasks by adding the complement consumption process from upstream Kafka to the complement module and the complement production process from the complement module to downstream Kafka, wherein the complement consumption process realizes the accurate consumption of the upstream Kafka data source corresponding to the information that failed to be processed in the data source processing stage or the information lost in the information transmission process detected in the downstream Kafka (i.e. the information sending stage), and the complement production process records the supplementary information produced into the downstream Kafka. In this way, the information transmission process (with information loss) under abnormal conditions in the Kafka system is completed.

[0050] Through the above-mentioned embodiments of the present disclosure, since the Offset information of the upstream Kafka of the message is stored in the entire message flow of the information transmission process, message complementation becomes possible, and complementation can be performed for information loss caused by any node in the entire data flow.

[0051] according to Figure 3 The embodiment shown in the figure is combined with another specific embodiment below, and with reference to Figure 4-5 , the above operation S201 occurring in the above data source processing stage and information sending stage is further explained respectively.

[0052] According to an embodiment of the present disclosure, for the data source processing stage, the above operation S201 includes: acquiring in real time through an exception capture mechanism the information field corresponding to the data source that failed to be processed in the data source processing stage; and using the information field corresponding to the data source that failed to be processed as the information field corresponding to the lost information.

[0053] Figure 4 The information transmission flow chart of the data source processing stage according to an embodiment of the present disclosure is schematically shown.

[0054] like Figure 4 As shown in the figure, because the data supplementation is required to be real-time, a process of reporting the failed Offset information in real time is required. An exception capture mechanism can be set in the real-time task to realize the real-time capture of the failed Offset information (i.e., the storage location of the data source recorded in the information field corresponding to the data source that failed to be processed in the data source processing stage), and the captured failed Offset information is reported to the supplementation module. For the data source that is successfully processed in the data source processing stage, new messages can be produced normally and transmitted to the downstream Kafka.

[0055] Through the above-mentioned embodiments of the present disclosure, since the process of real-time reporting of failed Offset information is added, the failed Offset information can be captured by the exception capture module added in the real-time task and directly reported to the complement module. The complement module can set the real-time reported failed Offset information to the highest priority, thereby ensuring timely complementation, further ensuring the real-time nature of the complement, and there is no need for the real-time task to provide a message reliability mechanism.

[0056] According to an embodiment of the present disclosure, for the information sending stage, the above operation S201 also includes: obtaining multiple information fields corresponding to all information sent within a preset time period in the information sending stage; sorting the multiple information fields; and determining the information field corresponding to the lost information based on the sorting result.

[0057] According to the embodiments of the present disclosure, in addition to real-time tasks, information loss may also occur in Kafka or other nodes. Therefore, only setting up the process of real-time reporting of failed Offset information for supplementation is not enough to ensure that all lost data is supplemented. Therefore, in this embodiment, the above-mentioned process of offline reporting of missing Offset information is also set up, such as Figure 3 As shown, this process can be implemented, for example, by setting up an offline Offset information statistics module, consuming downstream Kafka, and performing data statistics. Because each message in the downstream Kafka carries the corresponding Offset information in the upstream Kafka (i.e., the storage location information recorded in the information field corresponding to the information lost in the above information transmission process), the module can use a time window (i.e., the above preset time period) as an interval to count the missing Offset information in the information transmission process and report it to the complement module in batches. If the time window is relatively short, the module can also be called a quasi-real-time reporting module.

[0058] Figure 5The information transmission flow chart in the information sending stage according to an embodiment of the present disclosure is schematically shown.

[0059] like Figure 5 As shown in the figure, taking a 5-minute time window as an example, the Offset statistics module consumes the downstream Kafka, extracts the Offset information of the data source in the upstream Kafka corresponding to each message, and performs Offset statistics on the data that has been stored for 5 to 10 minutes (statisticed and reported). The statistical strategy includes sorting + traversal, because the information of each node in the information transmission process (including the upstream Kafka to the real-time task, the real-time task processing stage, and the real-time task to the downstream Kafka) will eventually be transmitted to the business party through the downstream Kafka. Therefore, the Offset information corresponding to the information lost at each node in the information transmission process can be determined according to the sorting and traversal results of the information in the downstream Kafka consumed by the Offset statistics module. Among them, the data in the interval of 0 to 5 minutes (to be counted) will be counted when its storage time reaches 5 to 10 minutes, and the data in the interval of more than 10 minutes (expired) represents the data that has been counted and reported.

[0060] According to an embodiment of the present disclosure, the above-mentioned information field includes information category, information partition and information offset, and the above-mentioned sorting of multiple information fields includes: obtaining multiple target information fields with the same information category and information partition from multiple information fields; and sorting the multiple target information fields according to the size of the information offset in each target information field.

[0061] Figure 6 The structure diagram of the information field according to the embodiment of the present disclosure is schematically shown.

[0062] According to the embodiments of the present disclosure, the storage location of a message in a Kafka system can be determined, for example, by Topic (i.e., the above-mentioned information category), Partition (i.e., the above-mentioned information partition) and Offset (i.e., the above-mentioned information offset). Offset is a record of the consumption location. Each Partition of each Topic in Kafka has an independent Offset list. The combination of the three can constitute the above-mentioned Offset information. Based on this, the above-mentioned Offset information of the upstream Kafka message saved in the entire information transmission process of the Kafka system can be expressed as follows: when the real-time task consumes the upstream Kafka, the Topic, Partition and Offset corresponding to the consumed data source are added as three additional fields to the processed message to obtain the information field (such as Figure 6 ) so that it flows with the data stream and provides the basis for precise complement.

[0063] According to the above-mentioned embodiments of the present disclosure, due to the queue structure of the Kafka system, the Offset of the same Topic and the same Partition should be continuous. After sorting multiple information fields in the above-mentioned manner, the Offset information corresponding to the messages lost in the information transmission process detected in the downstream Kafka can be counted, so that the lost Offset information can be reported to the complement module.

[0064] It should be noted that Kafka can correspond to multiple Brokers, each Broker can correspond to multiple Topics, each Topic can correspond to multiple Partitions on the Broker, and each Partition has its own Offset list to mark the position of the message. Offset can be equivalent to an array subscript, for example. Topic is used to characterize the category of each message published to the Kafka cluster (physically, messages of different Topics are stored separately, and logically, although the messages of a Topic are stored on one or more brokers, users only need to specify the Topic of the message to produce or consume data without having to worry about where the data is stored). Broker refers to one or more servers contained in the Kafka cluster, and Partition is the physical partition corresponding to each Topic, representing the actual storage location of the message.

[0065] Through the above-mentioned embodiments of the present disclosure, since the process of reporting the missing Offset information offline is added, and the downstream Kafka should store the information corresponding to each node in the information transmission process, the Offset information of the upstream Kafka corresponding to the message of the downstream Kafka can be parsed, and relying on the Offset continuity principle of Kafka, the Offset information lost in the information transmission process can be counted in the time-division window, and reported to the complement module for complementation, so that the Offset information statistics technology can be used to count the data lost by any node in the entire data link and realize the supplementary sending of information. At the same time, the time window for supplementary recording of messages can be freely selected, which further enhances the scalability of this solution.

[0066] According to the above implementation, another specific embodiment is described below. Figure 6 , the above operation S202 is further explained.

[0067] According to an embodiment of the present disclosure, the information field includes multiple information fields, and the operation S202 includes: reading the multiple information fields according to a preset priority; and creating a supplementary task for each read information field.

[0068] In combination with the above-mentioned embodiments of the present disclosure, the above-mentioned reading of multiple information fields according to a preset priority includes: preferentially reading the information fields determined in the data source processing stage; and after reading the information fields determined in the data source processing stage, reading the information fields determined in the information sending stage.

[0069] Figure 7 A flowchart of creating a supplementary task for an information field according to an embodiment of the present disclosure is schematically shown, through which the internal processing work of the complement module can be completed.

[0070] like Figure 7 As shown, the complement module of this embodiment includes, for example, the Maintainer process, the external Redis, the internal Redis, and the complement task. Among them, Redis (Remote Dictionary Server) is an open source log-type, Key-Value database written in ANSI C language, supporting the network, and can be based on memory and can also be persistent. The external Redis is used to store the Offset information reported from the outside (for example, it can include the Offset information reported in the above-mentioned data source processing stage and the information sending stage). The external Redis is divided into two keys, the real-time Key (used to store the real-time reported Offset information that failed to be processed in the above-mentioned data source processing stage) and the offline Key (used to store the Offset information lost during the information transmission process reported offline in the above-mentioned information sending stage). The value corresponding to each Key value is of List type, and the Offset information is stored in order in the List. The Maintainer process reads the Offset information from the external Redis with the above-mentioned preset priority, for example, it can be used to read the real-time Offset information first. The Maintainer then writes the Offset information into the internal Redis. The internal Redis is divided into N Key values. The Key values ​​and complement tasks correspond one to one. The Key value structure can be the same as the external Redis. The complement task takes the Offset information from its own Key for accurate complement. The Maintainer process can poll the consumption progress of each Key in the internal Redis to distribute the next round of Offset information: the Offset information in the external Redis is evenly distributed to the consumed Key values ​​in the internal Redis according to the preset priority order.

[0071] Through the above-mentioned embodiments of the present disclosure, a complement module that can complete data supplementation service is realized. Through the external Redis in the module, the Offset information reported from the outside can be accepted, the Maintainer can distribute the message, and the internal Redis can provide Offset information for multiple complement tasks in running states, and then the complement tasks can accurately consume according to the Offset information to complete the complement.

[0072] According to another embodiment of the present disclosure, the above-mentioned information transmission method can also be applied to the distributed snapshot mechanism of Apache Flink, which can provide low-cost message real-time reliability guarantee for the above-mentioned data source processing stage.

[0073] The above-mentioned information transmission method provided by the embodiment of the present disclosure does not require the user to maintain the message status himself in the task, and does not require the task to bear the performance burden brought by the message reliability guarantee. It is compatible with any real-time processing framework and can also make up for the data loss caused by any node in the entire data flow, thereby realizing a highly compatible, real-time and more reliable automatic complement mechanism.

[0074] Figure 8 A block diagram of an information transmission device according to an embodiment of the present disclosure is schematically shown.

[0075] like Figure 8 As shown, the information transmission device 800 includes a determination module 810 , a creation module 820 and a sending module 830 .

[0076] The determination module 810 is used to determine the information field corresponding to the lost information when information loss occurs during information transmission, wherein the lost information includes information that is not successfully sent to the business party, and the information field records the storage location of the data source corresponding to the lost information.

[0077] The creation module 820 is used to create a supplementary task for the information field, wherein the supplementary task includes: obtaining a data source corresponding to the lost information according to the storage location recorded in the information field, and processing the data source to obtain supplementary information identical to the lost information.

[0078] The sending module 830 is used to send supplementary information to the business party to realize information transmission.

[0079] According to an embodiment of the present disclosure, the above-mentioned information transmission process includes a data source processing stage, and the above-mentioned determination module includes a first acquisition unit and a definition unit.

[0080] The first acquisition unit is used to acquire in real time through an exception capture mechanism the information field corresponding to the data source that failed to be processed in the data source processing stage.

[0081] A definition unit is used to use the information field corresponding to the data source that failed to be processed as the information field corresponding to the lost information.

[0082] According to an embodiment of the present disclosure, the above-mentioned information transmission process also includes an information sending stage, and the above-mentioned determination module also includes a second acquisition unit, a sorting unit and a determination unit.

[0083] The second acquisition unit is used to acquire a plurality of information fields corresponding to all information sent within a preset time period during the information sending phase.

[0084] A sorting unit is used to sort multiple information fields.

[0085] The determination unit is used to determine the information field corresponding to the lost information according to the sorting result.

[0086] According to an embodiment of the present disclosure, the above-mentioned information field includes information category, information partition and information offset, and the above-mentioned sorting unit includes an acquisition subunit and a sorting subunit.

[0087] The acquisition subunit is used to acquire multiple target information fields having the same information category and information partition from multiple information fields.

[0088] The sorting subunit is used to sort the multiple target information fields according to the size of the information offset in each target information field.

[0089] According to an embodiment of the present disclosure, the above-mentioned information field includes multiple ones, and the above-mentioned creation module includes a reading unit and a creation unit.

[0090] The reading unit is used to read a plurality of information fields according to a preset priority.

[0091] A creation unit is used to create a supplementary task for each information field read.

[0092] According to an embodiment of the present disclosure, the above-mentioned information transmission process includes a data source processing stage and an information sending stage, and the above-mentioned reading unit includes a first reading sub-unit and a second reading sub-unit.

[0093] The first reading subunit is used to preferentially read the information field determined in the data source processing stage.

[0094] The second reading subunit is used to read the information field determined in the information sending stage after the information field determined in the data source processing stage is read.

[0095] According to the embodiments of the present invention, any one or more of the modules, units, and sub-units, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, units, and sub-units can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, units, and sub-units can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, units, and sub-units can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0096] For example, any multiple of the determination module 810, the creation module 820, and the sending module 830 can be combined in one module / unit / subunit for implementation, or any one of the modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functions of one or more of these modules / units / subunits can be combined with at least part of the functions of other modules / units / subunits and implemented in one module / unit / subunit. According to an embodiment of the present disclosure, at least one of the determination module 810, the creation module 820, and the sending module 830 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in a suitable combination of any of them. Alternatively, at least one of the determination module 810 , the creation module 820 , and the sending module 830 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0097] It should be noted that the information transmission device part in the embodiment of the present disclosure corresponds to the information transmission method part in the embodiment of the present disclosure. The description of the information transmission device part specifically refers to the information transmission method part, which will not be repeated here.

[0098] Fig. 9 A block diagram of a computer system suitable for implementing the information transmission method according to an embodiment of the present disclosure is schematically shown. Figure 8The computer system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0099] like Fig. 9 As shown, the computer system 900 according to the embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 902 or the program loaded from the storage part 908 to the random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0100] In RAM 903, various programs and data required for the operation of system 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other through bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0101] According to an embodiment of the present disclosure, the system 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The system 900 may further include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0102] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0103] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0104] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0105] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 902 and / or the RAM 903 described above and / or one or more memories other than the ROM 902 and the RAM 903 .

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0107] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. An information transmission method, comprising: In the event that information is lost during information transmission, determining the information field corresponding to the lost information includes: obtaining in real time through an exception capture mechanism the information field corresponding to the data source that failed to be processed in the data source processing stage of the information transmission process, and using the information field corresponding to the data source that failed to be processed as the information field corresponding to the lost information; or obtaining multiple information fields corresponding to all information sent within a preset time period in the information sending stage, sorting the multiple information fields, and determining the information field corresponding to the lost information according to the sorting result; wherein the lost information includes information that was not successfully sent to the business party, and the information field records the storage location of the data source corresponding to the lost information; Creating a supplementary task for the information field, wherein the supplementary task includes: acquiring a data source corresponding to the lost information according to the storage location recorded in the information field, and processing the data source to obtain supplementary information identical to the lost information; and The supplementary information is sent to the business party to realize information transmission.

2. The method according to claim 1, wherein: The information field includes an information category, an information partition, and an information offset, and the sorting of the plurality of information fields includes: Acquire a plurality of target information fields having the same information category and information partition from the plurality of information fields; and The plurality of target information fields are sorted according to the size of the information offset in each of the target information fields.

3. An information transmission method, comprising: In the event that information is lost during information transmission, determining an information field corresponding to the lost information, wherein the lost information includes information that is not successfully sent to the business party, and the information field records the storage location of the data source corresponding to the lost information, and the information field includes multiple; Creating a supplementary task for the information field includes: reading a plurality of the information fields according to a preset priority; and creating a supplementary task for each of the read information fields, wherein the supplementary task includes: acquiring a data source corresponding to the lost information according to a storage location recorded in the information field, and processing the data source to obtain supplementary information identical to the lost information; and The supplementary information is sent to the business party to realize information transmission.

4. The method according to claim 3, wherein: The information transmission process includes a data source processing stage and an information sending stage, and the reading of the plurality of information fields according to a preset priority includes: Prioritize reading the information fields determined in the data source processing stage; and After the information field determined in the data source processing stage is read, the information field determined in the information sending stage is read.

5. An information transmission device, comprising: A determination module, used to determine the information field corresponding to the lost information when information is lost during information transmission, wherein the lost information includes information that is not successfully sent to the business party, and the information field records the storage location of the data source corresponding to the lost information, and the determination module includes: a first acquisition unit, used to acquire in real time through an exception capture mechanism the information field corresponding to the data source that fails to be processed in the data source processing stage of the information transmission process, and a definition unit, used to use the information field corresponding to the data source that fails to be processed as the information field corresponding to the lost information; or a second acquisition unit, used to acquire multiple information fields corresponding to all information sent within a preset time period in the information sending stage, a sorting unit, used to sort the multiple information fields, and a determination unit, used to determine the information field corresponding to the lost information according to the sorting result; a creation module, configured to create a supplementary task for the information field, wherein the supplementary task comprises: acquiring a data source corresponding to the lost information according to a storage location recorded in the information field, and processing the data source to obtain supplementary information identical to the lost information; and The sending module is used to send the supplementary information to the business party to realize information transmission.

6. An information transmission device, comprising: a determination module, configured to determine, when information is lost during information transmission, an information field corresponding to the lost information, wherein the lost information includes information that is not successfully sent to the business party, and the information field records the storage location of the data source corresponding to the lost information, and the information field includes a plurality of information fields; a creation module, configured to create a supplementary task for the information field, the creation module comprising: a reading unit, configured to read a plurality of the information fields according to a preset priority; and a creation unit, configured to create a supplementary task for each of the read information fields, wherein the supplementary task comprises: obtaining a data source corresponding to the lost information according to a storage location recorded in the information field, and processing the data source to obtain supplementary information identical to the lost information; and The sending module is used to send the supplementary information to the business party to realize information transmission.

7. A computer system comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 4.

Images