Data sending and writing method, device, electronic device and readable storage medium

Abstract

The embodiments of the present application provide a method, device, electronic device and readable storage medium for sending and writing data. The method includes: obtaining serialized data from Kafka; determining the schema corresponding to the identification information carried by the serialized data; parsing the serialized data based on the schema to obtain the source data; determining whether there is an ES model corresponding to the source data, and if so, converting the source data into a format and writing it into the ES model. Based on this solution, serialized data can be obtained from Kafka in real time, and the corresponding schema can be determined based on the identification information, thereby parsing the serialized data based on the schema to obtain the source data, and converting the source data into a format and writing it into the corresponding ES model according to the mapping rules, thereby ensuring the consistency of the data format from data collection to data processing, and realizing real-time analysis and transformation of business data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology. Specifically, the present application relates to a method and device for sending and writing data, an electronic device and a readable storage medium. Background Art

[0002] The business processes in the financial system are relatively complex, and the amount of business data generated by each business node system is also relatively huge. It is necessary to obtain business data in a timely manner for processing and analysis to support operations.

[0003] At present, the business data is mostly processed in a periodic batch acquisition mode, which is slow and cannot meet the operational needs. In addition, the data structures of the data collected in each system may be different, which makes it inconvenient to process and analyze the business data and is not conducive to the use of business data. Summary of the invention

[0004] The purpose of this application is to solve at least one of the above technical defects. The technical solution adopted by this application is as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for sending data, the method comprising:

[0006] Get binlog data from the database of the monitored system;

[0007] Parse binlog data to obtain source data;

[0008] Based on whether there is a schema matching the source data, the source data is serialized to obtain serialized data, and the serialized data contains the identification information of the schema;

[0009] Publish serialized data to Kafka.

[0010] Optionally, based on whether there is a schema matching the source data, serialize the source data, including:

[0011] Make sure there is a schema in the local cache that matches the source data;

[0012] If it exists, the schema is used to serialize the source data;

[0013] If it does not exist, the schema is registered and the source data is serialized using the schema.

[0014] Optionally, register the schema, including:

[0015] Determine if there is a registered schema with the same name as the schema;

[0016] If it does not exist, register the schema;

[0017] If it exists, determine whether the data structure corresponding to the registered schema is consistent with the data structure of the source data. If not, register the schema; if consistent, use the registered schema as the schema.

[0018] In a second aspect, an embodiment of the present application provides a method for writing data, the method comprising:

[0019] Get serialized data from Kafka;

[0020] Determine the schema corresponding to the identification information carried by the serialized data;

[0021] Parse the serialized data based on the schema to obtain the source data;

[0022] Determine whether there is an ES model corresponding to the source data. If so, convert the source data into a new format and write it into the ES model.

[0023] Optionally, the above writing method further includes:

[0024] Determine whether the target data field exists in the source data. The target data field does not exist in the ES model.

[0025] If it exists, the ES model is expanded with the target data field.

[0026] Optionally, the above writing method further includes:

[0027] If there is no ES model corresponding to the source data, the source data is determined to be abnormal data.

[0028] Optionally, the above writing method further includes:

[0029] According to the preset retry strategy, determine whether there is an ES model corresponding to the abnormal data.

[0030] In a third aspect, an embodiment of the present application provides a data sending device, the device comprising:

[0031] The data collection module is used to obtain binlog data from the database of the monitored system;

[0032] The source data acquisition module is used to parse binlog data and obtain source data;

[0033] The serialization module is used to serialize the source data to obtain serialized data based on whether there is a schema matching the source data. The serialized data contains the identification information of the schema.

[0034] The data publishing module is used to publish serialized data to Kafka.

[0035] Optionally, the serialization module is specifically used to:

[0036] Make sure there is a schema in the local cache that matches the source data;

[0037] If it exists, the schema is used to serialize the source data;

[0038] If it does not exist, the schema is registered and the source data is serialized using the schema.

[0039] Optionally, the serialization module is used when registering the schema:

[0040] Determine if there is a registered schema with the same name as the schema;

[0041] If it does not exist, register the schema;

[0042] If it exists, determine whether the data structure corresponding to the registered schema is consistent with the data structure of the source data. If not, register the schema; if consistent, use the registered schema as the schema.

[0043] In a fourth aspect, an embodiment of the present application provides a data writing device, the device comprising:

[0044] Data acquisition module, used to obtain serialized data from Kafka;

[0045] The schema determination module is used to determine the schema corresponding to the identification information carried by the serialized data;

[0046] The data parsing module is used to parse the serialized data based on the schema to obtain the source data;

[0047] The data writing module is used to determine whether there is an ES model corresponding to the source data. If so, the source data is formatted and written into the ES model.

[0048] Optionally, the above device further includes a field expansion module, and the field expansion module is used to:

[0049] Determine whether the target data field exists in the source data. The target data field does not exist in the ES model.

[0050] If it exists, the ES model is expanded with the target data field.

[0051] Optionally, the above device further includes:

[0052] The abnormal data determination module is used to determine the source data as abnormal data when there is no ES model corresponding to the source data.

[0053] Optionally, the above device further includes:

[0054] The abnormal data retry module is used to determine whether there is an ES model corresponding to the abnormal data according to a preset retry strategy.

[0055] In a fifth aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory;

[0056] A memory, used for storing operation instructions;

[0057] The processor is used to execute the method shown in any implementation of the first aspect or any implementation of the second aspect of the present application by calling an operation instruction.

[0058] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method shown in any implementation of the first aspect or any implementation of the second aspect of the present application.

[0059] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0060] The solution provided in the embodiment of the present application obtains serialized data from Kafka, determines the schema corresponding to the identification information carried by the serialized data; parses the serialized data based on the schema to obtain the source data, determines whether there is an ES model corresponding to the source data, and if so, converts the source data into a format and writes it into the ES model. Based on this solution, it is possible to obtain serialized data from Kafka in real time, and determine the corresponding schema based on the identification information, thereby parsing the serialized data based on the schema to obtain the source data, and converting the source data into a format and writing it into the corresponding ES model according to the mapping rules, thereby ensuring the consistency of the data format from data acquisition to processing, and realizing real-time analysis and transformation of business data. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0062] Figure 1 A flowchart of a method for sending data provided in an embodiment of the present application;

[0063] Figure 2 A flowchart of a specific implementation method of the data sending method provided in an embodiment of the present application;

[0064] Figure 3 A schematic diagram of a data writing method provided in an embodiment of the present application;

[0065] Figure 4 The overall logical architecture diagram of data collection, transmission and writing provided in the embodiment of the present application;

[0066] Figure 5 A schematic diagram of the structure of a data sending device provided in an embodiment of the present application;

[0067] Figure 6 A schematic diagram of the structure of a data writing device provided in an embodiment of the present application;

[0068] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present invention.

[0070] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0071] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0072] The source data collected in the prior art is generally a JavaScript Object Notation (JSON) character set, and the structure rules of the data are uncertain, so it is not convenient for analysis and use, and the data structure change process cannot be monitored during the transmission process.

[0073] In the existing technology, business data is often acquired in batches periodically for processing, which is slow and cannot meet the needs of operational use.

[0074] The data sending and writing method, device, electronic device and readable storage medium provided in the embodiments of the present application are intended to solve at least one of the above technical problems in the prior art.

[0075] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0076] Figure 1 A schematic diagram of a data transmission method provided in an embodiment of the present application is shown. Figure 1 As shown, the method may mainly include:

[0077] Step S110: Obtain binlog (binary log) data from the database of the monitored system;

[0078] Step S120: Parse the binlog data to obtain source data;

[0079] Step S130: based on whether there is a schema matching the source data, the source data is serialized to obtain serialized data, and the serialized data contains identification information of the schema;

[0080] Step S140: Publish the serialized data to Kafka.

[0081] In the embodiment of the present application, data changes in the database of the monitored system can be monitored in real time based on Canal, and the source data can be obtained by parsing binlog.

[0082] In an embodiment of the present application, after parsing the source data, avro serialization can be performed according to a schema that matches the source data to obtain serialized data, and the serialized data can be published to Kafka.

[0083] In the embodiment of the present application, the identification information is used to identify the schema, so that when the serialized data is deserialized, the schema can be obtained through the identification information, thereby parsing the serialized data to obtain the source data.

[0084] In the embodiment of the present application, serialized data is published to Kafka, so that data consumers can obtain data in a timely manner, which can improve the timeliness of data acquisition to meet operational usage requirements.

[0085] The method provided in the embodiment of the present application parses the binlog data obtained from the database of the monitored system to obtain the source data, and serializes the source data based on the schema matching the source data, thereby obtaining serialized data and sending it to Kafka. Based on this solution, the collected data can be serialized and published to Kafka, and the schema used in serialization can be identified, which is conducive to improving the timeliness of business data, and provides a basis for obtaining serialized data from Kafka, parsing serialized data using the corresponding schema, and converting source data into the corresponding data format, which is conducive to processing and analyzing business data and facilitating the use of business data.

[0086] In an optional manner of the embodiment of the present application, serializing the source data based on whether there is a schema matching the source data includes:

[0087] Make sure there is a schema in the local cache that matches the source data;

[0088] If it exists, the schema is used to serialize the source data;

[0089] If it does not exist, the schema is registered and the source data is serialized using the schema.

[0090] In an embodiment of the present application, when the source data is serialized based on the schema, it can be determined whether there is a schema matching the source data in the local cache. If so, the schema matching the source data in the cache can be used for processing.

[0091] If there is no schema matching the source data, you can register a schema that matches the source data and serialize the source data using the registered schema. In actual use, you can store the registered schema in the local cache to reduce repeated registration of the schema.

[0092] In an optional manner of the embodiment of the present application, registering a schema includes:

[0093] Determine if there is a registered schema with the same name as the schema;

[0094] If it does not exist, register the schema;

[0095] If it exists, determine whether the data structure corresponding to the registered schema is consistent with the data structure of the source data. If not, register the schema; if they are consistent, use the registered schema as the schema. In an embodiment of the present application, the registration of the current schema can be completed through the schema registration center. Specifically, it can be determined whether there is a registered schema with the same name as the current schema in the schema registration center. If not, the current schema can be registered; if it exists, it can be determined whether the data structure corresponding to the schema is consistent with the data structure of the source data corresponding to the current schema. If they are consistent, the registered schema can be used for serialization; if they are inconsistent, register the current schema.

[0096] Figure 2 A flow chart of a specific implementation of the data sending method provided in an embodiment of the present application is shown in the figure, wherein the collection target mysql, that is, the collection change data from the database of the monitored system. Canal collects data, that is, obtains the source data by collecting and parsing binlog. Obtain information such as library name, table name, operation type, etc., that is, obtains the information required for matching the schema. Obtain the schema used this time, that is, obtains the schema that matches the source data format. Fill the schema with this data, that is, fill the source data into the schema. Perform avro encoding, that is, serialize the source data filled into the schema. Send to kafka, that is, publish the serialized data to kafka.

[0097] One table corresponds to one schema, and the naming rule of schemaName is: library name_table name, that is, the data table structure of various source data is respectively with schema, and the schema can be named by library name and table name. All columns in this data are filled into newschema, and the existence of the table oldschema is queried through schemaName, that is, when matching the schema with the source data, all fields of the source data are first filled into the schema, and the schema is named by the source data, and the name of the schema corresponding to the source data is used to find out whether the schema exists in the cache. Whether the hashcode of oldschema is the same as that of newschema, that is, there is a schema in the cache with the same name as the schema corresponding to the source data, then the hash values ​​of the two can be compared to see if they are the same. If they are the same, the schema in the cache is used to fill the source data.

[0098] If there is no schema with the same name as the schema corresponding to the source data in the cache, or there is a schema with the same name as the schema corresponding to the source data in the cache, but the hash values ​​of the two are different, you can register the schema with the schema registry. Check whether the schema registry has a version with the same structure as newschema, that is, query the registry to see if there is version information of the schema with the same structure as the schema corresponding to the source data. Get the registered existschema through the version, that is, when there is version information of the schema with the same structure as the schema corresponding to the source data in the registry, get the schema that matches the source data through the version information. When the schema registry does not have version information of the schema with the same structure as the schema corresponding to the source data, register the schema corresponding to the source data. After the schema registry completes the registration of the schema, the source data can be filled into the schema.

[0099] Figure 3 A schematic diagram of a data writing method provided in an embodiment of the present application is shown. Figure 3 As shown, the method may mainly include:

[0100] Step S210: Obtain serialized data from Kafka;

[0101] Step S220: Determine the schema corresponding to the identification information carried by the serialized data;

[0102] Step S230: Parse the serialized data based on the schema to obtain source data;

[0103] Step S240: Determine whether there is an ES (Elasticsearch, data copy) model corresponding to the source data. If so, convert the source data into a new format and write it into the ES model.

[0104] In an embodiment of the present application, serialized data can be obtained from Kafka, and the corresponding schema can be obtained through identification information, so that the serialized data can be deserialized based on the schema to obtain the source data.

[0105] In the embodiment of the present application, the identification information is used to identify the schema, and according to the correspondence between the identification information and the schema, the schema matching the source data can be obtained to implement the parsing of the source data.

[0106] In actual use, the deserialization and parsing of serialized data can be implemented based on KafkaDeserializationSchema.

[0107] In the embodiment of the present application, since serialized data can be obtained from Kafka and the serialized data can be parsed in time to obtain the source data, data consumers can obtain the source data in a timely manner, which can improve the timeliness of data acquisition to meet operational usage requirements.

[0108] In an embodiment of the present application, after the source data is obtained, if there is an ES model corresponding to the source data, the source data can be formatted and written into the ES model.

[0109] When absorbing source data into the ES model, the watermark technology can be superimposed through the Flink stream data window to ensure the data sequence within the window time and write the source data into the ES model in batches.

[0110] Since the source data is converted into a format and then written into the ES model, the data format is unified, and data can be obtained from the ES model for processing and analysis, making it easier to use the data.

[0111] The method provided in the embodiment of the present application obtains serialized data from Kafka, determines the schema corresponding to the identification information carried by the serialized data; parses the serialized data based on the schema to obtain the source data, determines whether there is an ES model corresponding to the source data, and if so, converts the format of the source data and writes it into the ES model. Based on this solution, it is possible to obtain serialized data from Kafka in real time, and determine the corresponding schema based on the identification information, thereby parsing the serialized data based on the schema to obtain the source data, and converting the format of the source data and writing it into the corresponding ES model according to the mapping rules, thereby ensuring the consistency of the data format from data collection to processing, and realizing real-time analysis and transformation of business data.

[0112] In an optional manner of the embodiment of the present application, the above method further includes:

[0113] Determine whether the target data field exists in the source data. The target data field does not exist in the ES model.

[0114] If it exists, the ES model is expanded with the target data field.

[0115] In the embodiment of the present application, the source data may include a target data field, but the ES model does not include the target data field. The ES model may be expanded with the target data field.

[0116] In an optional manner of the embodiment of the present application, the above method further includes:

[0117] If there is no ES model corresponding to the source data, the source data is determined to be abnormal data.

[0118] In the embodiment of the present application, the source data and the ES model are pre-configured with a corresponding relationship. If there is no ES model corresponding to the source data, the source data can be determined as abnormal data. The abnormal data cannot be directly written into the ES model.

[0119] In an optional manner of the embodiment of the present application, the above method further includes:

[0120] According to the preset retry strategy, determine whether there is an ES model corresponding to the abnormal data.

[0121] In an embodiment of the present application, a retry strategy can be preset for abnormal data, for example, it is determined again whether there is an ES model corresponding to the abnormal data after each preset time period. If the abnormal data cannot be written into the ES model after the number of retries exceeds the specified number, the abnormal data can be defined as dirty data.

[0122] In an embodiment of the present application, a data management system can be deployed to collect and write data into an ES model, and changes in the data structure can be monitored and the current data structure of the data can be displayed.

[0123] Figure 4 The overall logical architecture diagram of data collection, transmission and writing provided by the embodiment of the present application is shown.

[0124] The data source system includes multiple business subsystems that generate source data in the business process. The data acquisition and sending device module subscribes to and listens to the source data provided by the data source system, corresponding to the data sending method provided in the embodiment of the present application. After serialization, the source data will be published on the kafka acquisition bus. The Flink data parsing and writing module corresponds to the data writing method provided in the embodiment of the present application. By subscribing to the kafka bus, the serialized data stream is obtained, and the parsed and deserialized data is obtained. The data matched to the ES model through the metadata rules can be written into the ES index. After being processed by the data writing method of the embodiment of the present application, the batch processing module provides secondary refinement of the ES library data, and the oas-app-apply data service module can obtain real-time data through the ES library to provide data services to the outside.

[0125] Based on Figure 1 The same principle as shown in the method, Figure 5 A schematic diagram of the structure of a data sending device provided in an embodiment of the present application is shown. Figure 5 As shown, the data sending device 30 may include:

[0126] The data collection module 310 is used to obtain binlog data from the database of the monitored system;

[0127] The source data acquisition module 320 is used to parse the binlog data and obtain the source data;

[0128] A serialization module 330 is used to serialize the source data to obtain serialized data based on whether there is a schema matching the source data, and the serialized data contains identification information of the schema;

[0129] The data publishing module 340 is used to publish the serialized data to Kafka.

[0130] The device provided in the embodiment of the present application parses the binlog data obtained from the database of the monitored system to obtain the source data, and serializes the source data based on the schema matching the source data, thereby obtaining serialized data and sending it to Kafka. Based on this solution, the collected data can be serialized and published to Kafka, and the schema used in serialization can be identified, which is conducive to improving the timeliness of business data, and provides a basis for obtaining serialized data from Kafka, parsing serialized data using the corresponding schema, and converting source data into the corresponding data format, which is conducive to processing and analyzing business data and facilitating the use of business data.

[0131] Optionally, the serialization module is specifically used to:

[0132] Make sure there is a schema in the local cache that matches the source data;

[0133] If it exists, the schema is used to serialize the source data;

[0134] If it does not exist, the schema is registered and the source data is serialized using the schema.

[0135] Optionally, the serialization module is used when registering the schema:

[0136] Determine whether there is version information corresponding to the schema;

[0137] If it exists, register the schema with the version information;

[0138] If it does not exist, create the version information corresponding to the schema and register the schema through the version information.

[0139] It can be understood that the above modules of the data sending device in this embodiment have the function of realizing Figure 1The functions of the corresponding steps of the data sending method in the embodiment shown in . The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented separately or integrated with multiple modules. For the functional description of each module of the above data sending device, please refer to Figure 1 The corresponding description of the method for sending data in the embodiment shown in is not repeated here.

[0140] Based on Figure 3 The same principle as shown in the method, Figure 6 A schematic diagram of the structure of a data writing device provided in an embodiment of the present application is shown. Figure 6 As shown, the data writing device 40 may include:

[0141] The data acquisition module 410 is used to obtain serialized data from Kafka;

[0142] A schema determination module 420, used to determine the schema corresponding to the identification information carried by the serialized data;

[0143] The data parsing module 430 is used to parse the serialized data based on the schema to obtain the source data;

[0144] The data writing module 440 is used to determine whether there is an ES model corresponding to the source data. If so, the source data is formatted and then written into the ES model.

[0145] The device provided in the embodiment of the present application obtains serialized data from Kafka, determines the schema corresponding to the identification information carried by the serialized data; parses the serialized data based on the schema to obtain the source data, determines whether there is an ES model corresponding to the source data, and if so, converts the source data into a format and writes it into the ES model. Based on this solution, it is possible to obtain serialized data from Kafka in real time, and determine the corresponding schema based on the identification information, thereby parsing the serialized data based on the schema to obtain the source data, and converting the source data into a format and writing it into the corresponding ES model according to the mapping rules, thereby ensuring the consistency of the data format from data collection to data processing, and realizing real-time analysis and transformation of business data.

[0146] Optionally, the above device further includes a field expansion module, and the field expansion module is used to:

[0147] Determine whether the target data field exists in the source data. The target data field does not exist in the ES model.

[0148] If it exists, the ES model is expanded with the target data field.

[0149] Optionally, the above device further includes:

[0150] The abnormal data determination module is used to determine the source data as abnormal data when there is no ES model corresponding to the source data.

[0151] Optionally, the above device further includes:

[0152] The abnormal data retry module is used to determine whether there is an ES model corresponding to the abnormal data according to a preset retry strategy.

[0153] It can be understood that the above modules of the data writing device in this embodiment have the function of realizing Figure 3 The functions of the corresponding steps of the method for writing data in the embodiment shown in . The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented separately or integrated with multiple modules. For the functional description of each module of the above data writing device, please refer to Figure 3 The corresponding description of the method for writing data in the embodiment shown in is not repeated here.

[0154] An embodiment of the present application provides an electronic device, including a processor and a memory;

[0155] A memory, used for storing operation instructions;

[0156] The processor is used to execute the method provided in any embodiment of the present application by calling an operation instruction.

[0157] As an example, Figure 7 A schematic diagram of the structure of an electronic device applicable to an embodiment of the present application is shown. Figure 7 As shown, the electronic device 2000 includes: a processor 2001 and a memory 2003. The processor 2001 and the memory 2003 are connected, such as through a bus 2002. Optionally, the electronic device 2000 may also include a transceiver 2004. It should be noted that in actual applications, the transceiver 2004 is not limited to one, and the structure of the electronic device 2000 does not constitute a limitation on the embodiments of the present application.

[0158] The processor 2001 is applied in the embodiment of the present application to implement the method shown in the above method embodiment. The transceiver 2004 may include a receiver and a transmitter. The transceiver 2004 is applied in the embodiment of the present application to implement the function of the electronic device of the embodiment of the present application communicating with other devices when executed.

[0159] Processor 2001 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 2001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0160] The bus 2002 may include a path to transmit information between the above components. The bus 2002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 2002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0161] The memory 2003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0162] Optionally, the memory 2003 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 2001. The processor 2001 is used to execute the application code stored in the memory 2003 to implement the method provided in any embodiment of the present application.

[0163] The electronic device provided in the embodiment of the present application is applicable to any embodiment of the above method and will not be described in detail here.

[0164] The embodiment of the present application provides an electronic device, which, compared with the prior art, obtains serialized data from Kafka, determines the schema corresponding to the identification information carried by the serialized data; parses the serialized data based on the schema to obtain the source data, determines whether there is an ES model corresponding to the source data, and if so, converts the source data into a format and writes it into the ES model. Based on this solution, it is possible to obtain serialized data from Kafka in real time, and determine the corresponding schema based on the identification information, thereby parsing the serialized data based on the schema to obtain the source data, and converting the source data into a format and writing it into the corresponding ES model according to the mapping rules, thereby ensuring the consistency of the data format from data collection to data processing, and realizing real-time analysis and transformation of business data.

[0165] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method shown in the above method embodiment is implemented.

[0166] The computer-readable storage medium provided in the embodiments of the present application is applicable to any embodiment of the above method and will not be described in detail here.

[0167] The embodiment of the present application provides a computer-readable storage medium. Compared with the prior art, the serialized data is obtained from Kafka, and the schema corresponding to the identification information carried by the serialized data is determined; the serialized data is parsed based on the schema to obtain the source data, and it is determined whether there is an ES model corresponding to the source data. If so, the source data is formatted and written into the ES model. Based on this solution, serialized data can be obtained from Kafka in real time, and the corresponding schema can be determined based on the identification information, so that the serialized data can be parsed based on the schema to obtain the source data, and the source data is formatted and written into the corresponding ES model according to the mapping rules after format conversion, thereby ensuring the consistency of the data format from data collection to processing, and realizing real-time analysis and transformation of business data.

[0168] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0169] The above are only some embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for sending data, characterized in that: include: Get binary log data from the database of the monitored system; Parse the binlog data to obtain source data; Based on whether there is a schema matching the source data, serializing the source data to obtain serialized data, wherein the serialized data contains identification information of the schema; Publishing the serialized data to Kafka; Based on whether there is a schema matching the source data, the source data is serialized, including: Determine that a schema matching the source data exists in the local cache, wherein it is confirmed that the schema name and hash value in the cache are the same as those in the source data; If it exists, the source data is serialized using the schema; If it does not exist, register the schema and use the schema to serialize the source data.

2. The method according to claim 1, characterized in that The registering of the schema includes: Determine whether there is a registered schema with the same name as the schema; If it does not exist, register the schema; If it exists, determine whether the data structure corresponding to the registered schema is consistent with the data structure of the source data. If not, register the schema; if consistent, use the registered schema as the schema.

3. A method for writing data, characterized in that: include: Get serialized data from Kafka; Determine the schema corresponding to the identification information carried by the serialized data; Parsing the serialized data based on the schema to obtain source data; Determine whether there is a data copy ES model corresponding to the source data. If so, convert the format of the source data and write it into the ES model. Determine whether a target data field exists in the source data, and the target data field does not exist in the ES model; If it exists, the ES model is expanded with the target data field.

4. The method according to claim 3, characterized in that The method further comprises: If there is no ES model corresponding to the source data, the source data is determined to be abnormal data.

5. The method according to claim 4, characterized in that The method further comprises: According to the preset retry strategy, it is determined whether there is an ES model corresponding to the abnormal data.

6. A data transmission device, characterized in that: include: The data collection module is used to obtain binlog data from the database of the monitored system; A source data acquisition module is used to parse the binlog data and obtain source data; A serialization module, configured to serialize the source data to obtain serialized data based on whether there is a schema matching the source data, wherein the serialized data contains identification information of the schema, including determining that there is a schema matching the source data in a local cache, wherein it is confirmed that the schema name and hash value in the cache are the same as those of the source data; If it exists, the source data is serialized using the schema; If it does not exist, register the schema and use the schema to serialize the source data; The data publishing module is used to publish the serialized data to Kafka.

7. A data writing device, characterized in that: include: Data acquisition module, used to obtain serialized data from Kafka; A schema determination module, used to determine the schema corresponding to the identification information carried by the serialized data; A data parsing module, used to parse the serialized data based on the schema to obtain source data; a data writing module, used to determine whether there is an ES model corresponding to the source data, and if so, convert the format of the source data and write it into the ES model, and also includes determining whether there is a target data field in the source data, and the target data field does not exist in the ES model; If it exists, the ES model is expanded with the target data field.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store operation instructions; The processor is used to execute the method according to any one of claims 1 to 5 by calling the operation instruction.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images