Data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a storage medium, relates to the technical field of big data, and is used for ensuring that data cannot be lost during data acquisition and transmission. The method comprises the following steps: firstly, responding to a service data sending operation of a data sending end aiming at a target scene; determining a service data transmission mode set set by the target scene; and then, indicating the data sending end to transmit the service data according to the service data transmission mode set. Through the method, different service data transmission mode sets are set for different target scenes, lossless transmission of the data is realized, and it is ensured that the data cannot be lost in the collection and transmission process.

Description

Technical Field

[0001] This application mainly relates to the field of big data technology, and particularly relates to a data processing method, apparatus, electronic device, and storage medium. Background Art

[0002] With the rapid development of information technology, there has been an explosive growth in various information and data. The data types and sources corresponding to these data are diverse, and how to collect and store massive data has become the focus of research by all parties.

[0003] Due to the large number of data generation channels, large data volume, and diverse data formats, it is difficult to collect and use data. In the prior art, data loss is likely to occur during data collection and transmission, which will also cause inconvenience in subsequent data usage.

[0004] Therefore, how to ensure that data is not lost during data collection and transmission is an urgent problem to be solved currently. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and storage medium, which are used to ensure that data is not lost during data collection and transmission, and further improve the data interaction and transmission efficiency.

[0006] In a first aspect, this application provides a data processing method, including:

[0007] In response to a business data sending operation of a data sender in a target scenario, determining a set of business data transmission methods set for the target scenario; wherein, each business data transmission method included in the set of business data transmission methods is used for: lossless transmission of business data in the target scenario;

[0008] Instructing the data sender to transmit the business data according to the set of business data transmission methods.

[0009] In an optional implementation manner, in response to a business data sending operation of a data sender in a target scenario, determining a set of business data transmission methods set for the target scenario includes:

[0010] According to the attribute identifier of the business data, determining the data type of each business data; wherein, the business data carries an attribute identifier, and the attribute identifier is used to identify the data type of the business data;

[0011] Based on the data type and the target scenario, determining a set of business data transmission methods set for the target scenario.

[0012] Through the above method, a corresponding set of business data transmission methods can be set for different data types and target scenarios.

[0013] In an alternative embodiment, after determining the data type of each business data according to the attribute identifier of the business data, it further includes:

[0014] Obtain the data type of each business data;

[0015] If the data type of the business data is business interaction data, set a data synchronization operation policy for the business interaction data according to the target scenario;

[0016] If the data type of the business data is log data, perform a configuration file setting on the transmission path of the log data according to the business scenario, so that the log data can be losslessly transmitted in the target scenario.

[0017] Through the above method, performing a configuration file setting when collecting log data can adapt to different application scenarios and ensure that the business data transmission method during log data transmission according to the set configuration file can achieve lossless transmission.

[0018] In an alternative embodiment, if the business data is business interaction data, setting a data synchronization operation policy for the business interaction data includes:

[0019] Determine the storage form of the business interaction data based on the business type of the business interaction data;

[0020] Based on the storage form, set a data synchronization operation policy for the business interaction data; among them, the data synchronization operation policy is divided into: setting a full table, an incremental table, a new and changed table, and a special table.

[0021] Through the above method, setting different data synchronization operation policies can ensure that business interaction data will not be lost during collection and transmission.

[0022] In a second aspect, the present application provides a data processing device, including:

[0023] A determination module, configured to determine a set of business data transmission methods set for the target scenario in response to a business data sending operation of the data sending end in the target scenario; wherein, each business data transmission method included in the set of business data transmission methods is used for: lossless transmission of business data in the target scenario;

[0024] An indication module, configured to instruct the data sending end to transmit the business data according to the set of business data transmission methods.

[0025] In an alternative embodiment, when determining the set of service data transmission methods set for the target scenario in response to the service data sending operation of the data sending end for the target scenario, the determining module is specifically configured to:

[0026] Determine the data type of each service data according to the attribute identifier of the service data; wherein, the service data carries an attribute identifier, and the attribute identifier is used to identify the data type of the service data;

[0027] Based on the data type and the target scenario, determine the set of service data transmission methods set for the target scenario.

[0028] In an alternative embodiment, after determining the data type of each service data according to the attribute identifier of the service data, the determining module is further configured to:

[0029] Obtain the data type of each service data;

[0030] If the data type of the service data is service interaction data, set a data synchronization operation policy for the service interaction data according to the target scenario;

[0031] If the data type of the service data is log data, perform a configuration file setting on the transmission path of the log data according to the target scenario so that the log data is losslessly transmitted in the target scenario.

[0032] In an alternative embodiment, when, if the service data is service interaction data, setting a data synchronization operation policy for the service interaction data, the determining module is specifically configured to:

[0033] Determine the storage form of the service interaction data based on the service type of the service interaction data;

[0034] Based on the storage form, set a data synchronization operation policy for the service interaction data; wherein, the data synchronization operation policy is divided into: setting a full table, an incremental table, a new and changed table, and a special table.

[0035] In a third aspect, the present application provides an electronic device, including:

[0036] A memory for storing a computer program;

[0037] A processor for implementing the steps of the above-mentioned data processing method when executing the computer program stored in the memory.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0039] Through the technical solutions in one or more of the above embodiments of the present application, the embodiments of the present application have at least the following beneficial effects:

[0040] In the embodiments of the present application, first, in response to the operation of the data sending end for sending service data in a target scenario, a set of service data transmission methods set for the target scenario is determined, where each service data transmission method included in the set of service data transmission methods is used for lossless transmission of service data in the target scenario; then, the data sending end is instructed to transmit the service data according to the set of service data transmission methods. Through the above method, different sets of service data transmission methods are set for different application scenarios, and lossless transmission of service data in the target scenario can be realized according to the service data transmission methods, ensuring that data will not be lost during the process of acquisition and transmission, and guaranteeing the integrity of the data.

[0041] For the various aspects in the second to fourth aspects above and the possible technical effects that each aspect may achieve, please refer to the description of the possible technical effects that can be achieved in the first aspect and various possible solutions in the first aspect above, and will not be repeated here. Description of the Drawings

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. In the drawings:

[0043] Figure 1 It is a schematic diagram of an application scenario provided by the embodiments of the present application;

[0044] Figure 2 It is a schematic flowchart of an implementation process of a data processing method provided by the embodiments of the present application;

[0045] Figure 3 It is a schematic structural diagram of a data processing device provided by the embodiments of the present application;

[0046] Figure 4 It is a schematic structural diagram of an electronic device provided by the embodiments of the present application. Detailed Embodiments

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions of this application in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are part of the technical solutions of this application, rather than all of them. Based on the embodiments described in this application document, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope protected by the technical solutions of this application.

[0048] It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The connection between A and B can represent two situations: A is directly connected to B and A is connected to B through C. Additionally, in the description of this application, terms such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0049] The following explains and illustrates some of the technical solutions of the embodiments of this application to facilitate the understanding of those skilled in the art.

[0050] (1) Flume: It is a highly available, highly reliable, and distributed system for massive log collection, aggregation, and transmission provided by Cloudera. Flume supports customizing various data senders in the log system to collect data. At the same time, Flume provides the ability to perform simple processing on the data and write it to various data receivers.

[0051] (2) HDFS: It refers to a distributed file system designed to run on general-purpose hardware, which is a highly fault-tolerant system suitable for deployment on inexpensive machines.

[0052] (3) Sqoop: It is an open-source tool mainly used for data transfer between Hadoop (Hive) and traditional databases (such as mysql and postgresql). It can import data from a relational database (such as mysql, Oracle, etc.) into the HDFS of Hadoop, and can also import data from HDFS into a relational database.

[0053] Further, based on the above technical solutions for explanation, the following briefly introduces the design concept of the embodiments of this application. It should be noted that the application scenarios described below are only used to illustrate the embodiments of this application rather than to limit them. In specific implementations, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0054] In the prior art, due to the large number of data generation channels, large amounts of generated data, and diverse data formats, it is difficult to collect and use data. During data acquisition and transmission, data loss problems are likely to occur, which will also cause inconvenience in subsequent data usage.

[0055] In view of this, how to ensure that data is not lost during data acquisition and transmission is an urgent problem to be solved currently. Therefore, the embodiments of the present application provide a data processing method, which includes: First, in response to a business data sending operation of a data sending end for a target scenario, determine a set of business data transmission methods set for the target scenario; wherein, each business data transmission method included in the set of business data transmission methods is used for: lossless transmission of business data in the target scenario; Further, instruct the data sending end to transmit the business data according to the set of business data transmission methods.

[0056] Refer to Figure 1 shown, which is a schematic diagram of an application scenario applicable to the embodiments of the present application. As Figure 1 shown, this scenario mainly includes a terminal 10 and a server 11. Among them, information interaction can be carried out between the terminal 10 and the server 11 through a communication network, and the communication methods adopted by the communication network can include: wireless communication methods and wired communication methods.

[0057] In some scenarios, the terminal 10 can access the network through cellular mobile communication technology and communicate with the server 11. This cellular mobile communication technology can include the fifth generation mobile communication (5th Generation Mobile Networks, 5G) technology.

[0058] In some scenarios, the terminal 10 can access the network through short-range wireless communication methods and communicate with the server 11. This short-range wireless communication method can include wireless fidelity (Wireless Fidelity, Wi-Fi) technology.

[0059] The embodiments of the present application do not impose any restrictions on the number of the above devices. As Figure 1 shown, only the terminal 10 and the server 11 are taken as examples for description. Next, a brief introduction to the above devices and their respective functions will be given.

[0060] The terminal 10 is a device that can provide voice and / or data connectivity to users, including: handheld terminal devices with wireless connection functions, in-vehicle terminal devices, etc.

[0061] In some scenarios, the terminal device 10 includes, but is not limited to: mobile phones, tablet computers, laptop computers, palmtop computers, Mobile Internet Devices (MIDs), wearable devices, Virtual Reality (VR) devices, Augmented Reality (AR) devices, wireless terminal devices in industrial control, wireless terminal devices in driverless driving, wireless terminal devices in smart grids, wireless terminal devices in transportation safety, wireless terminal devices in smart cities, or wireless terminal devices in smart homes, etc.

[0062] In addition, a client related to data processing can be installed on the terminal 10. The client can be software (such as an APP, a browser, etc.), or a web page, a small program, etc. In the embodiment of the present application, the terminal 10 can use the above-mentioned client related to data processing and can perform information interaction related to the data processing scenario with the server 11.

[0063] Furthermore, the server 11 can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), as well as big data and artificial intelligence platforms.

[0064] Furthermore, in the embodiment of the present application, the above-mentioned server 11 can be equipped with a data processing device corresponding to the above-mentioned terminal 10, and this device is used to process various service data sent by the terminal 10.

[0065] It should be noted that the following describes the preferred embodiments of the present application with reference to the accompanying drawings of the specification. The preferred embodiments described here are only used to illustrate and explain the present application and are not used to limit the present application. And without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0066] Refer to Figure 2 As shown, it is a schematic diagram of the implementation process of a data processing method provided by the embodiment of the present application. The specific implementation process of this method is as follows:

[0067] S1: In response to the operation of sending service data by the data sending end for the target scenario, determine the set of service data transmission methods set for the target scenario.

[0068] S2: Instruct the data sending end to transmit the service data according to the set of service data transmission methods.

[0069] Among them, each business data transmission method in the set of business data transmission methods is used to losslessly transmit business data in the target scenario.

[0070] First of all, in response to the business data sending operation of the data sending end in the target scenario, determine the set of business data transmission methods set for the target scenario.

[0071] Specifically, when data collection is performed and the data sending end sends business data to the data receiving end to perform the business data sending operation, the data receiving end needs to determine the set of business data transmission methods set in the target scenario according to the target scenario.

[0072] In some embodiments, when determining the set of business data transmission methods set for the target scenario, it specifically includes:

[0073] First, according to the attribute identifier of the business data, determine the data type of each business data. It should be noted that the business data carries an attribute identifier, and the attribute identifier is used to identify the data type of the business data.

[0074] For example, the attribute identifier can be the file name suffix of the business data. Then, the business interaction data can be data with file name suffixes such as.JOSN and.XML, and the log data can be data with a file name suffix of.log. Of course, the attribute identifier of the business data can also be other identifiers that can distinguish data types, and the embodiments of the present application do not make specific limitations here.

[0075] Further, after determining the data type of each business data according to the attribute identifier of the business data, it may further include: based on the data type of each business data, configure different transmission methods for business data of different data types.

[0076] In the embodiments of the present application, if the data type of the business data is business interaction data, set a data synchronization operation for the business interaction data according to the target scenario.

[0077] In the embodiments of the present application, the collected business interaction data uses sqoop as an important component for transmission. After storing the business interaction data in the mysql database, use sqoop to transfer the business interaction data from the mysql database to the HDFS database. When using sqoop to transfer the business interaction data, it is necessary to set a data synchronization operation strategy to achieve lossless transmission of the business interaction data in the target scenario.

[0078] Specifically, when setting the data synchronization operation strategy, it is necessary to determine the storage form of the business interaction data according to the business type of the business interaction data, and then set different data synchronization operation strategies according to the storage form of the business interaction data.

[0079] In the embodiments of the present application, according to different storage forms of service interaction data, a full table, an incremental table, a new and changed table, and a special table can be set, where:

[0080] The full table is used to store the full amount of data within a period. Whether the data has changed or not needs to be recorded. For daily full amounts, a complete set of data is stored every day as a partition. It is used in business scenarios where the data size of the table is not large, new data is added every day, and old data is modified.

[0081] The incremental table is used to record the newly added data within the storage period, that is, on the basis of the original table data, the new data generated within this period is newly added, and the unchanged data will not be recorded. For daily increments, an incremental data is stored every day as a partition, which is suitable for business scenarios where the table data is large and only new data is added every day.

[0082] The new and changed table is used to store the newly added data and the changed data. For daily new and changed data, the data whose creation time and operation time are both today is stored, which is suitable for business scenarios where the table data volume is large and new data is added and changed every day.

[0083] The special table is used to record some objective data without changes, which generally only needs to be stored once, such as gender, ethnicity, region, etc.

[0084] By setting different data synchronization strategies, lossless transmission of service interaction data can be achieved, ensuring the integrity of business data.

[0085] In the embodiments of the present application, if the data type of the business data is log data, the transmission path of the log data is set in the configuration file according to the target scenario, so that the log data is losslessly transmitted in the target scenario.

[0086] Specifically, in the embodiments of the present application, when collecting log data, the Flume log system can be used to collect log data. Specifically, the Flume log collection system uses the Source data collector to collect log data from the Logfile log folder in the web server. Further, the collected log data is transmitted to the Sink layer of the Flume log collection system through the channel of the Flume log collection system, and finally the log data is stored in a database such as HDFS.

[0087] In the embodiments of the present application, when using the Flume log system to collect log data, corresponding configuration file settings can be made according to different target scenarios.

[0088] For example, the Channel is used to store and transmit Events. In the Flume logging system, various types of Channels can be used to achieve different functions. For example, the Memory Channel, the File Channel, etc.

[0089] The Sink layer is used to define the destination of the data, such as HDFS, Kafka, etc. By configuring the Sink layer, for example, using the Sink component of the logger type, the Event is output to the standard output stream.

[0090] Therefore, in the embodiments of the present application, different Channel configuration files, Sink configuration files, Source configuration files, etc. can be set according to different target scenarios. By setting different configuration files, the complete circulation of data in different scenarios can be realized, and different business data transmission methods can also be obtained by setting different configuration files.

[0091] In the Flume logging system, there are various configuration instructions that can be applied to different business scenarios. Therefore, configuration instructions can be added according to different target scenarios to ensure lossless transmission when the data sender sends data to the data receiver. How to specifically set the configuration instructions or configuration files will not be elaborated here.

[0092] For example, during the development of the Internet, a large number of unstable factors have emerged, such as a large amount of spam information and a large number of network attacks. Therefore, in order to reduce the losses suffered from network attacks, it is necessary to do a good job in network monitoring to facilitate the timely discovery of spam and attack information in the network. This requires the timely and correct acquisition of a large amount of monitoring data. Therefore, for the network attack scenario, different configuration files can be set for the Flume logging collection system, or different data synchronization strategies can be set for the business interaction data to form a set of business data transmission methods.

[0093] After determining the set of business data transmission methods according to the target scenario and the business data type, the data receiver instructs the data sender to transmit the business data according to the set of business data sending methods.

[0094] Through the above method, a corresponding set of business data transmission methods is set for different application scenarios. Each business data transmission method in the set of business data transmission methods is used to achieve lossless transmission of business data in the target scenario, ensuring that data will not be lost during the data collection and transmission process.

[0095] Based on the same inventive concept, the embodiments of the present application also include a data processing device. Refer to Figure 3 As shown, the device includes: a determination module 301 and an indication module 302, where

[0096] A determination module 301, configured to determine a set of service data transmission methods set for a target scenario in response to a service data sending operation of a data sender for the target scenario; wherein, each service data transmission method included in the set of service data transmission methods is used for: lossless transmission of service data in the target scenario;

[0097] An indication module 302, configured to instruct the data sender to transmit the service data according to the set of service data transmission methods.

[0098] In an optional implementation manner, when determining the set of service data transmission methods set for the target scenario in response to the service data sending operation of the data sender for the target scenario, the determination module 301 is specifically configured to:

[0099] Determine the data type of each service data according to the attribute identifier of the service data; wherein, the service data carries an attribute identifier, and the attribute identifier is used to identify the data type of the service data;

[0100] Based on the data type and the target scenario, determine the set of service data transmission methods set for the target scenario.

[0101] In an optional implementation manner, after determining the data type of each service data according to the attribute identifier of the service data, the determination module 301 is further configured to:

[0102] Obtain the data type of each service data;

[0103] If the data type of the service data is service interaction data, set a data synchronization policy for the service interaction data according to the target scenario;

[0104] If the data type of the service data is log data, perform a configuration file setting on the transmission path of the log data according to the target scenario, so that the log data is losslessly transmitted in the target scenario.

[0105] In an optional implementation manner, when, if the service data is service interaction data, setting a data synchronization operation policy for the service interaction data, the determination module 301 is specifically configured to:

[0106] Determine the storage form of the service interaction data based on the service type of the service interaction data;

[0107] Based on the storage form, set a data synchronization operation policy for the service interaction data; wherein, the data synchronization operation policy is divided into: setting a full table, an incremental table, a new and changed table, and a special table.

[0108] Based on the same inventive concept, an embodiment of the present application further provides an electronic device, which can implement the functions of the foregoing data processing method. Referring to Figure 4 as shown, the electronic device includes:

[0109] At least one processor 401 and a memory 402 connected to the at least one processor 401. In the embodiments of the present application, the specific connection medium between the processor 401 and the memory 402 is not limited. Figure 4 In this example, the processor 401 and the memory 402 are connected through a bus 400. The bus 400 is Figure 4 represented by a thick line in this example. The connection manners between other components are only for illustrative purposes and are not limiting. The bus 400 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 4 only a thick line is used to represent it in this example, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 401 can also be referred to as a controller, and the name is not limited.

[0110] In the embodiments of the present application, the memory 402 stores instructions executable by the at least one processor 401. By executing the instructions stored in the memory 402, the at least one processor 401 can execute the data processing method described above. The processor 401 can implement Figure 3 the functions of each module in the device shown.

[0111] Among them, the processor 401 is the control center of the device, and can connect various parts of the entire control device through various interfaces and lines. By running or executing the instructions stored in the memory 402 and calling the data stored in the memory 402, various functions of the device and process data, so as to monitor the device as a whole.

[0112] In a possible design, the processor 401 may include one or more processing units. The processor 401 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above modem processor may not be integrated into the processor 401. In some embodiments, the processor 401 and the memory 402 can be implemented on the same chip, and in some embodiments, they can also be separately implemented on independent chips.

[0113] The processor 401 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application specific integrated circuit, a field programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the data processing method disclosed in combination with the embodiments of the present application may be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0114] The memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 402 may include at least one type of storage medium, for example, it may include flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disc, and so on. The memory 402 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 402 in the embodiments of the present application may also be a circuit or any other device capable of implementing a storage function, for storing program instructions and / or data.

[0115] By designing and programming the processor 401, the code corresponding to the data processing method described in the foregoing embodiments can be solidified into the chip, so that the chip can execute Figure 1 the steps of the data processing method of the illustrated embodiments when running. How to design and program the processor 401 is a well-known technology to those skilled in the art and will not be elaborated here.

[0116] Based on the same inventive concept, the embodiments of the present application also provide a storage medium that stores computer instructions, and when the computer instructions run on a computer, the computer is caused to execute the data processing method discussed above.

[0117] In some possible embodiments, various aspects of the data processing method provided in this application can also be implemented in the form of a program product, which includes program code. When the program product runs on a device, the program code is used to cause the control device to execute the steps in the data processing method according to various exemplary embodiments of this application described above in this specification.

[0118] Those skilled in the art should understand that the embodiments of this application can be provided as a method, a system, or a computer program product. Therefore, this application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0119] This application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of this application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one or more of the processes or multiple processes and / or blocks Figure 1 one or more of the blocks or multiple blocks.

[0120] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one or more of the processes or multiple processes and / or blocks Figure 1 one or more of the blocks or multiple blocks.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more of the processes or multiple processes and / or blocks Figure 1 one or more of the blocks or multiple blocks.

[0122] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to cover these changes and modifications.

Claims

1. A data processing method, characterized in that, the method includes: responding to a business data sending operation of a data sending end for a target scenario, and determining a set of business data transmission methods set for the target scenario; wherein, each business data transmission method included in the set of business data transmission methods is used for: lossless transmission of business data in the target scenario; instructing the data sending end to transmit the business data according to the set of business data transmission methods.

2. The method according to claim 1, characterized in that, the responding to a business data sending operation of a data sending end for a target scenario, and determining a set of business data transmission methods set for the target scenario includes: determining the data type of each business data according to the attribute identifier of the business data; wherein, the business data carries the attribute identifier, and the attribute identifier is used to identify the data type of the business data; determining a set of business data transmission methods set for the target scenario based on the data type and the target scenario.

3. The method according to claim 2, characterized in that, after determining the data type of each business data according to the attribute identifier of the business data, it further includes: obtaining the data type of each business data; if the data type of the business data is business interaction data, setting a data synchronization operation policy for the business interaction data according to the target scenario; if the data type of the business data is log data, performing a configuration file setting on the transmission path of the log data according to the target scenario, so that the log data is losslessly transmitted in the target scenario.

4. The method according to claim 3, characterized in that, the setting a data synchronization operation policy for the business interaction data if the business data is business interaction data includes: determining the storage form of the business interaction data based on the business type of the business interaction data; setting a data synchronization operation policy for the business interaction data based on the storage form; wherein, the data synchronization operation policy is divided into: setting a full table, an incremental table, a new and changed table, and a special table.

5. A data processing device, characterized in that, the device includes: a determination module, configured to respond to a business data sending operation of a data sending end for a target scenario, and determine a set of business data transmission methods set for the target scenario; wherein, each business data transmission method included in the set of business data transmission methods is used for: lossless transmission of business data in the target scenario; an instruction module, configured to instruct the data sending end to transmit the business data according to the set of business data transmission methods.

6. The device according to claim 5, characterized in that, when responding to a business data sending operation of a data sending end for a target scenario, and determining a set of business data transmission methods set for the target scenario, the determination module is specifically configured to: Determine the data types of each piece of the service data according to the attribute identifier of the service data; wherein, the service data carries the attribute identifier, and the attribute identifier is used to identify the data type of the service data. Based on the data type and the target scenario, determine the set of service data transmission methods set for the target scenario.

7. The apparatus according to claim 6, characterized in that after determining the data types of each piece of the service data according to the attribute identifier of the service data, the determining module is further configured to: obtain the data types of each piece of the service data; if the data type of the service data is service interaction data, set a data synchronization operation policy for the service interaction data according to the target scenario; if the data type of the service data is log data, perform a configuration file setting on the transmission path of the log data according to the target scenario, so that the log data is losslessly transmitted in the target scenario.

8. The apparatus according to claim 7, characterized in that when setting a data synchronization operation policy for the service interaction data if the service data is service interaction data, the determining module specifically is configured to: determine the storage form of the service interaction data based on the service type of the service interaction data; set a data synchronization operation policy for the service interaction data based on the storage form; wherein, the data synchronization operation policy is divided into: setting a full table, an incremental table, a new and changed table, and a special table.

9. An electronic device, characterized in that comprising: a memory for storing a computer program; a processor for implementing the method according to any one of claims 1-4 when executing the computer program stored on the memory.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and the computer program realizes the method according to any one of claims 1-4 when executed by a processor.

Images