Data processing method and device, medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a medium and electronic equipment, and the method comprises the steps: segmenting target business data into at least two data fragments in response to a detected data uploading operation of the target business data; wherein each data fragment comprises at least two data items; executing a mapping operation based on the data type to which the data item belongs through a data mapping node, and converting the data item into a key value pair; determining a data mapping result corresponding to the data fragment based on the key value pair corresponding to the data item through the data mapping node, and transmitting the data mapping result to a data reduction node; and performing reduction operation on the data mapping result corresponding to each data fragment of the key pair in the key value pair through the data reduction node so that the key in the key value pair only corresponds to one value, and obtaining a data processing result. By executing the technical scheme, the data processing efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence technology, big data and cloud computing technology, and can be applicable to financial technology scenarios. The present application specifically relates to a data processing method, device, medium and electronic device. Background Art

[0002] Currently, commercial banks generally perform centralized data processing on physical machines. On the one hand, centralized data processing requires physical machines to remain powered on for a long time, even when not processing data, which results in a waste of computing resources. On the other hand, centralized data processing concentrates data processing operations on a central physical machine. In large-scale data processing scenarios, it is easy to form performance bottlenecks due to excessive data volume, resulting in low data processing efficiency. Summary of the invention

[0003] The present application provides a data processing method, device, medium and electronic device, which can improve the efficiency of data processing and improve the resource utilization of data processing.

[0004] According to a first aspect of the present application, a data processing method is provided, the method comprising:

[0005] In response to detecting a data upload operation of target business data, the target business data is divided into at least two data slices; wherein each data slice includes at least two data items;

[0006] Performing a mapping operation based on the data type to which the data item belongs through a data mapping node to convert the data item into a key-value pair;

[0007] Determine, by the data mapping node, a data mapping result corresponding to the data shard based on the key-value pair corresponding to the data item, and transmit the data mapping result to the data reduction node;

[0008] The data reduction node performs a reduction operation on the data mapping result corresponding to each data shard on the key in the key-value pair so that the key in the key-value pair corresponds to only one value, thereby obtaining a data processing result.

[0009] According to a second aspect of the present application, a data processing device is provided, the device comprising:

[0010] A data splitting module, configured to split the target business data into at least two data slices in response to detecting a data upload operation of the target business data; wherein each data slice includes at least two data items;

[0011] A data item conversion module, used to perform a mapping operation based on the data type to which the data item belongs through a data mapping node, and convert the data item into a key-value pair;

[0012] A mapping result determination module, configured to determine, through the data mapping node, a data mapping result corresponding to the data shard based on a key-value pair corresponding to the data item, and transmit the data mapping result to a data reduction node;

[0013] The first result determination module is used to perform a reduction operation on the data mapping result corresponding to each data shard on the key in the key-value pair through the data reduction node so that the key in the key-value pair corresponds to only one value, thereby obtaining a data processing result.

[0014] According to a third aspect of the present invention, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the data processing method described in the embodiment of the present application is implemented.

[0015] According to the fourth aspect of the present invention, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data processing method described in the embodiment of the present application is implemented.

[0016] According to a fifth aspect of the present application, an embodiment of the present application provides a computer program product, including a computer program, which implements the data processing method as described in the embodiment of the present application when executed by a processor.

[0017] The technical solution of the present application uses a distributed data processing method to process target business data, and can dynamically allocate computing resources according to actual needs, thereby improving the resource utilization of data processing. The embodiment of the present application divides the target business data into at least two data slices, and distributes the data slices to different data mapping nodes for parallel processing, which is conducive to improving data processing efficiency. The embodiment of the present application performs mapping operations and reduction operations in sequence through data mapping nodes and data reduction nodes to obtain data processing results, supports expanding the data processing system by adding data mapping nodes and data reduction nodes, and improves the scalability of data processing.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 is a structural diagram of a data processing system provided according to an embodiment;

[0021] Figure 2 is a flow chart of a data processing method provided according to Embodiment 1;

[0022] Figure 3 is a flow chart of a data processing method provided according to Embodiment 2;

[0023] Figure 4 is a structural diagram of a data processing device provided in Embodiment 3 of the present application;

[0024] Figure 5 It is a structural schematic diagram of an electronic device provided in Example 4 of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", "target" and "candidate" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Embodiment 1

[0028] Figure 1 is a structural diagram of a data processing system provided according to an embodiment. Figure 1 The data processing system shown is used to execute the data processing method provided in the embodiment of the present application, see Figure 1 The data processing system includes a job tracking node (JobTracker), a task tracking node (Task Tracker), a data mapping node (map task) and a data reduction node (reduce task). Among them, the task tracking node, the data mapping node and the data reduction node are functional nodes, and the job tracking node is a non-functional node. The task tracking node, the data mapping node and the data reduction node implement the corresponding data processing functions by calling the FaaS function (Function as a Service). Among them, the data mapping node is used to convert the data items in the data shard into key-value pairs; the data reduction node is used to reduce the key-value pairs converted by the data mapping node so that the key in the key-value pair corresponds to only one value.

[0029] The task tracking node is responsible for detecting the job execution status of the data mapping nodes and data reduction nodes mounted on it, and assigning jobs according to the job execution status of each node. There can be multiple task tracking nodes on a physical machine. Each task tracking node has a fixed number of slots for mounting data mapping nodes and data reduction nodes. The task tracking node automatically destroys or creates new data mapping nodes and data reduction nodes according to the job execution status of the data mapping nodes and data reduction nodes. Figure 1 The fact that two data mapping nodes and one data reduction node are mounted under one task tracking node is only an example and does not limit the number of data mapping nodes and data reduction nodes that can be mounted under the task tracking node.

[0030] The task tracking node will also report its heartbeat data (Heatbeat) to the job tracking node. The heartbeat data can reflect the job execution status of the task tracking node. Once a task tracking node goes down, the job tracking node can generate another task tracking node on the functional platform and let the new task tracking node continue to execute the tasks of the down task tracking node based on the heartbeat data of the original task tracking node.

[0031] Figure 2 It is a flowchart of the data processing method provided according to the first embodiment. This embodiment can be applicable to the case of processing data corresponding to non-real-time services. The method can be configured to be executed by a data processing device. The data processing device is implemented in the form of hardware and / or software and can be integrated into an electronic device running this data processing system.

[0032] like Figure 2 As shown, the method includes:

[0033] S210. In response to detecting a data upload operation of target business data, the target business data is divided into at least two data slices; wherein each data slice includes at least two data items.

[0034] S220. Perform a mapping operation based on the data type to which the data item belongs through a data mapping node to convert the data item into a key-value pair.

[0035] S230. Determine, by the data mapping node, a data mapping result corresponding to the data shard based on the key-value pair corresponding to the data item, and transmit the data mapping result to a data reduction node.

[0036] S240. Perform a reduction operation on the data mapping result corresponding to each data shard on the key in the key-value pair through the data reduction node so that the key in the key-value pair corresponds to only one value, thereby obtaining a data processing result.

[0037] The target business data refers to the data to be processed. The data upload operation of the target business data will trigger the data processing execution. The target business data can be uploaded regularly or on demand. Optionally, the target business data is the data corresponding to the non-real-time business. Exemplarily, the non-real-time business can be the background business that can be executed offline or in a low-priority period, such as financial report generation, credit risk assessment, data warehouse maintenance, data backup and archiving.

[0038] The target business data is divided into at least two data shards. The target business data is sent to the data mapping node according to the granularity of the data shards. Optionally, the target business data is sharded according to the number of data mapping nodes so that each data shard has a corresponding data mapping node.

[0039] Each data shard includes at least two data items. The data mapping node reads the data shard, performs a mapping operation based on the data type of the data item in the data shard, and converts the data item into a key-value pair. Optionally, the data mapping node is a functional node that processes the data shard by calling a FaaS function and converts the data item into a key-value pair.

[0040] The data types of data items include original data and reduction results. Original data refers to target business data that has only been processed by aggregation. Original data has not been processed in other ways. When the target business data is the data corresponding to the credit risk assessment, the original data is aggregated according to the identification field and will not be disorganized.

[0041] The mapping logic corresponding to the mapping operation is different for different data types of data items. The key and value in the key-value pair are determined according to actual business needs, and the key-value pair should be able to describe the corresponding relationship between the unique identifier and the business data.

[0042] A data item has at least one key-value pair. The key-value pairs corresponding to each data item in the data shard are integrated to obtain the data mapping result corresponding to the data shard. Optionally, the data mapping node uses a cloud storage service to temporarily store the data mapping result. The data mapping node sends the data mapping result to the data reduction node, and the data mapping result can be directly processed by the data reduction node without format conversion.

[0043] Optionally, the data reduction node is a functional node, which performs a reduction operation on the data mapping results corresponding to the key in the key-value pair for each data shard by calling a FaaS function. The reduction operation refers to aggregating, counting or further processing the key-value pairs in the data mapping results. Optionally, the further processing may be to process the key-value pairs in the data mapping results using a preset business logic to obtain business indicators corresponding to the preset business logic.

[0044] The output of the data reduction node is the data processing result, which is also in the form of a key-value pair. In the data processing result, the key of the key-value pair corresponds to only one value.

[0045] The technical solution of the present application uses a distributed data processing method to process target business data, and can dynamically allocate computing resources according to actual needs, thereby improving the resource utilization of data processing. The embodiment of the present application divides the target business data into at least two data slices, and distributes the data slices to different data mapping nodes for parallel processing, which is conducive to improving data processing efficiency. The embodiment of the present application performs mapping operations and reduction operations in sequence through data mapping nodes and data reduction nodes to obtain data processing results, supports expanding the data processing system by adding data mapping nodes and data reduction nodes, and improves the scalability of data processing.

[0046] In an optional embodiment, the data reduction node performs a reduction operation on the data mapping results corresponding to the key in the key-value pair for each data shard so that the key in the key-value pair corresponds to only one value, thereby obtaining a data processing result, including: determining, by the data reduction node, the value corresponding to the key in the key-value pair in the data mapping results corresponding to each data shard; if there are at least two values ​​corresponding to the key in the key-value pair, reducing the value corresponding to the key in the key-value pair based on a preset business logic by the data reduction node to obtain a business indicator corresponding to the preset business logic; using the business indicator as a value, updating the key-value pair so that the key in the key-value pair corresponds to only one value, thereby obtaining a data processing result.

[0047] The data reduction node needs to perform a unique record check on the data mapping result to ensure that the key-value pair in the data processing result corresponds to only one value. The data reduction node performs comprehensive statistics on the key-value pairs in the data mapping results corresponding to each data shard, and determines the value corresponding to the key in the key-value pair. If there are at least two values ​​corresponding to the key in the key-value pair. The data reduction node needs to further process the key-value pair. Optionally, a preset business logic is deployed in the data reduction node. The data reduction node uses the preset business logic to reduce the value corresponding to the key in the key-value pair to obtain the business indicator corresponding to the preset business logic. Among them, the preset business logic and the business indicator corresponding to the preset business logic are determined according to actual business needs, and their specific contents are not limited here. The target business data, the preset business logic and the business indicators are all related to the business. Exemplarily, the target business data is the data corresponding to the credit risk assessment, and the preset business logic can be to determine the credit score using the business data related to the credit risk assessment. Among them, the credit score is the preset business indicator corresponding to the business logic.

[0048] In this way, at least two values ​​corresponding to the key can be reduced to one business indicator. The business indicator is used as the value to update the key-value pair to ensure that the key remains unchanged, and it can be ensured that the key in the key-value pair corresponds to only one value. Optionally, after the data reduction node updates the key-value pair, it adjusts the format of the key-value pair and determines the data processing result based on the format-adjusted key-value pair. In this way, the format of the key-value pair corresponding to the data processing result can be ensured to be consistent.

[0049] The above technical solution uses a data reduction node to check the uniqueness of the data mapping result. When there are at least two values ​​corresponding to the key in the key-value pair, the preset business logic is used to reduce the value corresponding to the key in the key-value pair, ensuring that the key in the key-value pair corresponds to only one value. Reduction by using the preset business logic can flexibly handle various complex data situations and meet the needs of different business scenarios.

[0050] In an optional embodiment, the method further includes: if there is only one value corresponding to the key in the key-value pair, taking the key-value pair as the data processing result.

[0051] If there is only one value corresponding to the key in the key-value pair, the data reduction node does not need to reduce the value corresponding to the key in the key-value pair, and directly uses the key-value pair as the data processing result.

[0052] The above technical solution proposes a corresponding data processing method for the case where there is only one value corresponding to the key in the key-value pair, thereby ensuring the availability of the data processing method.

[0053] In an optional embodiment, the target service data is data corresponding to a non-real-time service; and the data mapping node and the data reduction node are functionalized nodes.

[0054] The above technical solution uses functionalized data mapping nodes and data reduction nodes to process the data corresponding to non-real-time services, realizes distributed data processing, supports expanding the data processing system by adding data mapping nodes and data reduction nodes, improves the scalability of data processing, and is conducive to improving data processing efficiency.

[0055] Embodiment 2

[0056] Figure 3 It is a flow chart of the data processing method provided according to the second embodiment. This embodiment is further optimized on the basis of the above embodiment.

[0057] like Figure 3 As shown, the method includes:

[0058] S310. In response to detecting a data upload operation of target business data, the target business data is divided into at least two data slices; wherein each data slice includes at least two data items.

[0059] S320. Determine, by using the data mapping node, whether the data item includes a preset field, and determine the data type to which the data item belongs.

[0060] S330. If the data type is raw data, extract the identification field and the characteristic field from the data item, use the identification content corresponding to the identification field as the key, map the business data corresponding to the characteristic field as the value to the key, and convert the data item into a key-value pair; wherein the raw data refers to the target business data that has only been aggregated.

[0061] The original data refers to the target business data that has only been aggregated. The original data has not been processed in other ways. When the target business data is the data corresponding to the credit risk assessment, the original data may be, for example, the user ID, related expenses, time since the last consumption, and consumption purpose.

[0062] The data items in the original data include at least two fields, for example, an identification field and a characteristic field. The specific contents of the identification field and the characteristic field are determined according to actual business needs and are not limited here. Exemplarily, the identification field can be a user ID, and the characteristic field can be related expenses, time since last consumption, and purpose of consumption, etc. Among them, the identification field is unique, and the field content corresponding to the identification field, such as the user ID, is used as the key, and the related expenses, time since last consumption, and purpose of consumption are mapped to the key as the value. Convert the data item into a key-value pair. The key-value pair is used to describe the correspondence between the unique identifier and the business data.

[0063] S340: If the data type is a reduction result, adjust the value of the key-value pair corresponding to the data item, and update the key-value pair corresponding to the data item according to the value adjustment result.

[0064] The reduction result is generated by the data reduction node. The reduction result is not the final data processing result. The reduction result needs to be further adjusted by the data mapping node. The data items included in the reduction result are also in the form of key-value pairs. Optionally, the value of the key-value pair corresponding to the data item is adjusted, including updating the weight of the value in the key-value pair, or adding a new value to the key-value pair.

[0065] The data mapping node updates the key-value pair corresponding to the data item using the value adjustment result.

[0066] S350. Determine, through the data mapping node, a data mapping result corresponding to the data shard based on the key-value pair corresponding to the data item, and transmit the data mapping result to a data reduction node.

[0067] The data mapping node summarizes the key-value pairs corresponding to each data item and obtains the data mapping results corresponding to the data shards. The data mapping node transmits the data mapping results to the data reduction node.

[0068] In an optional embodiment, the data mapping result corresponding to the data shard is determined by the data mapping node based on the key-value pair corresponding to the data item, including: formatting the key-value pair corresponding to the data item based on a preset format by the data mapping node to unify the data format adopted by the key-value pair; and determining the data mapping result corresponding to the data shard based on the key-value pair after the format is unified.

[0069] The preset format is determined according to actual business needs, and its specific content is not limited here. After the data mapping node converts the data item into a key-value pair, the data mapping node adjusts the format of the key-value pair corresponding to the data item based on the preset format, unifies the data format of the key-value pair, and enables the data reduction node to directly process it.

[0070] The data mapping node determines the data mapping result corresponding to the data shard based on the key-value pairs in a unified format.

[0071] In the above technical solution, after converting the data item into a key-value pair, the data mapping node adjusts the key-value pair corresponding to the data item based on a preset format so that the key-value pair can be directly processed by the data reduction node, which is conducive to improving data processing efficiency.

[0072] S360. Perform a reduction operation on the data mapping result corresponding to each data shard on the key in the key-value pair through the data reduction node so that the key in the key-value pair corresponds to only one value, thereby obtaining a data processing result.

[0073] The key-value pairs converted by the data mapping node can be identified as (id, feature). Among them, id is the key and feature is the value. The data reduction node performs a reduction operation on several key-value pairs (id, feature) and outputs (id, score). If one id corresponds to at least two (id, score), then (id, score) needs to be further processed by the data mapping node as the reduction result. Specifically, the data mapping node converts (id, score) into a new (id, feature), and the data reduction node continues to reduce the new (id, feature). Repeat the above steps until one id corresponds to only one (id, score) output data processing result.

[0074] The technical solution of the present application determines whether the data item includes preset fields through the data mapping node, and determines the data type to which the data item belongs. If the data type is raw data, the identification field and the characteristic field are extracted from the data item, the identification content corresponding to the identification field is used as the key, and the business data corresponding to the characteristic field is mapped to the key as the value, and the data item is converted into a key-value pair; if the data type is a reduction result, the value of the key-value pair corresponding to the data item is adjusted, and the key-value pair corresponding to the data item is updated according to the value adjustment result. The present application adopts a corresponding processing method for data items of different data types through the data mapping node to convert the data item into a key-value pair to obtain a data mapping result, and provides a feasible key-value pair determination scheme, which provides data support for the subsequent data reduction node to perform a reduction operation on the data mapping result corresponding to each data shard of the key in the key-value pair so that the key in the key-value pair corresponds to only one value to obtain a data processing result.

[0075] Embodiment 3

[0076] Figure 4 This is a structural diagram of the data processing device provided in Example 3 of the present application. This embodiment can be applied to the situation of processing data corresponding to non-real-time services. The device can be implemented by software and / or hardware and can be integrated into electronic devices such as smart terminals.

[0077] like Figure 4 As shown, the data processing device 400 may include:

[0078] The data segmentation module 410 is used to segment the target business data into at least two data segments in response to detecting a data upload operation of the target business data; wherein each data segment includes at least two data items;

[0079] A data item conversion module 420, configured to perform a mapping operation based on the data type to which the data item belongs through a data mapping node, and convert the data item into a key-value pair;

[0080] A mapping result determination module 430, configured to determine, through the data mapping node, a data mapping result corresponding to the data shard based on the key-value pair corresponding to the data item, and transmit the data mapping result to a data reduction node;

[0081] The first result determination module 440 is used to perform a reduction operation on the data mapping results corresponding to the keys in the key-value pairs for each data shard through the data reduction node so that the key in the key-value pair corresponds to only one value, thereby obtaining a data processing result.

[0082] The technical solution of the present application uses a distributed data processing method to process target business data, and can dynamically allocate computing resources according to actual needs, thereby improving the resource utilization of data processing. The embodiment of the present application divides the target business data into at least two data slices, and distributes the data slices to different data mapping nodes for parallel processing, which is conducive to improving data processing efficiency. The embodiment of the present application performs mapping operations and reduction operations in sequence through data mapping nodes and data reduction nodes to obtain data processing results, supports expanding the data processing system by adding data mapping nodes and data reduction nodes, and improves the scalability of data processing.

[0083] Optionally, the data item conversion module 420 includes: a data type determination submodule, which is used to determine whether the data item includes a preset field through the data mapping node, and determine the data type to which the data item belongs; a first data item conversion submodule, which is used to extract an identification field and a feature field from the data item if the data type is raw data, use the identification content corresponding to the identification field as a key, and map the business data corresponding to the feature field as a value to the key, and convert the data item into a key-value pair; wherein the raw data refers to the target business data that has only been aggregated; and a second data item conversion submodule, which is used to adjust the value of the key-value pair corresponding to the data item if the data type is a reduction result, and update the key-value pair corresponding to the data item according to the value adjustment result.

[0084] Optionally, the mapping result determination module 430 includes: a format adjustment submodule, used to perform format adjustment on the key-value pairs corresponding to the data items based on a preset format through the data mapping node to unify the data format adopted by the key-value pairs; a mapping result determination submodule, used to determine the data mapping results corresponding to the data shards based on the key-value pairs after format unification.

[0085] Optionally, the reduction result determination module 440 includes: a value determination submodule, used to determine the value corresponding to the key in the key-value pair in the data mapping result corresponding to each data shard through the data reduction node; a value reduction submodule, used to reduce the value corresponding to the key in the key-value pair based on a preset business logic through the data reduction node if there are at least two values ​​corresponding to the key in the key-value pair, so as to obtain a business indicator corresponding to the preset business logic; a processing result determination submodule, used to update the key-value pair using the business indicator as the value of the reduction result so that the key in the key-value pair corresponds to only one value, so as to obtain a data processing result.

[0086] Optionally, the device 400 further includes a second result determination module, wherein the second result determination module is configured to use the key-value pair as the data processing result if there is only one value corresponding to the key in the key-value pair.

[0087] Optionally, the target business data is data corresponding to a non-real-time business; and the data mapping node and the data reduction node are functionalized nodes.

[0088] The data processing device provided in the embodiment of the invention can execute the data processing method provided in any embodiment of the present application, and has the corresponding performance modules and beneficial effects for executing the data processing method.

[0089] In the technical solution of this application, the target business data involved are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0090] Embodiment 4

[0091] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0092] Figure 5The schematic diagram of the structure of the electronic device 510 of the embodiment that can be used to implement is shown. The electronic device 510 includes at least one processor 511, and a memory connected to the at least one processor 511 in communication, such as a read-only memory (ROM) 512, a random access memory (RAM) 513, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 511 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 512 or the computer program loaded from the storage unit 518 to the random access memory (RAM) 513. In the RAM513, various programs and data required for the operation of the electronic device 510 can also be stored. The processor 511, the ROM 512 and the RAM513 are connected to each other via a bus 514. An input / output (I / O) interface 515 is also connected to the bus 514.

[0093] A number of components in the electronic device 510 are connected to the I / O interface 515, including: an input unit 516, such as a keyboard, a mouse, etc.; an output unit 517, such as various types of displays, speakers, etc.; a storage unit 518, such as a disk, an optical disk, etc.; and a communication unit 519, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 519 allows the electronic device 510 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0094] The processor 511 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 511 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 511 performs the various methods and processes described above, such as a data processing method.

[0095] In some embodiments, the data processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 518. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 510 via the ROM 512 and / or the communication unit 519. When the computer program is loaded into the RAM 513 and executed by the processor 511, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the processor 511 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0096] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0097] The computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor, the functions / operations specified in the flow charts and / or block diagrams are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0098] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0099] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0100] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data processing server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0101] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0102] The present application also discloses a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the data processing method provided in any embodiment of the present application. The program product and the data processing method disclosed in each embodiment of the present application belong to the same inventive concept, so it is not described here.

[0103] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0104] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A data processing method, characterized in that: The method comprises: In response to detecting a data upload operation of target business data, the target business data is divided into at least two data slices; wherein each data slice includes at least two data items; Performing a mapping operation based on the data type to which the data item belongs through a data mapping node to convert the data item into a key-value pair; Determine, by the data mapping node, a data mapping result corresponding to the data shard based on the key-value pair corresponding to the data item, and transmit the data mapping result to the data reduction node; The data reduction node performs a reduction operation on the data mapping result corresponding to each data shard on the key in the key-value pair so that the key in the key-value pair corresponds to only one value, thereby obtaining a data processing result.

2. The method according to claim 1, characterized in that The performing a mapping operation based on the data type to which the data item belongs through the data mapping node to convert the data item into a key-value pair includes: Determining whether the data item includes a preset field by using the data mapping node, and determining the data type to which the data item belongs; If the data type is raw data, extract the identification field and the characteristic field from the data item, use the identification content corresponding to the identification field as the key, map the business data corresponding to the characteristic field as the value to the key, and convert the data item into a key-value pair; wherein the raw data refers to the target business data that has only been subjected to aggregation processing; If the data type is a reduction result, the value of the key-value pair corresponding to the data item is adjusted, and the key-value pair corresponding to the data item is updated according to the value adjustment result.

3. The method according to claim 1, characterized in that The determining, by the data mapping node, the data mapping result corresponding to the data shard based on the key-value pair corresponding to the data item includes: The data mapping node performs format adjustment on the key-value pairs corresponding to the data items based on a preset format, so as to unify the data format adopted by the key-value pairs; Based on the key-value pairs in a unified format, a data mapping result corresponding to the data shard is determined.

4. The method according to claim 1, characterized in that: The data reduction node performs a reduction operation on the data mapping result corresponding to each data shard on the key in the key-value pair so that the key in the key-value pair corresponds to only one value, and obtains a data processing result, including: Determine, by the data reduction node, the value corresponding to the key in the key-value pair in the data mapping result corresponding to each data shard; If there are at least two values ​​corresponding to the key in the key-value pair, the data reduction node reduces the values ​​corresponding to the key in the key-value pair based on the preset business logic to obtain a business indicator corresponding to the preset business logic; The business indicator is used as a value, and the key-value pair is updated so that the key in the key-value pair corresponds to only one value, thereby obtaining a data processing result.

5. The method according to claim 4, characterized in that The method further comprises: If there is only one value corresponding to the key in the key-value pair, the key-value pair is used as the data processing result.

6. The method according to claim 1, characterized in that The target business data is data corresponding to non-real-time business; the data mapping node and the data reduction node are functionalized nodes.

7. A data processing device, characterized in that: The device comprises: A data splitting module, configured to split the target business data into at least two data slices in response to detecting a data upload operation of the target business data; wherein each data slice includes at least two data items; A data item conversion module, used to perform a mapping operation based on the data type to which the data item belongs through a data mapping node, and convert the data item into a key-value pair; A mapping result determination module, configured to determine, through the data mapping node, a data mapping result corresponding to the data shard based on a key-value pair corresponding to the data item, and transmit the data mapping result to a data reduction node; The first result determination module is used to perform a reduction operation on the data mapping result corresponding to each data shard on the key in the key-value pair through the data reduction node so that the key in the key-value pair corresponds to only one value, thereby obtaining a data processing result.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data processing method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 6.

Images