Unlock instant, AI-driven research and patent intelligence for your innovation.

Data processing method and device, equipment and storage medium

A data processing and data table technology, applied in the direction of electrical digital data processing, special data processing applications, other database retrieval, etc., to achieve the effect of improving data processing efficiency

Active Publication Date: 2022-06-10
HUNDSUN TECH
View PDF13 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In view of this, the embodiment of the present application provides a data processing method, device, equipment and storage medium to solve the low efficiency of data processing in the prior art, and problems with certain limitations

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device, equipment and storage medium
  • Data processing method and device, equipment and storage medium
  • Data processing method and device, equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0057] Among them, different preset databases store data from different data sources, the data includes multiple field names, some of the multiple field names may have field values, and another part of the multiple field names may not Has a field value.

[0058] The preset database can be any type of relational database. In order to insert and update any type of relational data, first use the driver node to pull data from different data sources, and then store the data of at least one preset database in the preset In the data table of the database.

[0059] In a possible implementation, distributed reading of data from at least one preset database may specifically include the following steps:

[0060] (2) Create a temporary table (the data table of the preset database): based on the Spark platform, the read table data is stored in the data table containing the table structure in the memory, and cached, which can prevent the table data from being lost after the task execution ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a data processing method and device, equipment and a storage medium, the data processing method and device are applied to each actuator node in a plurality of actuator nodes in a data processing platform, the data processing platform is further provided with a driver node, and the method comprises the steps that the driver node is adopted to obtain information of a to-be-processed field, the information of the to-be-processed field comprises a field name of the to-be-processed field and a corresponding field value, judging whether a data table of each actuator node comprises the field name of the to-be-processed field or not by adopting each actuator node, and if a data table of a target actuator node in the plurality of actuator nodes comprises the field name of the to-be-processed field, executing the to-be-processed field according to the to-be-processed field. And if yes, operating the data table of the target actuator node according to the field value of the to-be-processed field by adopting the target actuator node. Data migration is not needed, multiple actuator nodes are adopted for parallel processing, the data processing efficiency is improved, and the method is suitable for various types of relational databases.

Description

technical field [0001] The present application relates to the field of computer technology, and in particular, to a data processing method, device, device, and storage medium. Background technique [0002] Spark is a fast and general-purpose computing engine designed for large-scale data processing. Spark supports the operation of various types of relational databases, such as reading and writing data from relational databases such as mysql, oralce, and postgrsql. [0003] At present, the method of inserting and updating relational databases based on the Spark platform often uses the method of merging, in which the data needs to be landed, that is, the data is migrated to the relational database to be inserted and updated, and then the data is inserted or updated to the data table in the relational database. [0004] However, this method is not only inefficient in data processing, but also has certain limitations because different databases have different constraints, so th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/903G06F11/34G06K17/00
CPCG06F16/903G06F11/3414G06K17/0022Y02D10/00
Inventor 闫长虎张绍震秦元黄智浩严羽
Owner HUNDSUN TECH