Calculation method and apparatus

A calculation method and technology for configuration files, applied in the field of big data, can solve problems such as inability to meet requirements, unfixed content, and high performance requirements, and achieve the effect of flexible and convenient configuration process, improved development efficiency, and reduced workload.

Inactive Publication Date: 2018-12-28
广东惠禾科技发展有限公司
View PDF3 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This is a task that is fixed for a business structure, but the specific content is not fixed. It is too cumbersome and cannot meet the principle of writing once and using it multiple t

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Calculation method and apparatus
  • Calculation method and apparatus
  • Calculation method and apparatus

Examples

Experimental program
Comparison scheme
Effect test

Example

[0068] First embodiment

[0069] figure 2 It shows a flowchart of the calculation method provided by the first embodiment of the present invention. This calculation method can be applied, but is not limited to being applied to Spark programs. In the following description, the application of the method in the Spark program is taken as an example for illustration, but it does not constitute a limitation on the protection scope of the present invention. Reference figure 1 , Calculation methods include:

[0070] Step S10: The processor of the electronic device reads and parses the configuration file.

[0071] The configuration file is configured for business requirements, and the business requirements referred to here are usually a data processing task.

[0072] In the Spark program, after the SparkContext is initialized, use the shell command to pass in the storage location of the configuration file on HDFS, use IO to read the configuration file, and parse its content according to the...

Example

[0172] Second embodiment

[0173] In the second embodiment, the calculation method is further explained by comparing the calculation method provided by the embodiment of the present invention with the SQL statement.

[0174] The default database in Hive has two tables: student and sc. The fields of student are sno, sname, sage, sex, and the fields of sc are sno, cno, and score. Figure 4 It shows a schematic diagram of the content of the student table provided by the second embodiment of the present invention. Figure 5 It shows a schematic diagram of the content of the score table provided by the second embodiment of the present invention.

[0175] Now there is a demand: I want to find out all the results of all female students. If you use sparkSQL directly, that is, use a SQL-like way, then the SQL statement should be:

[0176] select score, cno from sc where sno in(select sno from student wheresex=‘female’)

[0177] For ease of explanation, the SQL statement is transformed:

[0178] ...

Example

[0193] Fourth embodiment

[0194] The fourth embodiment of the present invention provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are read and run by a processor, the steps of the calculation method provided by the embodiments of the present invention are executed . The computer-readable storage medium can be implemented as, but not limited to figure 1 The memory 102 is shown.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to the technical field of big data, and provides a computing method and a device. The method comprises the following steps of: reading and parsing a configuration file, wherein the content of the configuration file comprises a plurality of nodes and at least one edge connected with the plurality of nodes, wherein each node is used for representing a data processing unit in abusiness process, and each edge is used for representing a data flow direction between two nodes; creating a plurality of nodes and constructing a directed acyclic graph representing a business process based on the plurality of nodes and at least one edge, wherein data processing operations corresponding to each node and each node are defined in a pre-generated package; The data processing operation corresponding to each node is executed according to the data flow in the directed acyclic graph until the data processing operation corresponding to each node is completed. When computing programsare developed for different business requirements, only configuration files need to be modified, and no changes need to be made to the code in the package, which significantly improves development efficiency.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular, to a computing method and device. Background technique [0002] With the development of big data technology and the rapid popularization of mobile Internet, the collected data has increased dramatically, which puts forward higher requirements for data processing speed. Offline computing frameworks such as Hadoop and Spark are currently more advanced technologies. If the speed requirement is relatively high, then choose Spark more. [0003] The current situation is that there are many transition stages from Hadoop to Spark in the industry, and a large number of enterprises still use Hive for ETL (Extract-Transform-Load). The bottom layer of Hive is to use Hadoop's MapReduce to implement SQL-like operations, and its performance is limited. Spark effectively combines Hive, uses Hive metadata, and combines Resilient Distributed Datasets (RDD) operations to use SQL-like Fast...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
Inventor 石城姚佳
Owner 广东惠禾科技发展有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap