Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Calculation method and apparatus

A calculation method and technology for configuration files, applied in the field of big data, can solve problems such as inability to meet requirements, unfixed content, and high performance requirements, and achieve the effect of flexible and convenient configuration process, improved development efficiency, and reduced workload.

Inactive Publication Date: 2018-12-28
广东惠禾科技发展有限公司
View PDF3 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This is a task that is fixed for a business structure, but the specific content is not fixed. It is too cumbersome and cannot meet the principle of writing once and using it multiple times.
In real work, the requirements will change at any time, and the performance requirements are high. The method of modifying the code each time is far from meeting the requirements.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Calculation method and apparatus
  • Calculation method and apparatus
  • Calculation method and apparatus

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0069] figure 2 A flow chart of the calculation method provided by the first embodiment of the present invention is shown. This calculation method can be applied to, but not limited to, Spark programs. In the following descriptions, the application of the method in the Spark program is taken as an example for illustration, but this does not constitute a limitation to the protection scope of the present invention. refer to figure 1 , the calculation method includes:

[0070] Step S10: the processor of the electronic device reads and parses the configuration file.

[0071] A configuration file is configured for a business requirement, which is usually a data processing task.

[0072] In the Spark program, after the SparkContext is initialized, use the shell command to pass in the storage location of the configuration file on HDFS, use IO to read the configuration file, and parse its content according to the format of the configuration file, where the configuration file can ...

no. 2 example

[0173] In the second embodiment, the calculation method provided by the embodiment of the present invention is further described by making an analogy to the SQL statement.

[0174] The default database in Hive has two tables student and sc. The fields of student include sno, sname, sage, and sex, and the fields of sc include sno, cno, and score. Figure 4 A schematic diagram of the content of the student table provided by the second embodiment of the present invention is shown. Figure 5 A schematic diagram of the contents of the score table provided by the second embodiment of the present invention is shown.

[0175] Now there is a requirement: I want to find out all the grades of all female students. If you directly use sparkSQL to operate, that is, use the SQL-like method, then the SQL statement should be:

[0176] select score, cno from sc where sno in (select sno from student where sex = 'female')

[0177] For the convenience of explanation, the SQL statement is modifi...

no. 4 example

[0194] The fourth embodiment of the present invention provides a computer-readable storage medium. Computer program instructions are stored on the computer-readable storage medium. When the computer program instructions are read and run by a processor, the steps of the calculation method provided by the embodiment of the present invention are executed. . The computer readable storage medium can be implemented as, but not limited to figure 1Memory 102 is shown.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to the technical field of big data, and provides a computing method and a device. The method comprises the following steps of: reading and parsing a configuration file, wherein the content of the configuration file comprises a plurality of nodes and at least one edge connected with the plurality of nodes, wherein each node is used for representing a data processing unit in abusiness process, and each edge is used for representing a data flow direction between two nodes; creating a plurality of nodes and constructing a directed acyclic graph representing a business process based on the plurality of nodes and at least one edge, wherein data processing operations corresponding to each node and each node are defined in a pre-generated package; The data processing operation corresponding to each node is executed according to the data flow in the directed acyclic graph until the data processing operation corresponding to each node is completed. When computing programsare developed for different business requirements, only configuration files need to be modified, and no changes need to be made to the code in the package, which significantly improves development efficiency.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular, to a computing method and device. Background technique [0002] With the development of big data technology and the rapid popularization of mobile Internet, the collected data has increased dramatically, which puts forward higher requirements for data processing speed. Offline computing frameworks such as Hadoop and Spark are currently more advanced technologies. If the speed requirement is relatively high, then choose Spark more. [0003] The current situation is that there are many transition stages from Hadoop to Spark in the industry, and a large number of enterprises still use Hive for ETL (Extract-Transform-Load). The bottom layer of Hive is to use Hadoop's MapReduce to implement SQL-like operations, and its performance is limited. Spark effectively combines Hive, uses Hive metadata, and combines Resilient Distributed Datasets (RDD) operations to use SQL-like Fast...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
Inventor 石城姚佳
Owner 广东惠禾科技发展有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com