ID generation method and system, computer and readable instruction storage medium
A computer and factor generation technology, applied in the Internet field, can solve problems such as not desirable, insufficient memory of a single machine, etc.
Pending Publication Date: 2021-05-18
GUANGZHOU HUANWANG TECH
View PDF0 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
In some scenarios, it is necessary to generate a unique ID for each piece of data in the process of data processing. In traditional development, the globally unique ID is generated through a database or other third-party open source libraries. In spark, itself It has an ID generator based on OpenHashMap, but it is only effective in a stand-alone scenario. There is no way to do it in a distributed scenario. However, if the amount of data is extremely large, reaching the level of 100 billion or 10 billion, then the stand-alone machine will have insufficient memory. Therefore, in big data scenarios, it is not advisable to rely on databases, third-party libraries, or stand-alone methods to generate IDs
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other implementations obtained by persons of ordinary skill in the art without making creative efforts fall within the protection scope of the present invention.
[0048] figure 1It is a schematic diagram of an ID generation method provided by the embodiment of this application. This embodiment is applicable to the situation of ID generation in a distributed cluster environment. This method is developed based on a computing engine (Spark) in a big data scenario. The method can be executed by an ID generation system provided in the embodiment of the present application, and the system can be realized by software and / or har...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention belongs to the technical field of Internet, and particularly relates to an ID generation method and system, a computer and a readable instruction storage medium. The method includes acquiring a first ID generation factor of a record of a to-be-generated ID; acquiring a second ID generation factor of the record of the to-be-generated ID; according to the first ID generation factor and the second ID generation factor, acquiring a third ID generation factor; according to the first ID generation factor and the second ID generation factor, calculating according to a first preset rule to obtain a fourth ID generation factor; and splicing the first ID generation factor, the second ID generation factor, the third ID generation factor and the fourth ID generation factor according to a preset rule to obtain a splicing result, and determining the generated splicing result as a target ID. The method is simple and easy to use, and supports the distributed generation of the unique ID in the calculation of a large amount of data, so as to facilitate the subsequent calculation.
Description
technical field [0001] The invention belongs to the technical field of the Internet, and in particular relates to an ID generation method, system, computer and readable instruction storage medium. Background technique [0002] In big data scenarios, the computing engine (Spark) is the most commonly used big data development framework, dedicated to big data processing. Spark provides a large number of libraries, including Spark Core, Spark SQL, Spark Streaming, MLlib, GraphX. Developers can seamlessly combine these libraries in the same application. In some scenarios, it is necessary to generate a unique ID for each piece of data in the process of data processing. In traditional development, the globally unique ID is generated through a database or other third-party open source libraries. In spark, itself It has an ID generator based on OpenHashMap, but it is only effective in a stand-alone scenario. There is no way to do it in a distributed scenario. However, if the amount...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F16/22
CPCG06F16/2228G06F16/2291
Inventor 宁炳剑
Owner GUANGZHOU HUANWANG TECH