Unlock instant, AI-driven research and patent intelligence for your innovation.

Data processing method and device, computer storage medium and electronic equipment

A data processing and processor technology, applied in the computer field, can solve the problems of large scale of model parameters and long time-consuming re-sharding operations, and achieve the effects of improving efficiency, improving business results, and reducing data operation pressure.

Pending Publication Date: 2022-05-06
BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, when the number of parameter servers is adjusted, this method needs to re-shard the model parameters, but due to the huge size of the model parameters, the re-sharding operation takes a long time

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device, computer storage medium and electronic equipment
  • Data processing method and device, computer storage medium and electronic equipment
  • Data processing method and device, computer storage medium and electronic equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details being omitted, or other methods, components, devices, steps, etc. may be adopted. In other instances, well-known technical solution...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to the technical field of computers, and provides a data processing method, a data processing device, a computer storage medium and electronic equipment which are applied to a data fragmentation terminal, the data fragmentation terminal is in communication connection with a plurality of parameter servers, and the parameter servers are used for maintaining to-be-stored data. The data processing method comprises the following steps: dividing to-be-stored data into a plurality of fragments with fragment serial numbers according to a preset fragment number; according to the preset fragment number and the number of the parameter servers, fragment serial numbers distributed to the parameter servers are determined, so that the parameter servers load fragments corresponding to the fragment serial numbers, and indexes are created for the fragments. According to the method and the device, when the number of the parameter servers is changed, the data re-fragmentation operation can be avoided, and the data loading efficiency is improved, so that the model training and model online efficiency is improved.

Description

technical field [0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, a data processing device, a computer storage medium and electronic equipment. Background technique [0002] In the training process of the model, it is necessary to deal with large-scale sparse features. The parameter scale corresponding to these sparse features is relatively large. In order to improve the processing efficiency of its update and query, it is generally necessary to segment them according to certain rules and store them in different in the parameter server. Moreover, as the amount of training data is updated, there is a need to adjust the number of parameter servers according to the size of the model parameters. [0003] In related technologies, a preset hash algorithm is generally used to calculate a hash value of a feature, and a remainder operation is performed between the hash value and the number of parameter servers t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22G06F16/23G06F16/2455
CPCG06F16/2228G06F16/2379G06F16/2455
Inventor 邢召龙刘近光王文生张克丰包勇军刘倩欣张林贺旭熊俊
Owner BEIJING WODONG TIANJUN INFORMATION TECH CO LTD