Mass video copy storage method

A video and copy technology, applied in the field of network video, can solve problems such as large cluster load imbalance, affecting client read and write responses, and large network bandwidth, and achieve the effect of solving load balancing problems.

Inactive Publication Date: 2020-05-15
CHINA CHANGFENG SCI TECH IND GROUPCORP
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Although HDFS provides a tool Balancer to dynamically adjust the cluster load, when using this tool, data migration takes up a large network bandwidth, which is inefficient, and the most important thing is that administrators need to manually operate
Since the DataNode may cause serious I / O burden due to data migration, which affects the read and write response to the client, it is often only suitable for use when the cluster is idle.
Assuming that the cluster performs long-term read and write operations, the load of some nodes may not be adjusted in time, causing the load imbalance of the cluster to increase

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Mass video copy storage method
  • Mass video copy storage method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0015] Such as figure 1 Shown, the realization mode of the present invention is as follows:

[0016] (1) Select according to the ratio of the remaining space to the total disk capacity to reduce the unbalanced load on the local rack. For the backup data stored in the local rack, under normal circumstances, there is little difference in the network conditions in the local rack, so it is only necessary to consider the remaining storage space ratio of the data node to select the node, instead of randomly selecting nodes in the local rack. The placement strategy of randomly selecting data nodes cannot ensure that the data nodes store data evenly, resulting in frequent I / O disk operations on some nodes, while some nodes are too idle. Therefore, when selecting data nodes on the local rack, select according to the ratio of the remaining space to the total disk capacity, so as to reduce the load imbalance on the local rack.

[0017] (2) The selection of cross-racks not only needs to...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a mass video copy storage method. When backup data is stored in a local rack, selection is carried out according to the proportion of residual space to the total capacity of a disk to reduce load imbalance on the local rack; and when data nodes are selected across the racks, not only the factor of residual space storage is considered, but also the influence of network overhead is considered. The load balancing problem is effectively solved. Meanwhile, according to the characteristic of Hadoop distributed operation, video data stored in a machine with high bandwidth, large memory and high disk utilization rate can be subjected to intelligent video analysis and processing to the maximum extent.

Description

technical field [0001] The invention relates to the technical field of network video, in particular to a method for storing massive video copies in a network environment. Background technique [0002] When HDFS distributed strategy selects the DataNode to store the copy, it has selected the appropriate node as much as possible, but it is not the most suitable. For example, considering the storage space, it only judges whether the copy can be stored in the remaining space, but ignores the selection The data node with the largest free space ratio. The method of randomly selecting data nodes has a high probability that the copies will be stored on some data nodes with high space usage, and correspondingly, there are only few copies on data nodes with low space usage. Especially after a large distributed cluster has been running for a long time, each data node has a relatively heavy storage load. If a small number of new no-load nodes join at this time, due to the random algori...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04N21/231H04N5/781
CPCH04N5/781H04N21/23103H04N21/23113
Inventor 张超赵凯
Owner CHINA CHANGFENG SCI TECH IND GROUPCORP
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap