Video acquisition and storage method and system based on distributed architecture

Abstract

The invention relates to the technical field of video collection and storage, in particular to a video collection and storage method and system based on a distributed architecture, and the method comprises the steps: generating video data based on video information collected in each region; transmitting the video data of each area to a corresponding storage node; identifying a data accumulation time point; calibrating each storage node based on the data receiving speed of each storage node; determining whether the storage of the video data is qualified or not based on the statistical calibration results of the storage nodes and the abnormal parameters, and scheduling the video data based on the data receiving difference quantity when determining that the storage of each video data is abnormal; and the video storage process is monitored, so that the storage efficiency of the video data is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video acquisition and storage, and particularly to a video acquisition and storage method and system based on a distributed architecture. Background Art

[0002] With the wide popularization of application scenarios such as video surveillance and video recording, the demand for video acquisition and storage is increasing day by day. In large-scale video acquisition and storage systems, the data volume is huge and the sources are extensive. Traditional centralized architectures face many challenges when processing these massive video data: Chinese Patent Application Publication No.: CN103152437A discloses a distributed video surveillance cloud storage system, which includes two storage devices. Platform software is embedded on each storage device. After the encoder converts the analog signal collected by the camera into a digital signal, it directly enters the storage device without passing through a video server. At the same time, multiple storage devices control the entire surveillance system through the platform software; the platform software includes two parts, one part is some service programs distributedly running on the storage device, and the other part is a client program used by users on Windows; thus, the above technical solution has the following problems: It does not consider monitoring the process of storing videos, which affects the storage efficiency of video data. Summary of the Invention

[0003] Therefore, the present invention provides a video acquisition and storage method and system based on a distributed architecture to overcome the problem in the prior art that the process of storing videos is not considered, which affects the storage efficiency of video data.

[0004] On the one hand, the present invention provides a video acquisition and storage method based on a distributed architecture, including: Generating video data based on the video information collected in each region, dividing the video information converted into an encoded format into several video segments, generating hash values of each video segment and time stamps corresponding to each video segment; Based on the preset transmission time points corresponding to each region and the data volume of the temporarily stored video data, transmitting the video data of each region to the corresponding storage node; Identifying data accumulation time points based on the data volume to be transmitted at several obtained time nodes; When a data accumulation time point is identified, calibrating each storage node based on the data receiving speed of each storage node to divide a single storage node into a slow node or a fast node; Determining whether the storage of video data is qualified based on the calibration results and abnormal parameters of the storage nodes, including, Determining that the storage of each video data is abnormal, and scheduling the video data based on the data reception difference amount; Alternatively, determine that the storage of each video data is qualified, and continue to use the current storage parameters to complete the storage of each data.

[0005] Further, the process of transmitting the video data of each region to the corresponding storage node includes: For a single region, when the single region reaches the corresponding transmission time node of the region and the data volume of the temporarily stored video data reaches the preset critical data volume, transmit the temporarily stored video data of the region to the corresponding storage node.

[0006] Further, the process of identifying the data accumulation time point based on the to-be-transmitted data volume of each obtained time node includes: Draw a time-domain curve of the to-be-transmitted data volume based on the to-be-transmitted data volume of each obtained time node; Identify each peak value in the time-domain curve of the to-be-transmitted data volume, and determine the time corresponding to each peak value as the data accumulation time point.

[0007] Further, the process of determining whether the storage of video data is qualified based on the calibrated results of the storage nodes includes: Record the ratio of the number of fast-through nodes calculated to the total number of each storage node as the operation rate; If the operation rate is less than or equal to the first preset operation rate, determine that the storage of each video data is abnormal, and schedule the video data based on the data reception difference amount; If the operation rate is less than or equal to the second preset operation rate and greater than the first preset operation rate, determine whether the storage of video data is qualified based on the abnormal parameter; If the operation rate is greater than the second preset operation rate, determine that the storage of each video data is qualified, and continue to use the current storage parameters to complete the storage of each data.

[0008] Further, the process of determining whether the storage of video data is qualified based on the abnormal parameter includes: For a single lag node, obtain each video data within the preset verification time, calculate the hash value of each obtained video data, compare the recalculated hash value with the original hash value of the video data, and determine the video data with inconsistent hash value comparison as abnormal data; Record the ratio of the number of lag nodes with abnormal data among the calculated lag nodes to the total number of each lag node as the abnormal parameter; If the abnormal parameter is less than or equal to the preset abnormal parameter, adjust the bit rate of the video data generated during the video data generation process to the corresponding value based on the abnormal parameter; If the abnormal parameter is greater than the preset abnormal parameter, determine that the storage of each video data is abnormal, and schedule the video data based on the data reception difference amount; Adjust the bit rate of the video data generated during the video data generation process to the corresponding value based on the abnormal parameter, where: The reduction amplitude of the bit rate is proportional to the abnormal parameter.

[0009] Furthermore, the process of scheduling video data based on the data reception difference amount includes: Obtain the storage data amount of the video data stored in each storage node within the preset verification time, calculate the variance of each storage data amount, and obtain the reception difference amount; If the reception difference amount is less than or equal to the preset reception difference amount, send out abnormal alarm information for each lag node; If the reception difference amount is greater than the preset reception difference amount, schedule the video data based on the smoothness of each storage node.

[0010] Furthermore, the process of scheduling video data based on the smoothness of each storage node includes: Draw a time-domain curve of the storage data amount based on the storage data amount of a single express node within each preset verification time, and record the difference between the maximum value and the minimum value in the calculated time-domain curve of the storage data amount as the smoothness; Arrange the smoothness of each express node in descending order, sequentially select a preset number of express nodes as the nodes to be coordinated, and adjust the data amount of the video data received by the nodes to be coordinated.

[0011] Furthermore, when completing the adjustment of the data amount of the video data received by each node to be coordinated, determining whether to adjust the storage parameters of the video data based on the data fluctuation parameter includes: Calculate the derivative of the obtained time-domain curve of the data amount to be transmitted at the current time node to obtain the data fluctuation parameter; When the data fluctuation parameter is less than or equal to the preset data fluctuation parameter, continue to use the current storage parameters to complete the storage of each data; When the data fluctuation parameter is greater than the preset data fluctuation parameter, increase the number of storage nodes to the corresponding value based on the data fluctuation parameter.

[0012] Furthermore, increasing the number of storage nodes to the corresponding value based on the data fluctuation parameter, where, The increase amplitude of the number of storage nodes is proportional to the data fluctuation parameter.

[0013] On the other hand, the present invention also provides a video acquisition and storage system using the above video acquisition and storage method based on a distributed architecture, including: A video acquisition module, which includes a number of video acquisition units for respectively acquiring video information in each area; the video acquisition unit includes an encoding mechanism for converting video information into an encoded format, and a generation mechanism for respectively generating original hash values based on a number of divided video segments; A data transmission module, which is connected to the video acquisition module and is used to transmit the video data of each video acquisition unit when the video data stored in the video acquisition unit reaches a preset critical data volume and reaches the preset transmission time point corresponding to the video acquisition unit; a data storage module, which is connected to the data transmission module and includes a plurality of storage nodes for respectively storing corresponding video data; A data identification module, which is respectively connected to the data transmission module and the data storage module, and is used to identify the data accumulation time point based on the amount of data to be transmitted of the data transmission module at each time node obtained, and, under the condition of identifying the data accumulation time point, calibrate each storage node based on the data reception speed of each storage node; A data discrimination module, which is respectively connected to the data transmission module, the data storage module and the data identification module, and is used to determine whether the storage of video data is qualified based on the calibration results of the storage nodes counted and the abnormal parameters, and, in the case of determining that the storage of each video data is abnormal, schedule the video data based on the data reception difference amount; A data statistics module, which is connected to the data storage module and is used to record the abnormal alarm information for each lag node sent by the data discrimination module.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows. Based on the preset transmission time points corresponding to each region and the amount of temporarily stored video data, the video data of each region is respectively transmitted to the corresponding storage node. According to the actual situations of different regions, the data transmission timing is reasonably arranged, avoiding the blindness and chaos in the data transmission process, effectively reducing the problems of data accumulation and transmission delay, and improving the efficiency and stability of data transmission.

[0015] Furthermore, by drawing a time-domain curve of the amount of data to be transmitted based on the amount of data to be transmitted at each time node obtained and identifying the peak value in the curve as the data accumulation time point, it is possible to intuitively and accurately discover the possible data accumulation situation in the data transmission process, provide early warning of the data accumulation problem, provide sufficient time for subsequent corresponding measures, help ensure the smoothness of data transmission, and further improve the storage efficiency of video data.

[0016] Furthermore, calibrate each storage node based on the data reception speed of each storage node, and classify the storage nodes into lag nodes and fast-through nodes, so as to clearly understand the performance status of each storage node. By calculating the proportion of the number of fast-through nodes in the total number of nodes and the abnormal parameters, the storage of video data can be comprehensively evaluated. The multi-dimensional evaluation method is more detailed and accurate, and can timely detect potential problems in the storage process. The operation rate characterizes the operation of each storage node in the case of data accumulation. When the operation rate is less than or equal to the first preset operation rate, a large number of storage nodes operate abnormally, resulting in risks in data storage. In this case, the video data is scheduled to further improve the storage efficiency of the video data.

[0017] Furthermore, when the abnormal parameter is less than or equal to the preset abnormal parameter, in this case, there is a small amount of video data that fails to be received normally, which is determined to be due to insufficient network bandwidth, resulting in an increase in the packet loss rate. The current network cannot support high-bitrate video transmission. At this time, the target bitrate of the video is reduced to ensure stable data transmission and further improve the storage efficiency of the video data.

[0018] Furthermore, when storage anomalies are determined, schedule video data based on the data reception difference. Calculate the reception difference by calculating the variance of the data storage amounts of each storage node, and take corresponding measures according to different situations. The data reception difference characterizes the difference in the data storage amounts during the operation of each storage node. When the reception difference is greater than the preset value, in this case, due to the excessive difference in the video data stored by each storage node, the storage nodes fail to coordinate fully, affecting the storage effect. At this time, schedule video data based on the smoothness of the storage nodes to further optimize the data storage allocation. Smoothness characterizes the data reception ability of a single fast-through node. The greater the smoothness, the greater the ability of the fast-through node to receive video data. When the reception difference is less than the preset value, each storage node operates reasonably and cooperatively. At this time, due to the existence of abnormal storage nodes, data accumulates. In this case, send an abnormal alarm message; effectively balance the load of each storage node, and improve the stability and reliability of the entire video data storage. Adjust the bitrate of the video data based on the abnormal parameter, and adjust the number of storage nodes based on the data fluctuation parameter, which can dynamically adjust the storage parameters according to the actual situation to ensure the storage quality of the video data and the performance optimization of the system.

[0019] Furthermore, after adjusting the amount of video data received by the coordination node, continuously monitor the transmission situation to determine the data fluctuation parameter, which characterizes the transmission situation of the adjusted data. When the data fluctuation parameter is greater than the preset data fluctuation parameter, in this case, only coordinating the storage nodes cannot improve the data accumulation situation. At this time, increase the number of storage nodes to the corresponding value based on the data fluctuation parameter to achieve high scalability, ensure the smoothness of data transmission, and further improve the storage efficiency of video data. Description of the Drawings

[0020] Figure 1 It is a flowchart of the steps of the video acquisition and storage method based on a distributed architecture according to an embodiment of the present invention; Figure 2 It is a block diagram of the modules of the video acquisition and storage system based on a distributed architecture according to an embodiment of the present invention; Figure 3 It is a logical decision diagram for calibrating each storage node based on the data reception speed of each storage node according to an embodiment of the present invention; Figure 4 It is a logical decision diagram for determining whether the storage of video data is qualified based on the abnormal parameter according to an embodiment of the present invention. Detailed Embodiments

[0021] In order to make the purpose and advantages of the present invention clearer, the present invention will be further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principle of the present invention and do not limit the protection scope of the present invention.

[0023] It should be noted that in the description of the present invention, the terms indicating directions or positional relationships such as "up", "down", "left", "right", "inside", "outside", etc. are based on the directions or positional relationships shown in the drawings. This is only for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention.

[0024] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and defined, the terms "installation", "connection", and "coupling" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the communication inside two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0025] Please refer to Figure 1 as shown, which is a step flowchart of a video acquisition and storage method based on a distributed architecture according to an embodiment of the present invention. The video acquisition and storage method of the present invention includes: S1, generating video data based on the video information collected in each region; S2, transmitting the video data of each region to the corresponding storage node based on the preset transmission time points corresponding to each region and the data volume of the temporarily stored video data; S3, identifying the data accumulation time points based on the data volume to be transmitted at each time node obtained; S4, calibrating each storage node based on the data reception speed of each storage node when the data accumulation time points are identified; S5, determining whether the storage of the video data is qualified based on the calibrated results of the storage nodes counted and the abnormal parameters, including judging that the storage of each video data is abnormal and scheduling the video data based on the data reception difference amount; or, judging that the storage of each video data is qualified and continuously using the current storage parameters to complete the storage of each data.

[0026] Specifically, each region is preset with corresponding preset transmission time points to enable the video information of each region to be transmitted in different time periods, so as to ensure the data transmission efficiency.

[0027] Based on the preset transmission time points corresponding to each region and the data volume of the temporarily stored video data, the video data of each region is respectively transmitted to the corresponding storage node. According to the actual situations of different regions, the data transmission timing is reasonably arranged, avoiding the blindness and chaos in the data transmission process, effectively reducing the problems of data accumulation and transmission delay, and improving the efficiency and stability of data transmission.

[0028] Specifically, for a single region, when the transmission time node is reached and the data volume reaches the preset critical data volume, the transmission is performed, which can make full use of network resources, avoid the bandwidth waste caused by frequent transmission due to too small data volume, and also avoid the problem of transmission timeout due to too large data volume, further improving the storage efficiency of video data.

[0029] It can be understood that the actual implementers can determine the preset transmission time points corresponding to each region according to the actual situation or based on the regular changes in the video acquisition data volume in different regions in historical data. In this embodiment, for a single region, the video data volume in this region at different time periods is obtained. Through the analysis of historical data, the change trend of the data volume in each region and the time when the transmission peak appears are statistically calculated. Based on the statistically obtained data, the time when the transmission peak appears is determined as the preset transmission time point corresponding to the region, so as to perform data transmission under the condition that the data volume is relatively stable.

[0030] Please refer to Figure 2 as shown in the figure, which is a module block diagram of a video acquisition and storage system based on a distributed architecture according to an embodiment of the present invention. The video acquisition and storage system of the present invention includes: A video acquisition module, which includes a number of video acquisition units for respectively acquiring video information in each region; the video acquisition unit includes an encoding mechanism for converting video information into an encoded format, and a generating mechanism for respectively generating original hash values based on a number of divided video segments; A data transmission module, which is connected to the video acquisition module and is used to transmit the video data of each video acquisition unit when the video data stored in the video acquisition unit reaches the preset critical data volume and reaches the preset transmission time point corresponding to this video acquisition unit; a data storage module, which is connected to the data transmission module and includes a number of storage nodes for respectively storing the corresponding video data; A data identification module, which is respectively connected to the data transmission module and the data storage module and is used to identify the data accumulation time point based on the amount of data to be transmitted of the data transmission module at each time node, and, under the condition of identifying the data accumulation time point, calibrate each storage node based on the data receiving speed of each storage node; A data discrimination module, which is respectively connected to the data transmission module, the data storage module and the data identification module and is used to determine whether the storage of video data is qualified based on the calibrated results of the storage nodes and abnormal parameters, and, in the case of determining that the storage of each video data is abnormal, schedule the video data based on the data receiving difference amount; A data statistics module, which is connected to the data storage module and is used to record the abnormal alarm information sent by the data discrimination module for each lag node.

[0031] Specifically, the video data includes each video segment converted into an encoded format, the original hash value corresponding to each video segment, and the time stamp corresponding to the original hash value; Specifically, the specific structure of the generation mechanism is not limited and can be any logical component. It can be understood that the generation mechanism can divide the encoded video into several video segments based on a fixed duration, generate a unique hash value for each video segment through a hash algorithm, and record the timestamp of each video segment corresponding to the hash value in the entire video. This will not be elaborated here.

[0032] Specifically, a single video acquisition unit includes several video acquisition devices. Each video acquisition device can achieve high-refresh-rate video acquisition at 340 frames per second. During the acquisition process, the acquisition device can perform real-time encoding on the video, converting the video into an encoded format to reduce the data volume and improve the transmission efficiency.

[0033] Specifically, after receiving the video data, the data storage module determines the specific storage node where the data should be stored according to the hash algorithm.

[0034] Specifically, each video data is stored on different nodes in chronological order. In addition, a metadata index is established for each video data to record the storage location, timestamp, and video encoding format of the video data for subsequent query and retrieval.

[0035] Specifically, this solution distributes the video data to each storage node for storage. At the same time, a data redundancy and fault tolerance mechanism is designed to ensure the security and reliability of the data.

[0036] Specifically, the process of generating video data includes dividing the video information converted into an encoded format into several video segments, generating the hash value of each video segment and the timestamp corresponding to each video segment; The process of transmitting the video data of each region to the corresponding storage node includes: For a single region, when the single region reaches the transmission time node corresponding to the region and the data volume of the video data temporarily stored reaches the preset critical data volume, the video data temporarily stored in the region is transmitted to the corresponding storage node.

[0037] Specifically, the process of identifying the data accumulation time point based on the amount of data to be transmitted at each obtained time node includes: Drawing a time-domain curve of the amount of data to be transmitted based on the amount of data to be transmitted at each obtained time node; Identifying each peak value in the time-domain curve of the amount of data to be transmitted and determining each peak value as the data accumulation time point; Please refer to Figure 3 As shown, it is a logical decision diagram for calibrating each storage node based on the data reception speed of each storage node in an embodiment of the present invention. The process of calibrating each storage node based on the data reception speed of each storage node in the present invention includes: If the data reception speed is less than or equal to the preset data reception speed, mark a single storage node as a lagging node; If the data reception speed is greater than the preset data reception speed, mark a single storage node as a fast - passing node.

[0038] It can be understood that the actual implementer can determine the preset data reception speed according to the actual situation or based on the average value of the data reception speeds of each storage node in historical data. Preferably, the preset data reception speed is selected within the range of [400 Mbps, 800 Mbps].

[0039] In this embodiment, optionally, the time nodes of the data volume to be transmitted at each obtained time node can be selected as the corresponding time nodes when the video data stored in the video acquisition unit reaches the preset critical data volume, or can be time nodes with equal interval durations; the data volume to be transmitted is the data volume of the video data that has not completed the transmission task when the data transmission module transmits the video data of each video acquisition unit.

[0040] Specifically, by plotting the time - domain curve of the data volume to be transmitted based on the data volume to be transmitted at each obtained time node and identifying the peak value in the curve as the data accumulation time point, it is possible to intuitively and accurately discover the possible data accumulation situation in the data transmission process, so as to give an early warning of the data accumulation problem, providing sufficient time for subsequent corresponding measures, which helps to ensure the smoothness of data transmission and further improves the storage efficiency of video data.

[0041] Specifically, the process of determining whether the storage of video data is qualified based on the calibration results of the storage nodes includes: Record the ratio of the number of fast - passing nodes calculated to the total number of all nodes as the operation rate; If the operation rate is less than or equal to the first preset operation rate, determine that the storage of each video data is abnormal, and schedule the video data based on the data reception difference amount; If the operation rate is less than or equal to the second preset operation rate and greater than the first preset operation rate, determine whether the storage of video data is qualified based on the abnormal parameter; If the operation rate is greater than the second preset operation rate, determine that the storage of each video data is qualified, and continue to use the current storage parameters to complete the storage of each data; Please refer to Figure 4 As shown, it is the logical decision diagram for determining whether the storage of video data is qualified based on the abnormal parameter in the embodiment of the present invention. The process of determining whether the storage of video data is qualified based on the abnormal parameter in the present invention includes: For a single lag node, obtain each video data within a preset verification time, calculate the hash value of each obtained video data, compare the recalculated hash value with the original hash value of the video data, and determine the video data with inconsistent hash value comparison as abnormal data; Record the ratio of the number of lag nodes with abnormal data among the calculated lag nodes to the total number of lag nodes as an abnormal parameter; If the abnormal parameter is less than or equal to the preset abnormal parameter, adjust the bit rate of the video data generated during the video data generation process to the corresponding value based on the abnormal parameter; If the abnormal parameter is greater than the preset abnormal parameter, determine that the storage of each video data is abnormal, and schedule the video data based on the data reception difference amount.

[0042] Specifically, the first preset operation rate is selected within the interval [0.41, 0.62], and the second preset operation rate is selected within the interval [0.78, 0.9].

[0043] Specifically, the preset abnormal parameter C0 is selected within the interval [0.1, 0.26].

[0044] Specifically, calibrate each storage node based on the data reception speed of each storage node, and divide the storage nodes into lag nodes and fast-pass nodes, so as to clearly understand the performance status of each storage node. By calculating the operation rate and abnormal parameter of the proportion of the number of fast-pass nodes in the total number of nodes, the storage of video data can be comprehensively evaluated. The multi-dimensional evaluation method is more detailed and accurate, and can timely discover potential problems in the storage process. The operation rate represents the operation situation of each storage node when there is data accumulation. When the operation rate is less than or equal to the first preset operation rate, there are a large number of abnormal operations of storage nodes, resulting in risks in data storage. In this case, the video data is scheduled, further improving the storage efficiency of the video data.

[0045] Specifically, when the abnormal parameter is less than or equal to the preset abnormal parameter, in this case, there is a small amount of video data that fails to be received normally, which is determined to be due to insufficient network bandwidth, resulting in an increase in the packet loss rate. The current network cannot support high-bit rate video transmission. At this time, reduce the target bit rate of the video to ensure stable data transmission, further improving the storage efficiency of the video data.

[0046] Specifically, adjust the bit rate of the video data generated during the video data generation process to the corresponding value based on the abnormal parameter, where: The reduction amplitude of the bit rate is proportional to the abnormal parameter.

[0047] In this embodiment, optionally, Compare the abnormal parameter with the first preset abnormal parameter comparison threshold and the second preset abnormal parameter comparison threshold; When the abnormal parameter is less than or equal to the first preset abnormal parameter comparison threshold, adjust the bit rate of the generated video data to 0.91 times the initial bit rate; When the abnormal parameter is less than or equal to the second preset abnormal parameter comparison threshold and greater than the first preset abnormal parameter comparison threshold, adjust the bit rate of the generated video data to 0.83 times the initial bit rate; When the abnormal parameter is greater than the second preset abnormal parameter comparison threshold, adjust the bit rate of the generated video data to 0.7 times the initial bit rate; The first preset abnormal parameter comparison threshold is taken as 0.52C0, and the second preset abnormal parameter comparison threshold is taken as 0.88C0.

[0048] Specifically, the process of scheduling video data based on the data reception difference amount includes: Obtain the storage data amounts of the video data stored in each storage node within the preset verification time, calculate the variance of each storage data amount, and obtain the reception difference amount; If the reception difference amount is less than or equal to the preset reception difference amount, send out abnormal alarm information for each lag node; If the reception difference amount is greater than the preset reception difference amount, schedule the video data based on the smoothness of each storage node.

[0049] Specifically, the preset reception difference amount is selected within the interval [2P0 2 , 4P0 2 , and P0 is the average value of each storage data amount.

[0050] Specifically, when receiving the abnormal alarm information for each lag node, each storage node starts a self - inspection operation, including obtaining various parameters of the hard disk through the SMART (Self - Monitoring, Analysis and Reporting Technology) tool, the temperature of the hard disk, the number of read / write errors, and the remaining life to check the health status of the storage device hard disk. When it is determined that the hard disk temperature is too high or there are many read / write errors, record and report to the data statistics module; Specifically, when storage anomalies are determined, video data is scheduled based on the data reception difference amount. The reception difference amount is obtained by calculating the variance of the data storage amounts of each storage node, and corresponding measures are taken according to different situations. The data reception difference amount characterizes the difference in the data storage amounts during the operation of each storage node. When the reception difference amount is greater than the preset value, in this case, due to the excessive difference in the video data stored in each storage node, the storage nodes fail to coordinate fully, affecting the storage effect. At this time, video data is scheduled based on the smoothness of the storage nodes to further optimize the data storage allocation. Smoothness characterizes the data reception ability of a single express node, and the greater the smoothness, the greater the ability of the express node to receive video data. When the reception difference amount is less than the preset value, each storage node operates in a reasonable and coordinated manner. At this time, due to the existence of abnormal storage nodes, data accumulates, and in this case, an abnormal alarm message is sent; effectively balancing the loads of each storage node improves the stability and reliability of the entire video data storage.

[0051] Adjust the bit rate of video data based on abnormal parameters and adjust the number of storage nodes based on data fluctuation parameters, which can dynamically adjust storage parameters according to the actual situation to ensure the storage quality of video data and the performance optimization of the system.

[0052] Specifically, the process of scheduling video data based on the smoothness of each storage node includes: Based on the stored data amounts of a single express node within each preset verification time obtained, plot a time-domain curve of the stored data amounts, and record the difference between the maximum value and the minimum value in the calculated time-domain curve of the stored data amounts as the smoothness; Arrange the smoothness of each express node in descending order, successively select a preset number of express nodes to be coordinated as the nodes to be coordinated, and adjust the data amount of the video data received by the nodes to be coordinated.

[0053] In this embodiment, optionally, The process of adjusting the data amount of the video data received by the nodes to be coordinated includes: Adjust the data proportion of the video data received by the nodes to be coordinated to 1.1 times the initial data proportion.

[0054] The data proportion is the ratio of the data amount of the video data received by a single storage node to the total data amount of the video data transmitted by the data transmission module.

[0055] Specifically, when the adjustment of the data amount of the video data received by each node to be coordinated is completed, determining whether to adjust the storage parameters of the video data based on the data fluctuation parameters includes: Calculate the derivative of the obtained time-domain curve of the data amount to be transmitted at the current time node to obtain the data fluctuation parameter; When the data fluctuation parameter is less than or equal to the preset data fluctuation parameter, continue to use the current storage parameters to complete the storage of each data; When the data fluctuation parameter is greater than the preset data fluctuation parameter, increase the number of storage nodes to the corresponding value based on the data fluctuation parameter.

[0056] Specifically, the preset data fluctuation parameter B0 is selected within the interval [0.51L0, 0.62L0], and L0 is the average value of the derivatives of each time node in the time domain curve of the data to be transmitted.

[0057] Specifically, after completing the adjustment of the data volume of the video data received by the to-be-coordinated node, continuously monitor the transmission situation to determine the data fluctuation parameter, which characterizes the transmission situation of the adjusted data. When the data fluctuation parameter is greater than the preset data fluctuation parameter, in this case, only coordinating the storage nodes cannot improve the data accumulation situation. At this time, increase the number of storage nodes to the corresponding value based on the data fluctuation parameter to achieve high scalability, ensure the smoothness of data transmission, and further improve the storage efficiency of video data.

[0058] Specifically, increase the number of storage nodes to the corresponding value based on the data fluctuation parameter, where the increase amplitude of the number of storage nodes is proportional to the data fluctuation parameter.

[0059] In this embodiment, optionally, compare the data fluctuation parameter with the first preset fluctuation comparison threshold and the second preset fluctuation comparison threshold; When the data fluctuation parameter is less than or equal to the first preset fluctuation comparison threshold, increase the number of storage nodes to 1.1 times the initial number of storage nodes; When the data fluctuation parameter is less than or equal to the second preset fluctuation comparison threshold and greater than the first preset fluctuation comparison threshold, increase the number of storage nodes to 1.15 times the initial number of storage nodes; When the data fluctuation parameter is greater than the second preset fluctuation comparison threshold, increase the number of storage nodes to 1.2 times the initial number of storage nodes; The first preset fluctuation comparison threshold is taken as 1.35B0, and the second preset fluctuation comparison threshold is taken as 1.66B0.

[0060] Specifically, after completing the increase of the storage nodes, based on the storage capacity and current load conditions of each storage node, evenly distribute the video data to the newly added nodes and the original nodes. In this embodiment, calculate the remaining available storage capacity of each node, and distribute the video data to be stored to each storage node according to the ratio of the remaining available storage capacity of each storage node.

[0061] So far, the technical solution of the present invention has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easily understood by those skilled in the art that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0062] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A video acquisition and storage method based on a distributed architecture, characterized in that: include: Generate video data based on the video information collected in each area, divide the video information converted into a coded format into a number of video segments, and generate a hash value of each video segment and a timestamp corresponding to each video segment; Based on the preset transmission time point corresponding to each area and the amount of temporarily stored video data, the video data of each area is transmitted to the corresponding storage node; Identify the data accumulation time point based on the amount of data to be transmitted at several acquired time nodes; When a data accumulation time point is identified, each storage node is calibrated based on the data receiving speed of each storage node to divide a single storage node into a slow node or a fast node; Determine whether the video data storage is qualified based on the statistical storage node calibration results and abnormal parameters, including: Determine storage anomalies for each video data, and schedule the video data based on the data reception difference amount; Or, it is determined that the storage of each video data is qualified, and the current storage parameters are continuously used to complete the storage of each data.

2. The video acquisition and storage method based on distributed architecture according to claim 1 is characterized in that: The process of transmitting the video data of each area to the corresponding storage node includes: For a single area, when the single area reaches the transmission time node corresponding to the area and the data volume of the temporarily stored video data reaches a preset critical data volume, the temporarily stored video data of the area is transmitted to the corresponding storage node.

3. The video acquisition and storage method based on distributed architecture according to claim 2 is characterized in that: The process of identifying the data accumulation time point based on the amount of data to be transmitted at each time node obtained includes: Draw a time domain curve of the amount of data to be transmitted based on the amount of data to be transmitted at each time node; Identify each peak value in the time domain curve of the amount of data to be transmitted, and determine the time corresponding to each peak value as the data accumulation time point.

4. The video acquisition and storage method based on distributed architecture according to claim 3 is characterized in that: The process of determining whether the storage of video data is qualified based on the calibration results of the statistical storage nodes includes: The ratio of the number of calculated fast-pass nodes to the total number of storage nodes is recorded as the operation rate; If the operation rate is less than or equal to the first preset operation rate, determining that storage of each video data is abnormal, and scheduling the video data based on the data reception difference amount; If the operation rate is less than or equal to the second preset operation rate and greater than the first preset operation rate, determining whether the storage of the video data is qualified based on the abnormal parameter; If the operation rate is greater than the second preset operation rate, it is determined that the storage of each video data is qualified, and the current storage parameters are continuously used to complete the storage of each data.

5. The video acquisition and storage method based on distributed architecture according to claim 4 is characterized in that: The process of determining whether the storage of video data is qualified based on abnormal parameters includes: For a single slow node, each video data within a preset verification time is obtained, a hash value of each video data obtained is calculated, the recalculated hash value is compared with the original hash value of the video data, and the video data with inconsistent hash value comparison is determined as abnormal data; The ratio of the number of hysteresis nodes with abnormal data in each hysteresis node to the total number of hysteresis nodes is recorded as an abnormal parameter; If the abnormal parameter is less than or equal to the preset abnormal parameter, adjusting the bit rate of the video data generated in the process of generating the video data to a corresponding value based on the abnormal parameter; If the abnormal parameter is greater than the preset abnormal parameter, it is determined that the storage of each video data is abnormal, and the video data is scheduled based on the data reception difference amount; The bit rate of the video data generated in the process of generating the video data is adjusted to a corresponding value based on the abnormal parameter, wherein: The reduction in bit rate is proportional to the abnormal parameter.

6. The video acquisition and storage method based on distributed architecture according to claim 5 is characterized in that: The process of scheduling video data based on the difference in data reception includes: Obtain the storage data volume of the video data stored in each storage node within a preset verification time, calculate the variance of each storage data volume, and obtain the reception difference amount; If the received difference is less than or equal to the preset received difference, an abnormal alarm message is issued for each sluggish node; If the received difference amount is greater than the preset received difference amount, the video data is scheduled based on the smoothness of each storage node.

7. The video acquisition and storage method based on distributed architecture according to claim 6 is characterized in that: The process of scheduling video data based on the smoothness of each storage node includes: Based on the storage data volume of a single fast-pass node obtained within each preset verification time, a storage data volume time domain curve is drawn, and the difference between the maximum value and the minimum value in the calculated storage data volume time domain curve is recorded as the smoothness; The smoothness of each fast-pass node is arranged in descending order, a preset scheduling number of fast-pass nodes are selected in turn as nodes to be coordinated, and the amount of video data received by the nodes to be coordinated is adjusted.

8. The video acquisition and storage method based on distributed architecture according to claim 7 is characterized in that: When the adjustment of the data amount of the video data received by each node to be coordinated is completed, determining whether to adjust the storage parameter of the video data based on the data fluctuation parameter includes: Calculate the derivative of the acquired time domain curve of the amount of data to be transmitted at the current time node to obtain the data fluctuation parameter; When the data fluctuation parameter is less than or equal to the preset data fluctuation parameter, the current storage parameter is continuously used to complete the storage of each data; When the data fluctuation parameter is greater than a preset data fluctuation parameter, the number of storage nodes is increased to a corresponding value based on the data fluctuation parameter.

9. The video acquisition and storage method based on distributed architecture according to claim 8, characterized in that: The number of storage nodes is increased to a corresponding value based on the data fluctuation parameter, where: The increase in the number of storage nodes is proportional to the data fluctuation parameter.

10. A video acquisition and storage system using the video acquisition and storage method based on a distributed architecture as described in any one of claims 1 to 9, characterized in that: include: A video acquisition module, which includes a plurality of video acquisition units for respectively acquiring video information in each area; The video acquisition unit includes an encoding mechanism for converting video information into an encoding format, and a generating mechanism for generating original hash values ​​based on the divided video segments; A data transmission module, connected to the video acquisition module, for transmitting the video data of each video acquisition unit when the video data stored in the video acquisition unit reaches a preset critical data amount and reaches a preset transmission time point corresponding to a single video acquisition unit; A data storage module, which is connected to the data transmission module and includes a plurality of storage nodes for storing corresponding video data respectively; a data identification module, which is connected to the data transmission module and the data storage module respectively, and is used to identify the data accumulation time point based on the amount of data to be transmitted of the data transmission module at each time node, and, under the condition that the data accumulation time point is identified, calibrate each storage node based on the data receiving speed of each storage node; A data discrimination module, which is respectively connected to the data transmission module, the data storage module and the data identification module, and is used to determine whether the storage of the video data is qualified based on the calibration results and abnormal parameters of the statistical storage nodes, and to schedule the video data based on the data reception difference amount; The data statistics module is connected to the data storage module and is used to record the abnormal alarm information for each sluggish node issued by the data identification module.

Images