Data transmission and storage method and system for data warehouse

Abstract

The present invention proposes a data transmission and storage method and system for a data warehouse, which belongs to the technical field of data warehouse. The method includes step S100: obtaining a set of time series data flows to be transmitted, DataFlowSet; S200: performing data flow truncation on DataFlowSet based on a time truncation threshold to obtain K truncated data flow subsets; S300: sequentially storing the K truncated data flow subsets into a P-layer circular stack of a data warehouse; S400: judging whether there are two consecutive circular stacks in the P-layer circular stack that are full at the same time, if so, reducing the size of the time truncation threshold and returning to step S200; otherwise, returning to step S300. The system is connected to a light field camera array to implement the method. The present invention uses a data warehouse of a circular stack to perform data transmission and storage, maintaining the structural and temporal integrity of the data itself to the greatest extent while avoiding data transmission blockage and improving data transmission efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of data warehouses, and in particular, relates to a data transmission and storage method and system for data warehouses. Background Art

[0002] A data warehouse is a subject-oriented, integrated, non-updatable, and time-changing data collection that supports the decision-making analysis process. With the development of data warehouse-related technology stacks, the traditional basic architecture of data warehouses has been formed. The basic data warehouse system includes data sources, data storage and computing, OLAP servers, and front-end tools and applications.

[0003] In the field of massive video transmission, such as the transmission of video frames generated by light field camera arrays, an intermediate storage warehouse is usually used to store video frames, and then some or all of the qualified video frames are selected for subsequent imaging and analysis according to different purposes. For example, for a user's video request to connect and play, the system analyzes and screens the data through the node, and then queries the node URL address where the specified video is stored in the data warehouse Hbase of the big data video, and edits and processes it during the query process.

[0004] However, since light field cameras usually have different depth acquisition modes at different time nodes when generating video frame data, the resolution and quantity of the obtained video frame data that changes over time are unpredictable. Using traditional storage structures as data warehouses will cause the data warehouse itself to be easily structurally chaotic and data transmission to be blocked. The data transmission efficiency and stability cannot meet the actual light field imaging needs.

[0005] Therefore, both the data warehouse transmission method and storage method for light field imaging data in the prior art need to be improved. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a data transmission and storage method and system for a data warehouse. The present invention utilizes a circular stack data warehouse to perform data transmission and storage, thereby maintaining the structural and temporal integrity of the data itself to the greatest extent while avoiding data transmission blockage and improving data transmission efficiency.

[0007] In a first aspect of the present invention, a data transmission and storage method for a data warehouse is proposed. The data warehouse is constructed using a P-layer ring stack, where P>2 and P is a positive integer.

[0008] Specifically, the method comprises the following steps:

[0009] S100: Obtain a set of time series data flows to be transmitted, DataFlowSet;

[0010] S200: Obtain a time truncation threshold TimeCur, and based on the time truncation threshold TimeCur, perform data flow truncation on the set of time-series data flows to be transmitted DataFlowSet to obtain K truncated data flow subsets;

[0011] As a specific application object of the above method, the set of time-series data flows to be transmitted DataFlowSet includes N video frame data, each of which has a time node value;

[0012] The N video frame data are sorted in the set of time-series data flows to be transmitted DataFlowSet according to the size of the time node values;

[0013] Specifically, in the technical solution of the present invention, the video frame data is acquired by a light field camera, and the light field camera has multiple acquisition modes, and the acquisition resolutions under different acquisition modes are different. The acquisition modes of the light field camera are different at different time nodes, so the resolutions of the video frame data corresponding to different time nodes are not completely the same;

[0014] Correspondingly, each truncated data stream subset contains M video frame data, and the difference between the maximum time node value and the minimum time node value of the video frame data in each truncated data stream subset is not greater than the time truncation threshold TimeCur; the truncation threshold TimeCur is a positive integer;

[0015] The M, N, and K are all positive integers, and Indicates rounding down; K≥2;

[0016] S300: sequentially storing the K truncated data stream subsets into the P-layer ring stack of the data warehouse;

[0017] S400: Determine whether two consecutive ring stacks of the P-layer ring stack are full at the same time. If yes, reduce the time cutoff threshold TimeCur and return to step S200.

[0018] Otherwise, return to step S300;

[0019] The data warehouse includes a continuous P-layer ring stack {Stack1, Stack2, ...Stack P}, Circular Stack Stack j With a predetermined storage size Mem j , j=1,2,…P.

[0020] More specifically, as a further improvement of step S300, the K truncated data flow subsets are DataFlowSub1, DataFlowSub2, ..., DataFlowSub K ;

[0021] If P ≥ K, the data flow subset DataFlowSub will be truncated iter Store to the ring stack Stack k , iter = 1, 2, ... K;

[0022] If P < K, then when iter is not greater than P, the data flow subset DataFlowSub will be truncated iter Store to the ring stack Stack iter ; When iter is greater than P, the data flow subset DataFlowSub will be truncated iter Store to ring stack , Represents the remainder when iter is divided by P.

[0023] The step S400 further includes: acquiring the video frame data stored in the full ring stack at one time, and clearing the full ring stack.

[0024] As a specific hardware implementation, the video frame data stored in the full-stack circular stack is acquired at one time through the data sending subsystem and sent to the light field imaging unit, and the full-stack circular stack is cleared.

[0025] In step S400, the time cutoff threshold TimeCur is reduced by the following judgment method:

[0026] Let the two ring stacks that are full at the same time be Stack a and Stack a+1 , the corresponding storage size is Mem a 、Mem a+1 , a=1,2,…P-1;

[0027] If the two consecutive ring stacks are full at the same time for the first time, the time cutoff threshold TimeCur is adjusted as follows:

[0028]

[0029] If the two consecutive ring stacks are full at the same time and this is not the first time, the time cutoff threshold TimeCur is adjusted as follows:

[0030]

[0031] Among them, TimeAdj is the adjusted time cutoff threshold; TimeDis is the time difference between the time when the simultaneous full stack occurs this time and the time when the simultaneous full stack occurs last time.

[0032] In a second aspect of the present invention, a data transmission and storage system for a data warehouse is proposed, wherein the system is connected to a light field camera array, each light field camera of the light field camera array acquires video frame data as data to be transmitted, the light field camera has multiple acquisition modes, the acquisition resolutions in different acquisition modes are different, and the acquisition modes of the light field camera are different at different time nodes.

[0033] The time series data generated by the light field camera array is video frame data, and each video frame data has a time node value.

[0034] The system of the second aspect can implement the method of the first aspect.

[0035] In terms of specific structure, the system includes a data warehouse and the following subsystems:

[0036] The data warehouse includes a multi-layer circular storage stack;

[0037] A data receiving subsystem, wherein the data receiving subsystem receives the time series data generated by the light field camera array to form a set of time series data streams to be transmitted;

[0038] A data stream truncation subsystem, wherein the data stream truncation subsystem performs data stream truncation on the set of time-series data streams to be transmitted based on a time truncation threshold value to obtain multiple truncated data stream subsets, each truncated data stream subset comprising multiple video frame data, and the difference between the maximum time node value and the minimum time node value of the video frame data in each truncated data stream subset is not greater than the time truncation threshold value; the truncation threshold value is a positive integer;

[0039] A data storage subsystem, the data storage subsystem sequentially storing a plurality of truncated data stream subsets into a multi-layered ring storage stack of the data warehouse;

[0040] The truncation threshold adjustment subsystem determines whether two consecutive ring stacks of the multi-layer ring storage stack are full at the same time, and if so, reduces the time truncation threshold.

[0041] As the core storage unit, the data warehouse includes a continuous P-layer ring stack {Stack1, Stack2, ...Stack P}, Circular Stack j With a predetermined storage size Mem j , j = 1, 2, ... P;

[0042] The multiple video frame data are sorted in the set of time-series data streams to be transmitted according to the size of the time node values; the resolution sizes of the video frame data corresponding to different time nodes are not completely the same.

[0043] The truncation threshold adjustment subsystem determines whether two consecutive ring stacks of the multi-layer ring storage stack are full at the same time, and if so, reduces the time truncation threshold, specifically including:

[0044] Let the two ring stacks that are full at the same time be Stack a and Stack a+1 , the corresponding storage size is Mem a 、Mem a+1 , a=1,2,…P-1;

[0045] If the two consecutive ring stacks are full at the same time for the first time, the time cutoff threshold is adjusted as follows:

[0046]

[0047] If the two consecutive ring stacks are full at the same time and this is not the first time, the time cutoff threshold is adjusted as follows:

[0048]

[0049] Among them, TimeCur is the time truncation threshold before adjustment; TimeAdj is the time truncation threshold after adjustment; TimeDis is the time difference between the time when the simultaneous full stack occurs this time and the time when the simultaneous full stack occurs last time.

[0050] The plurality of truncated data flow subsets are DataFlowSub1, DataFlowSub2, ...DataFlowSub K ; K ≥ 2;

[0051] The data storage subsystem sequentially stores the plurality of truncated data stream subsets into the multi-layer annular storage stack of the data warehouse, specifically comprising:

[0052] If P ≥ K, the data flow subset DataFlowSub will be truncated iter Store to the ring stack Stack iter , iter=1,2,…K; if P<K, then when iter is not greater than P, the data flow subset DataFlowSub will be truncated iter Store to the ring stack Stack iter ; When iter is greater than P, the data flow subset DataFlowSub will be truncated iter Store to ring stack , Represents the remainder when iter is divided by P.

[0053] As a further improvement, the system further includes: a data sending subsystem, which acquires the video frame data stored in the full-stack circular stack at one time and sends it to the light field imaging unit, and clears the full-stack circular stack.

[0054] The technical solution of the present invention adopts a data warehouse to store and transmit video frame data generated by light field camera array array acquisition, and the data warehouse is composed of a P-layer ring stack, the time series data flow set DataFlowSet to be transmitted includes N video frame data, each video frame data has a time node value; the N video frame data are sorted according to the size of the time node value in the time series data flow set DataFlowSet to be transmitted; the resolution size of the video frame data corresponding to different time nodes is not completely the same; based on the time truncation threshold TimeCur, the data flow of the time series data flow set DataFlowSet to be transmitted is truncated to obtain K truncated data flow subsets, each truncated data flow subset includes M video frame data, and the maximum time node value and the minimum time node value of the video frame data in each truncated data flow subset are The difference between the node values ​​is not greater than the time truncation threshold TimeCur, and it is determined whether there are two consecutive ring stacks of the P-layer ring stack that are full at the same time. If so, the size of the time truncation threshold TimeCur is reduced; the video frame data stored in the full ring stack is obtained at one time, and the full ring stack is cleared. This data transmission and sending method is well adapted to the data acquisition mode of the light field camera, that is, the problem of different data resolution sizes caused by different acquisition resolutions under different acquisition modes, avoids data transmission blockage and data overflow problems, and can avoid data transmission storm problems caused by frequent sending of individual or isolated sampling data; in addition, the truncation time threshold can be adjusted according to the full stack situation of the actual data storage ring stack, avoiding frequent full stacks of the ring stack, and finally achieving an ideal effect of feedback-balanced self-learning.

[0055] Further advantages of the present invention will be further reflected in detail in the specific embodiments section in conjunction with the drawings of the specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1It is a schematic diagram of the steps of a method for data transmission and storage in a data warehouse according to an embodiment of the present invention;

[0058] Figure 2 is a schematic diagram of the structure of data to be transmitted involved in various embodiments of the present invention;

[0059] Figure 3 is a schematic diagram of a ring stack structure involved in a data warehouse used in various embodiments of the present invention;

[0060] Figure 4 It is an architecture diagram of a data transmission and storage system for a data warehouse according to an embodiment of the present invention;

[0061] Figure 5 yes Figure 4 A schematic diagram of subsystem components of the data transmission and storage system for a data warehouse. DETAILED DESCRIPTION

[0062] The invention is further described below in conjunction with the accompanying drawings and specific implementation methods.

[0063] See also Figure 1 , Figure 1 It is a flowchart of the steps of a data transmission and storage method for a data warehouse according to an embodiment of the present invention.

[0064] exist Figure 1 In FIG. 1 , the method includes a cyclic judgment process of steps S100-S500, and each process is summarized as follows:

[0065] S100: Obtain a set of time series data streams to be transmitted;

[0066] S200: Based on a time truncation threshold, data stream truncation is performed on the set of time-series data streams to be transmitted to obtain K truncated data stream subsets;

[0067] S300: sequentially storing K truncated data stream subsets into a P-layer ring stack of a data warehouse;

[0068] S400: determining whether two consecutive ring stacks of the P-layer ring stack are simultaneously full, and if so, reducing the time cutoff threshold, and returning to step S200;

[0069] Otherwise, return to step S300.

[0070] Specifically, in the above step S100, a set of time series data flows to be transmitted DataFlowSet is obtained, and the set of time series data flows to be transmitted DataFlowSet includes N data.

[0071] As a specific type of data to be transmitted involved in various embodiments of the present invention, the set of time-series data flows to be transmitted DataFlowSet includes N video frame data.

[0072] See also Figure 2 The video frame data is acquired by a light field camera, which has multiple acquisition modes. The acquisition resolutions in different acquisition modes are different. The acquisition modes of the light field camera are different at different time nodes. Therefore, the resolutions of the video frame data corresponding to different time nodes are not completely the same.

[0073] by Figure 2 As an example, different video frames F1, F2, and Fn obtained by a light field camera in various acquisition modes are shown;

[0074] As an example, the different acquisition modes of the light field camera may be a plurality of different acquisition viewing angles.

[0075] More specifically, as a non-limiting example, the multiple different acquisition perspectives include:

[0076] A first acquisition viewing angle parallel to a first coordinate axis of the Cartesian coordinate system, a second acquisition viewing angle parallel to a second coordinate axis of the Cartesian coordinate system, and a third acquisition viewing angle parallel to a third coordinate axis of the Cartesian coordinate system.

[0077] by Figure 2 For example, at a time node (TimeID=1), video frame data F1 is acquired at a first resolution (1K*2K) in a Z-direction depth perspective (i.e., a third acquisition perspective parallel to a third coordinate axis (Z axis) of a Cartesian coordinate system);

[0078] At the time node (TimeID=2), the video frame data F2 is acquired at the second resolution (5K*2K) in the X-direction depth viewing angle (i.e., the third acquisition viewing angle parallel to the first coordinate axis (X axis) of the Cartesian coordinate system);

[0079] At the time node (TimeID=N), the video frame data F is acquired at the Nth resolution (5K*5K) in the Y-direction depth perspective (i.e., the third acquisition perspective parallel to the second coordinate axis (Y axis) of the Cartesian coordinate system). N .

[0080] Thus, a set of time-series data flows to be transmitted containing N video frame data is obtained, namely, DataFlowSet;

[0081] DataFlowSet={F1,F2,…,F N};

[0082] It can be seen that the N video frame data are sorted according to the size of the time node value in the time series data flow set DataFlowSet to be transmitted;

[0083] Since the acquisition modes of the light field camera are different at different time points, the resolutions of the video frame data corresponding to different time points are not completely the same;

[0084] At this time, the step S200 obtains the time truncation threshold TimeCur, and performs data flow truncation on the set of time-series data flows to be transmitted DataFlowSet based on the time truncation threshold TimeCur.

[0085] Correspondingly, each truncated data stream subset contains M video frame data, and the difference between the maximum time node value and the minimum time node value of the video frame data in each truncated data stream subset is not greater than the time truncation threshold TimeCur; the truncation threshold TimeCur is a positive integer;

[0086] The M, N, and K are all positive integers, and Indicates rounding down; K≥2;

[0087] Figure 3 A schematic diagram of a ring stack structure involved in a data warehouse used in various embodiments of the present invention is shown.

[0088] As mentioned above, the data warehouse is constructed using a P-layer ring stack, where P>2 and P is a positive integer.

[0089] For details, see Figure 3 The data warehouse includes a continuous P-layer ring stack from inside to outside {Stack1, Stack2, ...Stack P}, Circular Stack Stack j With a predetermined storage size Mem j , j=1,2,…P.

[0090] Figure 3 In FIG. 1 , the ID number of each ring stack is also shown, that is, the ring stack with ID=1 is Stack1, and its storage size is Mem1, and so on.

[0091] The storage characteristics of a stack are different from other common structures. It is characterized by "first in, last out" storage of data. The storage characteristics of a circular stack are that it is closed at both ends. Therefore, the data index is a continuous loop iteration and will not cause subscript overflow (i.e., data overflow).

[0092] When light field cameras generate data, different acquisition modes are usually cross-run, resulting in uncertain data size and quantity. If ordinary data structures are used, data overflow is likely to occur. At the same time, light field imaging obtains the latest (most recent or recent several) video frame data from the light field camera frame data each time, so it is necessary to ensure that the latest data is obtained each time. However, the "newness" of the data conflicts with the "data generation time". The first generated data is stored first, but becomes the "oldest" data. If ordinary storage structures, such as queues or arrays, are used, the earliest stored data is generally read first, thereby failing to achieve the technical effect of the first generated data being read first. Therefore, the present invention creatively adopts a stack storage method based on the imaging characteristics of light field cameras, and is a continuous P-layer ring stack from the inside to the outside {Stack1, Stack2, ...Stack P} storage method.

[0093] At the same time, the storage method of the ring stack adopted in the present invention is also closely related to the subsequent adjustment of the time cutoff threshold.

[0094] Specifically, when storing data, step S300 is specifically performed as follows:

[0095] The K truncated data flow subsets are DataFlowSub1, DataFlowSub2, ...DataFlowSub K ;

[0096] If P ≥ K, the data flow subset DataFlowSub will be truncated iter Store to the ring stack Stack iter , iter=1,2,…K; if P<K, then when iter is not greater than P, the data flow subset DataFlowSub will be truncated iter Store to the ring stack Stack iter ; When iter is greater than P, the data flow subset DataFlowSub will be truncated iter Store to ring stack , Represents the remainder when iter is divided by P.

[0097] The step S400 is performed as follows:

[0098] Let the two ring stacks that are full at the same time be Stack a and Stack a+1 , the corresponding storage size is Mem a 、Mem a+1 , a=1,2,…P-1;

[0099] Then the step S400 further includes:

[0100] If the two consecutive ring stacks are full at the same time for the first time, the time cutoff threshold TimeCur is adjusted as follows:

[0101]

[0102] If the two consecutive ring stacks are full at the same time and this is not the first time, the time cutoff threshold TimeCur is adjusted as follows:

[0103]

[0104] Among them, TimeAdj is the adjusted time cutoff threshold; TimeDis is the time difference between the time when the simultaneous full stack occurs this time and the time when the simultaneous full stack occurs last time.

[0105] Of course, sending data once when the stack is full can avoid frequent data sending.

[0106] The step S400 further includes: acquiring the video frame data stored in the full ring stack at one time, and clearing the full ring stack.

[0107] To achieve Figure 1 The method described, see Figure 4 , showing a data transmission and storage system for a data warehouse, the system being connected to a light field camera array.

[0108] exist Figure 4 In the embodiment, the light field camera array includes multiple light field cameras, each of which has multiple acquisition modes. The acquisition resolutions in different acquisition modes are different. The acquisition modes of the light field cameras are different at different time points. Therefore, the resolutions of the video frame data corresponding to different time points are not completely the same. For the specific format of the acquired video frame data, please refer to Figure 2 .

[0109] Figure 4 In the embodiment, the storage structure of the data transmission and storage system includes a data warehouse, which includes a multi-layer annular storage stack and is connected to a light field imaging unit.

[0110] In terms of the overall structure, when a full circular stack exists in the data warehouse, the video frame data stored in the full circular stack is acquired at one time and sent to the light field imaging unit, and the full circular stack is cleared;

[0111] Preferably, when there are two consecutive circular stacks in the data warehouse that are full at the same time, the video frame data stored in all the full circular stacks are acquired at one time and sent to the light field imaging unit, and the full circular stacks are cleared.

[0112] Figure 5 yes Figure 4 A schematic diagram of subsystem components of the data transmission and storage system for a data warehouse.

[0113] exist Figure 5 In the figure, it is shown that the system also includes:

[0114] A data receiving subsystem, wherein the data receiving subsystem receives the time series data generated by the light field camera array to form a set of time series data streams to be transmitted;

[0115] A data stream truncation subsystem, wherein the data stream truncation subsystem performs data stream truncation on the set of time-series data streams to be transmitted based on a time truncation threshold value to obtain multiple truncated data stream subsets, each truncated data stream subset comprising multiple video frame data, and the difference between the maximum time node value and the minimum time node value of the video frame data in each truncated data stream subset is not greater than the time truncation threshold value; the truncation threshold value is a positive integer;

[0116] A data storage subsystem, the data storage subsystem sequentially storing a plurality of truncated data stream subsets into a multi-layered ring storage stack of the data warehouse;

[0117] The truncation threshold adjustment subsystem determines whether two consecutive ring stacks of the multi-layer ring storage stack are full at the same time, and if so, reduces the time truncation threshold.

[0118] The data warehouse includes a continuous P-layer ring stack {Stack1, Stack2, ...Stack P}, Circular Stack j With a predetermined storage size Mem j , j = 1, 2, ... P;

[0119] The data sending subsystem acquires the video frame data stored in the full annular stack at one time and sends it to the light field imaging unit, and clears the full annular stack.

[0120] The time series data generated by the light field camera array is video frame data, each video frame data has a time node value; the multiple video frame data are sorted according to the size of the time node value in the set of time series data streams to be transmitted; the resolution size of the video frame data corresponding to different time nodes is not completely the same.

[0121] The truncation threshold adjustment subsystem determines whether two consecutive ring stacks of the multi-layer ring storage stack are full at the same time, and if so, reduces the time truncation threshold, specifically including:

[0122] Let the two ring stacks that are full at the same time be Stack aand Stack a+1 , the corresponding storage size is Mem a 、Mem a+1 , a=1,2,…P-1;

[0123] If the two consecutive ring stacks are full at the same time for the first time, the time cutoff threshold is adjusted as follows:

[0124]

[0125] If the two consecutive ring stacks are full at the same time and this is not the first time, the time cutoff threshold is adjusted as follows:

[0126]

[0127] Among them, TimeCur is the time truncation threshold before adjustment; TimeAdj is the time truncation threshold after adjustment; TimeDis is the time difference between the time when the simultaneous full stack occurs this time and the time when the simultaneous full stack occurs last time.

[0128] The plurality of truncated data flow subsets are DataFlowSub1, DataFlowSub2, ...DataFlowSub K ; K ≥ 2;

[0129] The data storage subsystem sequentially stores the plurality of truncated data stream subsets into the multi-layer annular storage stack of the data warehouse, specifically comprising:

[0130] If P ≥ K, the data flow subset DataFlowSub will be truncated iter Store to the ring stack Stack iter , iter=1,2,…K; if P<K, then when iter is not greater than P, the data flow subset DataFlowSub will be truncated iter Store to the ring stack Stack iter ; When iter is greater than P, the data flow subset DataFlowSub will be truncated iter Store to ring stack , Represents the remainder when iter is divided by P.

[0131] It can be seen that the present invention adopts a data warehouse to store and transmit the video frame data generated by the light field camera array array acquisition, and the data warehouse is composed of a P-layer ring stack, the time series data flow set DataFlowSet to be transmitted includes N video frame data, each video frame data has a time node value; the N video frame data are sorted according to the size of the time node value in the time series data flow set DataFlowSet to be transmitted; the resolution size of the video frame data corresponding to different time nodes is not exactly the same; based on the time truncation threshold TimeCur, the data flow of the time series data flow set DataFlowSet to be transmitted is truncated to obtain K truncated data flow subsets, each truncated data flow subset includes M video frame data, and the maximum time node value and the minimum time node value of the video frame data in each truncated data flow subset are The difference of the node values ​​is not greater than the time truncation threshold TimeCur, and it is determined whether there are two consecutive ring stacks of the P-layer ring stack that are full at the same time. If so, the size of the time truncation threshold TimeCur is reduced; the video frame data stored in the full ring stack is obtained at one time, and the full ring stack is cleared. This data transmission and sending method is well adapted to the data acquisition mode of the light field camera, that is, the problem of different data resolution sizes caused by different acquisition resolutions under different acquisition modes, avoids data transmission blockage and data overflow problems, and can avoid data transmission storm problems caused by frequent sending of individual or isolated sampling data; in addition, the truncation time threshold can be adjusted according to the full stack situation of the actual data storage ring stack, avoiding frequent full stacks of the ring stack, and finally achieving an ideal effect of feedback-balanced self-learning.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

[0133] The present invention does not particularly specify the structure of some modules, which shall be subject to the contents recorded in the prior art. The prior art mentioned in the above background technology section of the present invention can be used as part of the present invention to understand the meaning of some technical features or parameters. The scope of protection of the present invention shall be subject to the contents actually recorded in the claims.

Claims

1. A data transmission and storage method for a data warehouse, wherein the data warehouse is constructed using a P-layer ring stack, where P>2 and P is a positive integer; It is characterized in that The method comprises the following steps: S100: Obtain a set of time-series data flows to be transmitted DataFlowSet, wherein the set of time-series data flows to be transmitted DataFlowSet includes N video frame data, each of which has a time node value; the N video frame data are sorted in the set of time-series data flows to be transmitted DataFlowSet according to the size of the time node value; the resolutions of the video frame data corresponding to different time nodes are not completely the same; S200: Obtain a time truncation threshold TimeCur, and based on the time truncation threshold TimeCur, perform data flow truncation on the set of time-series data flows to be transmitted DataFlowSet to obtain K truncated data flow subsets, each of which contains M video frame data, and the difference between the maximum time node value and the minimum time node value of the video frame data in each truncated data flow subset is not greater than the time truncation threshold TimeCur; the truncation threshold TimeCur is a positive integer; The M, N, and K are all positive integers, and Indicates rounding down; K≥2; S300: sequentially storing the K truncated data stream subsets into the P-layer ring stack of the data warehouse; S400: Determine whether two consecutive ring stacks of the P-layer ring stack are full at the same time. If yes, reduce the time cutoff threshold TimeCur and return to step S200. Otherwise, return to step S300; The data warehouse includes a continuous P-layer ring stack {Stack1, Stack2, ...Stack P }, Circular Stack j With a predetermined storage size Mem j , j=1,2,…P.

2. A data transmission and storage method for a data warehouse as claimed in claim 1, characterized in that: The video frame data is acquired by a light field camera, and the light field camera has multiple acquisition modes, and the acquisition resolutions in different acquisition modes are different.

3. A data transmission and storage method for a data warehouse as claimed in claim 1, characterized in that: The K truncated data flow subsets are DataFlowSub1, DataFlowSub2, ...DataFlowSub K ; If P ≥ K, the data flow subset DataFlowSub will be truncated iter Store to the ring stack Stack iter , iter = 1, 2, ... K; If P < K, then when iter is not greater than P, the data flow subset DataFlowSub will be truncated iter Store to the ring stack Stack iter ; When iter is greater than P, the data flow subset DataFlowSub will be truncated iter Store to ring stack , Represents the remainder when iter is divided by P.

4. A data transmission and storage method for a data warehouse as claimed in claim 1, characterized in that: The step S400 further includes: acquiring the video frame data stored in the full ring stack at one time, and clearing the full ring stack.

5. A data transmission and storage method for a data warehouse as claimed in claim 1, characterized in that: Let the two ring stacks that are full at the same time be Stack a and Stack a+1 , the corresponding storage size is Mem a 、Mem a+1 , a=1,2,…P-1; Then the step S400 further includes: If the two consecutive ring stacks are full at the same time for the first time, the time cutoff threshold TimeCur is adjusted as follows: If the two consecutive ring stacks are full at the same time and this is not the first time, the time cutoff threshold TimeCur is adjusted as follows: Among them, TimeAdj is the adjusted time cutoff threshold; TimeDis is the time difference between the time when the simultaneous full stack occurs this time and the time when the simultaneous full stack occurs last time.

6. A data transmission and storage system for a data warehouse, the system being connected to a light field camera array, characterized in that: The system further comprises: A data warehouse, the data warehouse comprising a multi-layer circular storage stack; A data receiving subsystem, wherein the data receiving subsystem receives the time series data generated by the light field camera array to form a set of time series data streams to be transmitted; A data stream truncation subsystem, wherein the data stream truncation subsystem performs data stream truncation on the set of time-series data streams to be transmitted based on a time truncation threshold value to obtain multiple truncated data stream subsets, each truncated data stream subset comprising multiple video frame data, and the difference between the maximum time node value and the minimum time node value of the video frame data in each truncated data stream subset is not greater than the time truncation threshold value; the truncation threshold value is a positive integer; A data storage subsystem, the data storage subsystem sequentially storing a plurality of truncated data stream subsets into a multi-layered ring storage stack of the data warehouse; The truncation threshold adjustment subsystem determines whether two consecutive ring stacks of the multi-layer ring storage stack are full at the same time, and if so, reduces the time truncation threshold.

7. A data transmission and storage system for a data warehouse as claimed in claim 6, characterized in that: The data warehouse includes a continuous P-layer ring stack {Stack1, Stack2, ...Stack P }, Circular Stack j With a predetermined storage size Mem j , j = 1, 2, ... P; The time series data generated by the light field camera array is video frame data, each video frame data has a time node value; the multiple video frame data are sorted according to the size of the time node value in the set of time series data streams to be transmitted; the resolution size of the video frame data corresponding to different time nodes is not completely the same.

8. A data transmission and storage system for a data warehouse as claimed in claim 7, characterized in that: The truncation threshold adjustment subsystem determines whether two consecutive ring stacks of the multi-layer ring storage stack are full at the same time, and if so, reduces the time truncation threshold, specifically including: Let the two ring stacks that are full at the same time be Stack a and Stack a+1 , the corresponding storage size is Mem a 、Mem a+1 , a=1,2,…P-1; If the two consecutive ring stacks are full at the same time for the first time, the time cutoff threshold is adjusted as follows: If the two consecutive ring stacks are full at the same time and this is not the first time, the time cutoff threshold is adjusted as follows: Among them, TimeCur is the time truncation threshold before adjustment; TimeAdj is the time truncation threshold after adjustment; TimeDis is the time difference between the time when the simultaneous full stack occurs this time and the time when the simultaneous full stack occurs last time.

9. A data transmission and storage system for a data warehouse as claimed in claim 7, characterized in that: The plurality of truncated data flow subsets are DataFlowSub1, DataFlowSub2, ...DataFlowSub K ; K ≥ 2; The data storage subsystem sequentially stores the plurality of truncated data stream subsets into the multi-layer annular storage stack of the data warehouse, specifically comprising: If P ≥ K, the data flow subset DataFlowSub will be truncated iter Store to the ring stack Stack iter , iter = 1, 2, ... K; If P < K, then when iter is not greater than P, the data flow subset DataFlowSub will be truncated iter Store to the ring stack Stack iter ; When iter is greater than P, the data flow subset DataFlowSub will be truncated iter Store to ring stack , Represents the remainder when iter is divided by P.

10. A data transmission and storage system for a data warehouse as claimed in claim 7, characterized in that: The system further comprises: The data sending subsystem acquires the video frame data stored in the full annular stack at one time and sends it to the light field imaging unit, and clears the full annular stack.

Images