Hierarchical data store based data management system and method of operation thereof

The hierarchical data store-based system addresses high costs and real-time data management challenges by converting original data into lightweight data for efficient uploading and synchronization, reducing network and infrastructure costs while ensuring real-time data transfer.

JP7587890B2Active Publication Date: 2024-11-21PARTRIDGE SYST INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023534416
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-12-02
Filing Date
2021-11-22
Publication Date
2024-11-21
Estimated Expiration
2041-11-22

AI Technical Summary

Technical Problem

Existing data management systems face high infrastructure and network costs for uploading large amounts of data, particularly in IoT environments, and struggle with real-time data processing and management.

Method used

A hierarchical data store-based system that includes an original data store and a lightweight data store, utilizing a data uploader to convert original data into lightweight data for efficient uploading, a data tracer for synchronization, and a data coordinator for predefined tasks, all operating in a pipelined manner to reduce network costs and ensure real-time data transfer.

Benefits of technology

The system significantly reduces infrastructure and network costs while ensuring real-time data uploading and management, improving weight reduction processing efficiency and minimizing direct data access, thus optimizing data sharing and synchronization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007587890000001
    Figure 0007587890000001
  • Figure 0007587890000002
    Figure 0007587890000002
  • Figure 0007587890000003
    Figure 0007587890000003
Patent Text Reader

Abstract

A hierarchical data store-based data management system and its operating method are provided. The data management system according to one embodiment of the present invention may include a data uploader that acquires original data from one or more data sources, converts the acquired original data into corresponding lightweight data through a lightweight process, and uploads the converted lightweight data to a lightweight data store. In response to detecting that specific original data has been uploaded to the original data store, the data tracer updates a mapping table with the upload location of the specific original data. This data management system can effectively reduce network costs for uploading data and infrastructure costs for sharing data.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a hierarchical data store-based data management system and an operating method thereof, and more particularly to a system for efficiently managing large volumes of data based on a hierarchical data store including an original data store and a lightweight data store, and an operating method thereof. [Background technology]

[0002] As IoT (Internet of Things) technology develops, there is a rapid increase in demand for systems that can process and manage large volumes of data generated in real time. For example, IoT systems such as autonomous vehicles and smart factories generate large volumes of sensing data in real time through various sensors, and there is a demand for systems that can efficiently process and manage such large volumes of sensing data.

[0003] In response to such demands, a system for uploading and managing large amounts of data to cloud storage has been proposed recently. The proposed system can guarantee excellent user accessibility by providing the ability to upload and download data anytime and anywhere.

[0004] However, uploading large amounts of data to cloud storage requires significant infrastructure construction costs and network costs, and due to issues with available network bandwidth, it is almost impossible to continuously upload large amounts of data to cloud storage in real time. Summary of the Invention [Problem to be solved by the invention]

[0005] The technical problem to be solved through one embodiment of the present invention is to provide a data management system and an operating method of the system that can reduce infrastructure construction costs for sharing data and network costs for uploading data.

[0006] Another technical problem to be solved through one embodiment of the present invention is to provide a system and an operating method of the system that can efficiently manage large amounts of data based on a hierarchical data store including an original data store and a lightweight data store.

[0007] Another technical problem to be solved by an embodiment of the present invention is to provide an apparatus and an operating method of the apparatus capable of ensuring real-time upload of large amounts of data.

[0008] Yet another technical problem to be solved by an embodiment of the present invention is to provide an apparatus and an operating method of the apparatus capable of performing optimal weight reduction processing according to the type of data.

[0009] Yet another technical problem to be solved through one embodiment of the present invention is to provide an apparatus and an operating method of the apparatus that can provide synchronization and tracing functions between original data and lightweight data.

[0010] Yet another technical problem to be solved by an embodiment of the present invention is to provide an apparatus and an operating method thereof that can reduce network costs for downloading original data.

[0011] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person skilled in the art to which the present invention pertains from the following description. [Means for solving the problem]

[0012] In order to solve the above technical problems, a data management system according to one embodiment of the present invention includes a data uploader that acquires original data from one or more data sources, converts the acquired original data into corresponding lightweight data through a lightweight processing, uploads the converted lightweight data to a lightweight data store, and registers upload information of the converted lightweight data in a mapping table, and a data tracer that, in response to detecting that specific original data has been uploaded to the original data store, queries the mapping table for an entry of lightweight data corresponding to the specific original data, and updates the upload location of the specific original data in the queried entry.

[0013] In one embodiment, the lightweight data store may be embodied as a store that is more user accessible than the original data store.

[0014] In one embodiment, the data uploader may perform the operations of obtaining the original data, converting it to the lightweight data, and uploading it to the lightweight data store in a pipelined manner.

[0015] In one embodiment, the data management system may further include a data coordinator that downloads the particular original data from the original data store and performs a predefined task using the particular original data.

[0016] In one embodiment, the data coordinator is configured to include a downloader that downloads the specific original data, and a task worker that performs the task, and the operations of the downloader and the task worker may be performed in a pipelined manner.

[0017] In order to solve the above technical problems, a data uploader according to one embodiment of the present invention includes a communication interface, a memory for storing one or more instructions, and a processor for acquiring original data by executing the one or more stored instructions, performing a first lightweight processing in response to a determination that the acquired original data is of a first type, and performing a second lightweight processing different from the first lightweight processing in response to a determination that the acquired original data is of a second type different from the first type, and uploading lightweight data converted through the first lightweight processing or the second lightweight processing via the communication interface.

[0018] In order to solve the above technical problems, a data upload method according to one embodiment of the present invention is a data upload method performed on a computing device, and includes the steps of acquiring original data, performing a first lightweight processing step in response to a determination that the acquired original data is of a first type, performing a second lightweight processing step different from the first lightweight processing step in response to a determination that the acquired original data is of a second type different from the first type, and uploading lightweight data converted through the first lightweight processing or the second lightweight processing.

[0019] In order to solve the above technical problem, a computer program according to one embodiment of the present invention can be stored on a computer-readable recording medium when combined with a computing device to execute the steps of acquiring original data, performing a first lightweight processing step in response to a determination that the acquired original data is of a first type, performing a second lightweight processing step different from the first lightweight processing step in response to a determination that the acquired original data is of a second type different from the first type, and uploading lightweight data converted through the first lightweight processing or the second lightweight processing.

[0020] In order to solve the above technical problems, a data coordinator according to one embodiment of the present invention is a data coordinator for a hierarchical data store including a lightweight data store and an original data store, and includes a communication interface, a memory for storing one or more instructions, and a processor for receiving an identifier of specific lightweight data stored in the lightweight data store via the communication interface by executing the one or more stored instructions, downloading original data corresponding to the specific lightweight data from the original data store using the received identifier, and performing a predefined task using the downloaded original data.

[0021] In order to solve the above technical problems, a data coordination method according to one embodiment of the present invention is a data coordination method performed in a computing device for a hierarchical data store including a lightweight data store and an original data store, and includes the steps of receiving an identifier of specific lightweight data stored in the lightweight data store, downloading original data corresponding to the specific lightweight data from the original data store using the received identifier, and performing a predefined task using the downloaded original data.

[0022] In order to solve the above technical problem, a computer program according to one embodiment of the present invention may be stored on a computer-readable recording medium to execute steps of receiving an identifier of specific lightweight data stored in a lightweight data store, downloading original data corresponding to the specific lightweight data from an original data store using the received identifier, and performing a predefined task using the downloaded specific original data. Effect of the Invention

[0023] According to the above-described embodiment of the present invention, original data may be converted into lightweight data through a lightweight process, and the converted lightweight data may be uploaded. This may ensure real-time uploading of data and may significantly reduce network costs for uploading data.

[0024] Also, by implementing only a lightweight data store as a store with high user accessibility (such as cloud storage), the cost of building infrastructure for data sharing can be significantly reduced.

[0025] In addition, by applying different lightweight algorithms according to the type and characteristics of the original data, rather than a simple compression method, the efficiency of the lightweight process can be improved.

[0026] In addition, by performing the lightening process, data uploading operation, etc. in a pipelining manner, the real-time nature of data uploading can be more reliably guaranteed.

[0027] In addition, by providing a data tracer that detects the upload of original data and tracks the upload location, the correspondence between the original data and the lightweight data can be effectively managed, and data synchronization between the original data and the lightweight data can be ensured.

[0028] In addition, by providing a data coordinator that downloads original data and performs predefined tasks, users can minimize the direct download of original data, which can significantly reduce network costs for downloading original data and infrastructure costs for the original data store.

[0029] Furthermore, by performing the download of original data and task execution operations in a pipelined manner, the task execution time can be significantly reduced.

[0030] The effects of the technical concept of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is an exemplary block diagram illustrating a hierarchical data store-based data management system according to an embodiment of the present invention. [Diagram 2] 1 is an exemplary diagram illustrating an implementation method of a data uploader according to an embodiment of the present invention; [Diagram 3] 1 is an exemplary diagram illustrating an implementation method of a data coordinator according to an embodiment of the present invention. [Figure 4] 4 is an exemplary flow chart illustrating a method of operation of a data uploader according to one embodiment of the present invention. [Diagram 5] 1 is an exemplary diagram for explaining a weighting processing method of a data uploader according to an embodiment of the present invention; [Figure 6] 1 is an exemplary diagram for explaining a weighting processing method of a data uploader according to an embodiment of the present invention; [Figure 7] 1 is an exemplary diagram for explaining a weighting processing method of a data uploader according to an embodiment of the present invention; [Figure 8] FIG. 2 is an exemplary diagram illustrating a pipelining scheme applicable to a data uploader according to an embodiment of the present invention. [Figure 9] FIG. 2 is an exemplary diagram illustrating a pipelining scheme applicable to a data uploader according to an embodiment of the present invention. [Figure 10] 4 is an exemplary flow chart illustrating a method of operation of a data tracer in accordance with one embodiment of the present invention. [Figure 11] 4 is an exemplary flow chart illustrating a method of operation of a data coordinator according to one embodiment of the present invention. [Figure 12]FIG. 2 is an exemplary diagram illustrating a pipelining method applicable to a data coordinator according to an embodiment of the present invention. [Figure 13] FIG. 2 is an exemplary diagram illustrating a pipelining method applicable to a data coordinator according to an embodiment of the present invention. [Figure 14] FIG. 1 illustrates an example computing device capable of implementing devices and / or systems according to various embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Specific structural or functional descriptions of embodiments in accordance with the inventive concepts disclosed herein are provided solely for purposes of illustrating embodiments in accordance with the inventive concepts, which may be embodied in various forms and are not limited to the embodiments described herein.

[0033] Since the embodiments according to the concept of the present invention can be modified in various ways and can have various forms, the embodiments are illustrated in the drawings and described in detail herein, but it is not intended to limit the embodiments according to the concept of the present invention to the particular disclosed forms, and includes modifications, equivalents, or alternatives within the spirit and technical scope of the present invention.

[0034] Terms such as "first" or "second" are used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another component, for example, a first component can be named a second component, and similarly, a second component can be named a first component, without departing from the scope of the inventive concept.

[0035] When an element is referred to as being "coupled" or "connected" to another element, it should be understood that although it may be directly coupled or connected to the other element, there may be other elements between the elements. Conversely, when an element is referred to as being "directly coupled" or "directly connected" to another element, it should be understood that there are no other elements between the elements. Expressions describing the relationship between elements, such as "between," "immediately between," or "directly adjacent to," should be interpreted in the same manner.

[0036] The terms used in this specification are merely used to describe certain embodiments and are not intended to limit the present invention. A singular expression includes a plural expression unless the context clearly indicates otherwise. In this specification, the terms "include" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preclude the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0037] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as idealized or overly formal unless expressly defined in this specification.

[0038] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or restricted by such embodiments. The same reference numerals in each drawing indicate the same elements.

[0039] FIG. 1 is an exemplary configuration diagram illustrating a hierarchical data store-based data management system according to an embodiment of the present invention.

[0040] As shown in FIG. 1, the data management system may be a system that uses a hierarchical data store 30 to manage original data obtained from one or more data sources 1-1, 1-2, . . . , 1-n.

[0041] Here, the data sources 1-1, 1-2, ..., 1-n are modules that generate or provide original data, and may be, for example, sensors that generate sensing data in real time, data models that generate predetermined data, or devices / systems equipped with them. As a more specific example, the data sources 1-1, 1-2, ..., 1-n may be Internet of Things (IoT) systems (or sensor-based systems) such as autonomous vehicles, smart homes, smart farms, smart factories, etc., or various sensors that constitute the systems. In this case, the data management system may function as a system that efficiently manages a large amount of sensing data generated in real time from various sensors using the hierarchical data store 30. For example, the data management system may function as a system that manages a large amount of sensing data generated in real time from various sensors of an autonomous vehicle using the hierarchical data store 30, and analyzes the sensing data through the target system 3 to verify normal operation of the autonomous vehicle. In this case, the target system 3 may be an artificial intelligence model trained to determine whether the operation of the autonomous vehicle is normal or not from the sensing data, or a computing device / system equipped with the model. Hereinafter, the reference numeral "1" will be used to collectively refer to one or more data sources 1-1, 1-2, ..., 1-n, or to refer to any data source (1-1 or 1-2 or ... or 1-n).

[0042] As shown in the figure, the data management system may include a data uploader 10, a hierarchical data store 30, a data tracer 50, and a data coordinator 70. However, this is merely a preferred embodiment for achieving the object of the present invention, and it goes without saying that some components may be added or deleted as necessary. In addition, it should be noted that each component (10, 50, 70, etc.) of the data management system shown in FIG. 1 represents a functional component that is functionally separated, and that multiple components may be embodied in a form that is integrated with each other in an actual physical environment.

[0043] Also, in an actual physical environment, each of the components (10, 50, 70, etc.) may be separated into multiple detailed functional elements or may be implemented in multiple physical computing devices. For example, a first function of the data uploader 10 may be implemented in a first computing device and a second function of the data uploader 10 may be implemented in a second computing device. Or, a particular function of the data uploader 10 may be implemented in multiple computing devices.

[0044] The computing device may be, for example, but is not limited to, a notebook computer, a desktop, a laptop, etc., and may include any type of device equipped with computer capabilities. For an example of a computing device, see Figure 14. Each component of the data management system will now be described.

[0045] The data uploader 10 may be a computing module or computing device having an upload function for the original data acquired from the data source 1. More specifically, the data uploader 10 may convert the original data acquired from the data source 1 into lightweight data through a lightweight process, and upload the converted lightweight data to the lightweight data store 31. By doing so, the network cost required for uploading data may be significantly reduced, and data uploading may be performed in real time even in an environment where a large amount of original data is generated. Hereinafter, for convenience of explanation, the data uploader 10 will be abbreviated as "uploader 10".

[0046] 1 shows an example in which the data source 1 and the uploader 10 have a many-to-one relationship. However, the data source 1 and the uploader 10 may be designed to have a one-to-one relationship or a many-to-many relationship. In addition, the uploader 10 may be embodied as a separate computing device separate from the data source 1, or may be embodied as a computing module mounted on the data source 1 or a system related to the data source 1.

[0047] The uploader 10 may be implemented as one or more computing devices (or modules), although the specific implementation may vary depending on the embodiment.

[0048] In one embodiment, the uploader 10 may be embodied as a single computing device. For example, in an environment where the load (e.g., computing cost) required of the uploader 10 is not large (e.g., when the amount of original data generated is small or the computing cost required for the lightening process is small), operations such as obtaining original data, lightening process, and uploading data may be performed together within a single computing device.

[0049] In another embodiment, the uploader 10 may be embodied as a plurality of computing devices. For example, as shown in FIG. 2, the uploader 10 may be embodied as a first computing device 11 that acquires original data (or original data chunks (ODC3, ODC4, etc.)) and a second computing device 13 that performs a lightening process and uploads light data (or light data chunks (LDC1, etc.)). In this case, the first computing device 11 and the second computing device 13 may be connected via a communication cable (crossover cable, direct cable, etc.), and the second computing device 13 may acquire the original data from the first computing device 11, for example, via a shared directory. However, this is not limited thereto. As another example, the uploader 10 may be embodied as a first computing device (e.g., 11) that acquires original data, a second computing device that performs a lightening process, and a third computing device that uploads data.

[0050] In yet another embodiment, the uploader 10 may be embodied as multiple computing devices, but may operate in different manners depending on the measured load. Specifically, when the measured load is equal to or less than a reference value, all operations of the uploader 10 may be performed by a single computing device, and when the measured load is equal to or greater than the reference value, operations of the uploader 10 may be performed by multiple computing devices. For example, when the measured load is equal to or greater than a reference value, the uploader 10 may operate as illustrated in FIG. 2.

[0051] The operation method of the uploader 10 will be described in more detail later with reference to Figures 4 and subsequent drawings. In the present invention, the operation method of the uploader 10 may also be named a "data upload method".

[0052] Referring again to FIG.

[0053] The hierarchical data store 30 may refer to a store in which lightweight data and original data are stored, and may include a lightweight data store 31, an original data store 35, and a mapping table 33, as shown.

[0054] The light data store 31 is a store in which light data is stored (uploaded), and may be a conceptually upper layer store. For example, light data uploaded by the data uploader 10 may be stored in the light data store 31. The light data store 31 may be used for reviewing or sharing data among multiple users, but the scope of the present invention is not limited thereto.

[0055] Next, the original data store 35 is a store where original data is stored (uploaded) and may be a conceptually lower-level store. For example, original data that is later uploaded by a user may be stored in the original data store 35.

[0056] Each of the stores 31 and 35 may be implemented in various forms, such as a file, a database, a cloud storage, a local storage, a blockchain network, etc. Therefore, the scope of the present invention is not limited by the implementation form of the stores 31 and 35.

[0057] In one embodiment, the lightweight data store 31 may be embodied as a store with relatively high user accessibility, and the original data store 35 may be embodied as a store with relatively low user accessibility. For example, the lightweight data store 31 may be embodied as a cloud storage that is easily accessible to a large number of users, and the original data store 35 may be embodied as a local storage (such as a local data center). In such a case, the infrastructure construction costs for sharing data may be significantly reduced.

[0058] 1 shows an example in which the hierarchical data store 30 is composed of two hierarchical data stores 31 and 35, but the scope of the present invention is not limited thereto, and the hierarchical data store 30 may be composed of three or more hierarchical levels. For example, the hierarchical data store 30 may be composed of an original data store (e.g., 35), a first lightweight data store (e.g., 31) in which first lightweight data is stored, and a second lightweight data store in which second lightweight data generated by performing additional lightweight processing (e.g., data summarization) on the first lightweight data is stored.

[0059] Next, the mapping table 33 is a table that stores the correspondence between the light data and the original data, and each entry of the mapping table 33 can store upload information of the light data and the original data.

[0060] The upload information of the lightweight data may include, but is not limited to, an identifier (ID, name, etc.) of the lightweight data, an upload location of the lightweight data, etc. The upload information of the original data may include, but is not limited to, an identifier of the original data, an upload location of the original data, etc. The lightweight data and the corresponding original data may share the same identifier, or a separate identifier may be assigned to each data depending on the implementation method.

[0061] The mapping table 33 may be designed and implemented in various ways. For example, the mapping table 33 may be implemented as a hash table having an identifier (ID, name, etc.) shared between the original data and the light data as a key, and an upload location of the light data or the original data as a value. However, the scope of the present invention is not limited thereto.

[0062] Next, the data tracer 50 may be a computing module or computing device having a tracing function for the original data and / or the lightweight data. Specifically, the data tracer 50 may monitor the original data store 35 to detect that specific original data is uploaded. In response, the data tracer 50 may update the mapping table 33 by recording the upload location of the specific original data in a related entry of the mapping table 33. As another example, the data tracer 50 may detect that specific lightweight data is uploaded to the lightweight data store 31, and in response, record the upload location of the specific lightweight data in a related entry of the mapping table 33. Hereinafter, for convenience of explanation, the data tracer 50 will be abbreviated as "tracer 50".

[0063] In one embodiment, the tracer 50 may perform data synchronization between the lightweight data store 31 and the original data store 35. For example, the tracer 50 may detect that specific original data has been deleted from the original data store 35, and in response thereto, delete the corresponding lightweight data from the lightweight data store 31. Alternatively, the tracer 50 may detect that specific lightweight data has been deleted from the lightweight data store 31, and in response thereto, delete the corresponding original data from the original data store 35. As another example, the tracer 50 may check the correspondence between the lightweight data store 31 and the original data store 35, and perform data synchronization based on the check result. For example, the tracer 50 may generate lightweight data for the specific original data and upload it to the lightweight data store 31 in response to the check result that lightweight data corresponding to the specific original data does not exist. In this embodiment, data synchronization may be performed periodically or aperiodically. For example, the tracer 50 may perform data synchronization when the available computing resources are equal to or greater than a reference value.

[0064] The method of operation of the tracer 50 will be described in more detail below with reference to Figure 10. In the present invention, the method of operation of the tracer 50 may also be termed a "data tracing method."

[0065] Next, the data coordinator 70 may be a computing module or computing device capable of performing a predefined task using data stored in the hierarchical data store 30. For example, the data coordinator 70 may download specific original data from the original data store 35 in response to a user request (command) and transmit the specific original data to the target system 3 to perform a predetermined task. In this case, the user's access to the original data store 35 to download the original data is minimized, and the network cost for downloading the original data can be significantly reduced. In this case, the data coordinator 70 is preferably located in a location physically close to the original data store 35 and connected to the original data store 35 via a line (such as a high-bandwidth dedicated line) that can ensure high-speed data download. The data coordinator 70 and the target system 3 may communicate with each other via an appropriate communication cable and / or communication method (such as Ethernet, LVDS, etc.). Hereinafter, for convenience of explanation, the data coordinator 70 will be abbreviated to "coordinator 70".

[0066] The tasks may include any processing operations using original data and / or lightweight data, such as, for example, training the target system 3 (e.g., training an artificial intelligence-based anomaly detection model), validating the data source 1 related system via the target system 3 (e.g., determining the operation anomalies of the data source 1 related system via an anomaly detection model), etc. Thus, the scope of the present invention is not limited to a particular type of task.

[0067] Meanwhile, the coordinator 70 may be implemented as one or more computing devices (or modules), although the specific implementation manner may vary depending on the embodiment.

[0068] In one embodiment, as shown in FIG. 3, the coordinator 70 may be embodied as a task worker 71 and a plurality of downloaders 73-75. The task worker 71 is a computing device that performs a predefined task, and may be connected to the plurality of downloaders 73-75 via a communication device such as a switch 72. The downloaders 73-75 download original data (or original data chunks (ODC)) from the original data store 35, and the task worker 71 transmits (injects) the downloaded original data (ODC) to the target system 3 to perform the predefined task. The task worker 71 may obtain the original data (ODC) from the plurality of downloaders 73-75, for example, via a shared directory, but the scope of the present invention is not limited thereto. For reference, FIG. 3 assumes an environment in which the data transmission speed to the target system 3 exceeds the data download speed. In a different environment, the number of task workers 71 and downloaders 73-75 may be designed differently from that illustrated in FIG. 3.

[0069] The operation method of the coordinator 70 will be described in more detail later with reference to Figures 11 to 13. In the present invention, the operation method of the coordinator 70 may also be named a "data coordinating method."

[0070] Meanwhile, at least a part of each component (e.g., 10) shown in FIG. 1 may represent software or hardware such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). However, the components are not limited to software or hardware, and may be configured to be in an addressable storage medium or to be executed by one or more processors. The functions provided in the components may be implemented by further subdivided components, or may be implemented by a single component that performs a specific function by combining multiple components.

[0071] A hierarchical data store based data management system according to an embodiment of the present invention has been described above with reference to Figures 1 to 3. Various operational methods that may be performed in the data management system will now be described in detail with reference to Figure 4 and subsequent drawings.

[0072] Each step of the method described below may be realized by one or more instructions executed by a processor of a computing device, and for ease of understanding, the description will be continued assuming the system configuration illustrated in Fig. 1. Therefore, when the subject of an operation of a particular step is omitted, it may be understood that the operation may be performed by the components (10, 50, 70, etc.) of the data management system illustrated in Fig. 1.

[0073] 4 is an exemplary flowchart showing an operation method of the data uploader 10 according to an embodiment of the present invention. However, this is merely a preferred embodiment for achieving the object of the present invention, and it goes without saying that some steps may be added or deleted as necessary.

[0074] As shown in Fig. 4, the operating method may start with step S100 of acquiring original data. For example, an uploader 10 may acquire the original data from one or more data sources 1. Such original data may be, for example, large volume data generated in real time, but is not limited thereto.

[0075] In step S120, the acquired original data is subjected to a weight reduction process, so that the original data can be converted into lightweight data. For example, the uploader 10 can perform a first weight reduction process in response to a determination that the acquired original data is a first type, and can perform a second weight reduction process different from the first weight reduction process in response to a determination that the acquired original data is a second type different from the first type. In other words, for the efficiency of the weight reduction process, different weight reduction algorithms can be applied depending on the type of the original data. The weight reduction process method will be described in more detail below.

[0076] In one embodiment, the original data may be a single image. For example, the original data may be a single view image generated by a single camera. In such a case, the uploader 10 may perform a first process to reduce the bit depth and / or the number of channels of the image (e.g., converting an 8-bit / 3-channel image to an 8-bit / 1-channel image). Alternatively, the uploader 10 may perform a second process to reduce the size of the image. Alternatively, the uploader 10 may perform a third process to convert a plurality of single images into a video image (e.g., an mp4 file) by applying a video compression algorithm (e.g., H.264). Alternatively, the uploader 10 may perform a weight reduction process based on various combinations of the above-mentioned first to third processes. For example, the uploader 10 may convert a single image into a video image (video) by sequentially performing the first process, the second process, and the third process.

[0077] In another embodiment, as shown in Fig. 5, the original data 21 may be a multi-image composed of a plurality of images. For example, the original data 21 may be a multi-view image captured by multiple cameras. In this case, the uploader 10 may concatenate the plurality of images 21 and perform a lightweight process on the concatenated image 23 to convert it into a lightweight image 25. In this case, the lightweight process may be performed, for example, by a single-image lightweight process method.

[0078] In yet another embodiment, as shown in FIG. 6, the original data 31 may be a message (i.e., a message object; msg) in a predefined format (see FIG. 6). In such a case, the uploader 10 may store one or more messages 31 in a message queue 33 (msgQ) and serialize the message queue 33 to convert it into binary data 35. The uploader 10 may also compress the binary data 33 to convert it into lightweight data 37. At this time, the compression may be performed based on a lossless-based compression algorithm (such as zip), but is not limited to this. The uploader 10 may perform the above-mentioned lightweight processing every time the message queue 33 is full, or may perform the lightweight processing at a preset cycle. The uploader 10 may also directly serialize one or more messages 31 without using the message queue 33.

[0079] In yet another embodiment, the original data may be data generated by a LIDAR (Light Imaging Detection and Ranging) or RADAR (Radio Detection And Ranging) sensor. In this case, the uploader 10 may convert the original data into an image in the form of a point cloud and perform a weight reduction process on the converted image to generate a video image, which is the first lightweight data. In this case, the weight reduction process may be performed, for example, by a single image weight reduction process method. Alternatively, the uploader 10 may compress the original data to generate the second lightweight data. Alternatively, the uploader 10 may generate both the first lightweight data and the second lightweight data from the original data.

[0080] In yet another embodiment, the original data may be a video image (image) composed of a plurality of frames. In such a case, the uploader 10 may perform a weight reduction process to delete some of the plurality of frames that are less important. In this case, the frames that are less important may be, for example, frames with little object movement, frames with a small number of objects, frames with a high overlap rate with the previous frame, frames that do not include an object of interest, etc., but are not limited thereto. In some cases, the uploader 10 may randomly delete a preset number of frames from the plurality of frames.

[0081] In yet another embodiment, as shown in FIG. 7, the original data 41 is an image, and the uploader 10 can apply differentiated compression quality (or degree of weight reduction) to each region (411, 413, etc.) of the original image 41. For example, the uploader 10 can detect an object region 411 from the original image 41, compress the object region 411 with high quality, and compress the background region 413 with low quality (see image 43). In this case, the detection of the object region 411 can be performed based on, for example, a computer vision algorithm or an artificial intelligence-based object detection model (CNN model, etc.), but is not limited thereto. As another example, the uploader 10 can detect a plurality of object regions from the original image 41, and apply differentiated compression quality according to the importance of the object regions. That is, a relatively high compression quality can be applied to a first object region with high importance, and a relatively low compression quality can be applied to a second object region with low importance. Here, the importance of an object region may be determined based on, for example, the size of the region (object), whether it is an object of interest, etc., but is not limited to this.

[0082] In yet another embodiment, the original data is an image, and the uploader 10 may perform the lightweight process based on a deep-learning model. In this case, the deep-learning model may be composed of an encoder and a decoder structure, where the encoder is trained to extract feature data from an input image, and the decoder may be a neural network trained to decode the feature data and output an image identical to the input image. In this case, the uploader 10 may extract feature data from the original image using the encoder, compress the extracted feature data, and convert it into lightweight data. Later, the lightweight data may be decompressed and decoded via a decoder to convert it into the original image.

[0083] In yet another embodiment, the uploader 10 may perform the weight reduction process based on a combination of the various embodiments described above.

[0084] Referring again to FIG.

[0085] In step S140, the lightweight data may be tagged with metadata. The metadata may include, but is not limited to, an identifier, size, and environmental data of the original data. The environmental data may include, but is not limited to, data related to the sensing environment, such as information on time, location (place), weather, climate, topography, road conditions, and the like. For reference, the uploader 10 may generate environmental data by collecting information provided on the Internet based on time and location information at the time of acquiring the original data. However, the environmental data may be generated but is not limited to the above.

[0086] In one embodiment, this step S140 may be omitted.

[0087] In step S160, the light data may be uploaded. Specifically, the uploader 10 may upload the light data to the light data store 31. Then, the uploader 10 may register upload information, such as an identifier of the light data and a storage location of the light data store 31, in the mapping table 33. For example, the uploader 10 may register the upload information in a new entry in the mapping table 33.

[0088] Meanwhile, according to one embodiment of the present invention, the above steps (operations) S100 to S160 may be performed in a pipelining manner by the uploader 10. By doing so, the real-time nature of data uploading can be more reliably guaranteed. Hereinafter, for ease of understanding, this embodiment will be described with reference to Figs. 8 and 9.

[0089] FIG. 8 conceptually illustrates an exemplary pipelining scheme that may be applied to uploader 10 according to one embodiment of the present invention.

[0090] As shown in Fig. 8, the uploader 10 can perform data acquisition operations 51-1 to 51-3, weight reduction processing operations 53-1 to 53-3, metadata tagging operations 55-1 to 55-3, and data upload operations 57-1 to 57-3 for each of the original data chunks ODC1, ODC2, and ODC3 in a pipelined manner. However, the basic unit constituting the pipeline may be designed in a larger or smaller unit than that illustrated in Fig. 8.

[0091] In such a case, the data upload may be more efficient by parallelizing the processing operations for the multiple original data chunks. For example, in step 3, the metadata tagging operation 55-1 for the first original data chunk ODC1, the lightening processing operation 53-2 for the second original data chunk ODC2, and the data retrieval operation 51-3 for the third original data chunk ODC3 may be performed in parallel.

[0092] Figure 9 shows an example of the pipelining method illustrated in Figure 8. Specifically, Figure 9 shows a process of pipelining when the uploader 10 is implemented in the configuration illustrated in Figure 2. Also, Figure 9 assumes that the first computing device 11 and the second computing device 13 constituting the uploader 10 communicate with each other via a shared directory. The following description will be given with reference to Figure 9.

[0093] In steps 1 and 2, the first computing device 11 may obtain the first original data chunk ODC1 and store it in the shared directory. Then, the second computing device 13 may obtain the first original data chunk ODC1 through the shared directory and convert the first original data chunk ODC1 into the first light data chunk LDC1 through a lightening process. The second computing device 13 may monitor the shared directory and obtain the original data chunk every time the original data chunk is stored in the shared directory.

[0094] In step 3, the first computing device 11 may obtain the second original data chunk ODC2 and store it in the shared directory, and the second computing device 13 may tag the first light data chunk LDC1 with metadata M. These two operations may be performed in parallel.

[0095] In step 4, the second computing device 13 can convert the second original data chunk ODC2 into a second light data chunk LDC2 and upload the first light data chunk (LDC1+M). These two operations can be performed in parallel.

[0096] In step 5, the first computing device 11 may obtain the third original data chunk ODC3 and store it in the shared directory, and the second computing device 13 may tag the second light data chunk LDC2 with metadata M. These two operations may be performed in parallel.

[0097] As above, a data upload method according to an embodiment of the present invention has been described with reference to FIGS. 4 to 9. According to the above-described method, original data may be converted into lightweight data through a lightweight process, and the converted lightweight data may be uploaded. As a result, real-time data uploading may be guaranteed, and network costs for data uploading may be significantly reduced. In addition, real-time data uploading may be more reliably guaranteed by performing the lightweight process, data uploading, etc. in a pipelining manner. Furthermore, efficiency of the lightweight process may be improved by applying different lightweight algorithms according to the type and characteristics of the original data, rather than a simple compression method.

[0098] A method of operation of the tracer 50 according to one embodiment of the present invention will now be described with reference to FIG.

[0099] 10 is an exemplary flowchart showing an operation method of the tracer 50 according to one embodiment of the present invention. However, this is merely a preferred embodiment for achieving the object of the present invention, and it goes without saying that some steps may be added or deleted as necessary.

[0100] 10, the method may begin with step S200 of monitoring the original data store 35. For example, the tracer 50 may perform continuous monitoring to detect when original data is uploaded to the original data store 35.

[0101] In step S210, it may be determined whether an upload of specific original data has been detected.

[0102] In steps S220 and S230, in response to a determination that an upload has been detected, the mapping table 33 may be queried using an identifier of the particular original data, and the presence or absence of a related entry may be determined depending on the results of the query.

[0103] In step S240, in response to determining that the queried entry does not exist, the particular original data may be converted to lightweight data, and the converted lightweight data may be uploaded to the lightweight data store 31. As before, the tracer 50 may convert the particular original data into the corresponding lightweight data by performing an appropriate lightweight process depending on the type of the original data.

[0104] In step S250, in response to determining that the referenced entry exists, the mapping table 33 may be updated by recording the upload information (e.g., upload location) of the original data in the referenced entry. The tracer 50 may also update the referenced entry whenever the upload location of a particular original data item is changed.

[0105] On the other hand, if step S250 is performed after step S240, upload information such as an identifier of specific original data and an upload location may be registered in a new entry of the mapping table 33.

[0106] The operation method of the tracer 50 according to an embodiment of the present invention has been described above with reference to Fig. 10. As described above, the tracer 50 detects the upload of original data and traces the upload location, so that the correspondence between the original data and the light data can be effectively managed.

[0107] Hereinafter, a method of operation of the coordinator 70 according to an embodiment of the present invention will be described with reference to FIGS.

[0108] 11 is an exemplary flowchart showing an operation method of the coordinator 70 according to one embodiment of the present invention. However, this is merely a preferred embodiment for achieving the object of the present invention, and it goes without saying that some steps may be added or deleted as necessary.

[0109] 11, the operation method may start with step S300 of receiving a data identifier (such as an original data identifier or a lightweight data identifier). For example, the coordinator 70 may receive a shared identifier between the original data and the lightweight data from a user.

[0110] In one embodiment, multiple tasks may be predefined and each task may be assigned a task identifier, in which case the coordinator 70 may receive the task identifier in addition to the data identifier.

[0111] In step S320, the specific original data corresponding to the received data identifier can be downloaded. For example, the coordinator 70 can query the mapping table 33 with the received identifier to confirm the upload location information of the specific original data, and can download the specific original data from the original data store 35 using the confirmed upload location information.

[0112] In step S340, a predefined task may be performed using the specific original data. For example, if the predefined task is learning the target system 3, the coordinator 70 may transmit (inject) the specific original data into the target system 3 to perform the learning task.

[0113] In one embodiment, the coordinator 70 may further receive a task identifier, in which case the coordinator 70 may use the particular master data to perform (execute) the task corresponding to the task identifier.

[0114] In step S360, the execution result of the task may be provided. For example, the coordinator 70 may provide the execution result of the task to the user.

[0115] Meanwhile, according to an embodiment of the present invention, the download operation and data transmission operation (or task execution operation) of the coordinator 70 may be performed in a pipelined manner. In this way, the download of original data and the execution of tasks may be performed more efficiently. In the following, this embodiment will be described with reference to Figs. 12 and 13 for ease of understanding.

[0116] 12 conceptually illustrates an exemplary pipelining scheme that may be applied to a coordinator 70 in accordance with one embodiment of the present invention. In particular, FIG. 12 assumes that the coordinator 70 is implemented as shown in FIG.

[0117] As shown in Fig. 12, the operations of the multiple downloaders 73-75 and the task worker 71 may be performed in a pipelining manner. Specifically, by pipelining, the multiple downloaders 73-75 may perform an operation of providing the first multiple original data chunks ODC1-ODC3 to the task worker 71 and an operation of downloading the second multiple original data chunks ODC4-ODC6 in parallel (see step 2). Also, by pipelining, the multiple downloaders 73-75 may perform an operation of providing the second multiple original data chunks ODC4-ODC6 to the task worker 71, an operation of downloading the third multiple original data chunks ODC7-ODC9, and an operation of the task worker 71 transmitting the first multiple original data chunks ODC1-ODC3 to the target system 3 in parallel (see step 3).

[0118] Figure 13 shows an example of the pipelining method exemplified in Figure 12. Specifically, Figure 13 shows the process of pipelining when the coordinator 70 is embodied with n downloaders (72, 76, etc.) and one task worker 71. Also, Figure 13 assumes that the task worker 71 and the downloaders (72, 76, etc.) communicate with each other via a shared directory. The following description will be given with reference to Figure 13.

[0119] In step 0, an initial setting for downloading original data may be performed. Specifically, an identifier may be assigned to each downloader (72, 76, etc.), and a list of original data chunks may be shared between the downloaders (72, 76, etc.) and the task worker 71. Such initial setting may be performed by a user, but the scope of the present invention is not limited thereto. Also, the identifiers of the downloaders (72, 76, etc.) may be assigned, for example, as sequential values ​​(such as sequential values ​​from 1 to n), but are not limited thereto. However, in the following, for ease of understanding, the description will be continued on the assumption that the identifiers of the downloaders (72, 76, etc.) are assigned sequentially.

[0120] In step 1, a plurality of downloaders (72, 76, etc.) can download original data chunks (ODC1, ODCn, etc.) from the original data store 35. At this time, each downloader (72, 76, etc.) can download an original data chunk (ODC1, ODCn, etc.) corresponding to its own identifier. For example, when the total number of target original data chunks is 2n, the first downloader 72 can download the first data chunk (ODC1, etc.) and the n+1 data chunk (ODCn+1, etc.), and the n-th downloader 76 can download the n-th data chunk (ODCn, etc.) and the 2n-th data chunk (ODC2n, etc.). That is, each downloader (72, 76, etc.) can download an original data chunk (ODC1, ODCn, etc.) whose modular operation value matches its own identifier. By doing so, duplicate downloads of original data chunks can be prevented.

[0121] In addition, multiple downloaders (72, 76, etc.) can store downloaded original data chunks (ODC1, ODCn, etc.) in a shared directory, and the task worker 71 can monitor that the original data chunks (ODC1, ODCn, etc.) are stored in the shared directory.

[0122] In step 2, the multiple downloaders (72, 76, etc.) can download the next original data chunk (ODCn+1, ODC2n, etc.) from the original data store 35. Then, the task worker 71 can retrieve the stored original data chunk (ODC1, ODC2, etc.) from the shared directory and transmit it to the target system 3. These two operations can be performed in parallel.

[0123] For reference, the task worker 71 can also perform data transmission operations and memory loading operations in parallel by loading a second original data chunk (e.g., ODC2) onto memory while transmitting a first original data chunk (e.g., ODC1).

[0124] The operation method of the coordinator 70 according to an embodiment of the present invention has been described above with reference to Figures 11 to 13. As described above, the coordinator 70 downloads original data and performs predefined tasks, thereby minimizing the need for users to directly download original data. This can significantly reduce network costs for downloading original data, and can also significantly reduce infrastructure construction costs for the original data store. Furthermore, the operation of the coordinator 70 is performed in a pipelined manner, thereby significantly reducing task execution time.

[0125] Referring now to FIG. 14, an exemplary computing device 200 capable of implementing devices (eg, uploader 10) and / or systems according to various embodiments of the present invention will now be described.

[0126] FIG. 14 is an exemplary hardware configuration diagram of a computing device 200.

[0127] As shown in Fig. 14, a computing device 200 may include one or more processors 210, a bus 250, a communication interface 270, a memory 230 for loading a computer program 291 executed by the processor 210, and a storage 290 for storing the computer program 291. However, Fig. 14 shows only components related to the embodiment of the present invention. Therefore, a person skilled in the art to which the present invention pertains will understand that other general components may be included in addition to the components shown in Fig. 14.

[0128] The processor 210 can control the overall operation of each component of the computing device 200. The processor 210 can be configured to include a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), a graphic processing unit (GPU), or any other type of processor well known in the art of the present invention. The processor 210 can also perform calculations for at least one application or program for performing operations / methods according to embodiments of the present invention. The computing device 200 can include one or more processors.

[0129] In turn, the memory 230 can store various data, instructions and / or information. The memory 230 can load one or more programs 291 from the storage 290 to perform operations / methods according to embodiments of the present invention. The memory 230 can be implemented as a volatile memory such as a RAM, although the scope of the present invention is not limited in this respect.

[0130] The bus 250 may then provide a communication function between the components of the computing device 200. The bus 250 may be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

[0131] Next, the communication interface 270 can support wired and wireless Internet communication of the computing device 200. The communication interface 270 can also support various communication methods other than Internet communication. To this end, the communication interface 270 can be configured to include a communication module well known in the art of the present invention.

[0132] Next, the storage 290 may non-temporarily store the one or more programs 291. The storage 290 may include a non-volatile memory such as a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, or any other type of computer-readable recording medium well known in the technical field to which the present invention belongs.

[0133] In turn, the computer program 291 may include one or more instructions that, when loaded into the memory 230, cause the processor 210 to perform an operation / method according to an embodiment of the present invention. That is, the processor 210 executes the one or more instructions to perform the operation / method according to an embodiment of the present invention.

[0134] For example, the computer program 291 may include instructions to perform an operation of acquiring original data, an operation of performing a first lightening process in response to a determination that the acquired original data is of a first type, an operation of performing a second lightening process different from the first lightening process in response to a determination that the acquired original data is of a second type different from the first type, and an operation of uploading lightweight data converted through the first lightening process or the second lightening process. In this case, the uploader 10 according to an embodiment of the present invention may be embodied through the computing device 200.

[0135] As another example, the computer program 291 may include instructions for performing an operation of querying an entry of light data corresponding to the specific original data from the mapping table 33 in response to detecting that specific original data has been uploaded to the original data store 35, and an operation of updating an upload location of the specific original data in the query entry. In this case, the tracer 50 according to an embodiment of the present invention may be embodied through the computing device 200.

[0136] As yet another example, the computer program 291 may include instructions for receiving an identifier of a particular lightweight data stored in the lightweight data store 31, downloading original data corresponding to the particular lightweight data from the original data store 35 using the received identifier, and performing a predefined task using the downloaded original data. In this case, the coordinator 70 according to an embodiment of the present invention may be embodied through the computing device 200.

[0137] Above, with reference to FIG. 14, an exemplary computing device 200 capable of implementing devices and / or systems according to various embodiments of the present invention has been described.

[0138] The technical ideas of the present invention described above with reference to Figures 1 to 14 may be embodied as computer readable codes on a computer readable medium. The computer readable recording medium may be, for example, a portable recording medium (CD, DVD, Blu-ray disc, USB storage device, portable hard disk) or a fixed recording medium (ROM, RAM, computer-installed hard disk). The computer program recorded on the computer readable recording medium may be transmitted to another computing device via a network such as the Internet and installed in the other computing device, and thus may be used in the other computing device.

[0139] Although all components constituting the embodiments of the present invention have been described as being combined into one or operating in combination, the technical idea of ​​the present invention is not necessarily limited to such an embodiment, that is, all components may be selectively combined into one or more and operate within the scope of the purpose of the present invention.

[0140] Although the figures show operations in a particular order, it should not be understood that the operations must be performed in the particular order or sequentially shown, or that all of the operations shown must be performed to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various components in the above-described embodiments should not be understood as necessarily requiring such separation, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0141] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art will understand that the present invention can be embodied in other specific forms without changing the technical idea or essential features of the present invention. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting. The scope of protection of the present invention should be interpreted according to the appended claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the technical ideas defined by the present invention.

Claims

1. a data uploader that acquires original data from one or more data sources, converts the acquired original data into corresponding lightweight data through a lightweight process, uploads the converted lightweight data to a lightweight data store, and registers upload information of the converted lightweight data in a mapping table; a data tracer that, in response to detecting that specific original data has been uploaded to the original data store, queries the mapping table for an entry of lightweight data corresponding to the specific original data, and updates the query entry with an upload location of the specific original data; The data tracer comprises: In response to determining that the queried entry does not exist, a data management system generates lightweight data corresponding to the specific original data through a lightweight process, uploads the generated lightweight data to the lightweight data store, and registers upload information of the generated lightweight data in the mapping table.

2. The upload information includes a shared identifier between the original data and the corresponding lightweight data; The data management system of claim 1 , wherein the data tracer queries the mapping table using a shared identifier of the particular original data.

3. The data management system of claim 1 , wherein the lightweight data store is implemented as a store having higher user accessibility than the original data store.

4. The data uploader: The data management system of claim 1 , configured to include a first computing device that obtains the original data, and a second computing device that performs the weighting process.

5. The data uploader:

2. The data management system according to claim 1, wherein the operation of obtaining the original data, the operation of converting the original data into the lightweight data, and the operation of uploading the original data to the lightweight data store are performed in a pipelined manner.

Citation Information

Patent Citations

  • System and method for distributing book content

    JP2002203117A

  • Image processing apparatus, server device, information processing apparatuses, their control methods, program making computer device execute these control methods, and storage medium storing these programs to be readable in computer device

    JP2005102222A

  • Image providing system and control method thereof, and program

    JP2018180812A

  • Data storage management operations in a secondary storage subsystem using image recognition and image-based criteria

    US20170083517A1