Method for optimizing synchronization server balancing in an unstructured data collection environment

Abstract

A synchronization server balancing method in an unstructured data collection environment comprises: a step of acquiring unstructured data for situational judgment; a step of extracting summary information in the form of natural language for situational understanding from the acquired unstructured data using a large language model (LLM); a step of calculating a similarity score by comparing the extracted summary information with previously stored knowledge information; a step of determining a message broker to transmit among a plurality of distributed message brokers based on the calculated similarity score; and a step of transmitting data containing summary information and feature vectors extracted from the unstructured data to the determined message broker.

Description

Technology Field

[0001] The present invention relates to a synchronization server balancing method, and more specifically, to a synchronization server balancing method in an unstructured data collection environment. Background Technology

[0003] With the growing interest in digital twins, there is an increasing demand for methodologies to efficiently synchronize real-world data with virtual space or storage.

[0004] With the advancement of artificial intelligence technology, such as the emergence of large AI models, attention is being paid to technology for storing and managing feature vectors extracted from unstructured data, such as images and point clouds, and technology for searching for similarity between feature vectors, going beyond the processing technology for structured data refined by humans as exemplified in Fig. 1.

[0005] Accordingly, while research is currently underway on vector databases for batch storing and retrieving feature vectors, methodologies for efficiently synchronizing real-time feature vector data are still lacking.

[0006] In the past, there was a limitation in that a static policy (e.g., Round Robin, Hash Partitioning, Sticky Partitioning, etc.) was fixed in a message broker that supports real-time data processing, as exemplified in Fig. 2, and only one-dimensional data analysis was possible.

[0007] Specifically, in a changing environment, there is a need for optimization policies that dynamically reflect the situation and can easily and flexibly incorporate expert knowledge, rather than simple static policies. However, in message brokers that support real-time data processing, there is currently a lack of methodologies capable of referencing situational information rather than the data itself and optimizing synchronization by incorporating expert knowledge; therefore, it is necessary to explore solutions for this. The problem to be solved

[0009] The present invention has been devised to solve the above-mentioned problems, and the objective of the present invention is to provide a synchronization server balancing method that can optimize the resources of a message broker by referring to situational information of an unstructured data collection point when transmitting data including a feature vector to a distributed message broker. means of solving the problem

[0011] A synchronization server balancing method in an unstructured data collection environment according to an embodiment of the present invention for achieving the above objective comprises: a step in which a system acquires unstructured data for situational judgment; a step in which a system extracts summary information in the form of natural language (text) for situational understanding from the acquired unstructured data using a large language model (LLM); a step in which a system calculates a similarity score by comparing previously stored knowledge information with the extracted summary information; a step in which a system determines a message broker to transmit among a plurality of distributed message brokers according to the calculated similarity score; and a step in which data including summary information and a feature vector extracted from the unstructured data is transmitted to the determined message broker.

[0012] And the knowledge information may be knowledge information related to synchronization performance.

[0013] In addition, the step of calculating the similarity score can calculate the similarity score by comparing the summary information extracted from natural language with the knowledge information written in natural language.

[0014] The step of determining the message broker to transmit may include: determining the policy of the message broker based on the calculated similarity score; and determining the message broker to transmit based on the determined policy.

[0015] In addition, the step of determining the policy of the message broker is characterized by determining a policy for the compression rate of data and a policy for caching efficiency, in a synchronization server balancing method in an unstructured data collection environment.

[0016] And the step of determining the message broker policy can determine a policy to adjust the individual size and batch set size of data to be sent to the message broker based on the similarity score when determining a policy for caching efficiency.

[0017] Additionally, the step of determining the message broker to transmit according to the determined policy may measure the resource usage of each message broker at preset intervals, and if the resource usage of a specific message broker exceeds a preset maximum threshold, update the policy to adjust the individual size of the data and the size of the batch set so that the load of the message broker whose resource usage exceeds the maximum threshold is distributed to other message brokers.

[0018] The step of determining the message broker to transmit according to the determined policy can measure the resource usage of each message broker at preset intervals, and if the resource usage of a specific message broker is below a preset minimum threshold, update the policy to adjust the individual size of the data and the size of the batch set so that the load of other message brokers is distributed to the message broker whose resource usage is below the minimum threshold.

[0019] In addition, unstructured data may be image information or point clouds acquired through LiDAR sensors.

[0020] Meanwhile, a synchronization server balancing system in an unstructured data collection environment according to another embodiment of the present invention includes: a communication unit that acquires unstructured data for situational judgment; and a processor that extracts summary information in the form of natural language (text) for situational understanding from the acquired unstructured data using a large language model (LLM), calculates a similarity score by comparing the extracted summary information with previously stored knowledge information, determines a message broker to transmit among a plurality of distributed message brokers according to the calculated similarity score, and transmits data including summary information and a feature vector extracted from the unstructured data to the determined message broker.

[0021] A synchronization server balancing method in an unstructured data collection environment according to another embodiment of the present invention comprises: a step in which a system extracts summary information in the form of natural language (text) for situational awareness from unstructured data by utilizing a large language model (LLM); a step in which a system calculates a similarity score by comparing the extracted summary information with knowledge information related to synchronization performance stored in advance; a step in which a system determines a policy for determining a message broker to transmit among a plurality of message brokers configured in a distributed manner according to the calculated similarity score; and a step in which a system determines a message broker to transmit according to the determined policy.

[0022] In addition, a synchronization server balancing system in an unstructured data collection environment according to another embodiment of the present invention includes: a communication unit that acquires unstructured data for situational judgment; and a processor that utilizes a large language model (LLM) to extract summary information in the form of natural language (text) for situational understanding from unstructured data, calculates a similarity score by comparing the extracted summary information with knowledge information related to synchronization performance stored in advance, determines a policy to determine a message broker to transmit among a plurality of message brokers configured in a distributed manner according to the calculated similarity score, and determines a message broker to transmit according to the determined policy. Effects of the invention

[0024] As described above, according to embodiments of the present invention, when transmitting data including a feature vector to a distributed message broker, the resources of the message broker can be optimized by referring to the situational information of the unstructured data collection point.

[0025] In addition, according to embodiments of the present invention, high-level policy setting becomes possible by utilizing contextual information, and expert knowledge written in a natural language basis can be used as a standard for policy, thereby allowing the basis for policy setting to be known and enabling flexible policy setting without the expert having to enter detailed system information. Brief explanation of the drawing

[0027] FIG. 1 is a diagram illustrating an example of a technology that stores and manages feature vectors extracted from unstructured data and searches for similarity between feature vectors; FIG. 2 is a diagram provided in the description of a conventional message broker; FIG. 3 is a diagram provided in the description of the configuration of a synchronization server balancing system in an unstructured data collection environment according to an embodiment of the present invention; FIG. 4 is a diagram provided in the description of a more detailed configuration of a processor according to an embodiment of the present invention; and FIG. 5 is a diagram provided in the description of a synchronization server balancing method in an unstructured data collection environment according to an embodiment of the present invention. Specific details for implementing the invention

[0028] The present invention will be described in more detail below with reference to the drawings. To clearly explain the invention, parts unrelated to the description have been omitted from the drawings, and in the drawings, the width, length, thickness, etc., of the components may be exaggerated for convenience.

[0029] FIG. 3 is a diagram provided to describe the configuration of a synchronization server balancing system in an unstructured data collection environment according to one embodiment of the present invention.

[0030] The synchronization server balancing system in an unstructured data collection environment according to the present embodiment (hereinafter collectively referred to as the 'system') is provided to optimize the resources of a message broker by referring to situational information of an unstructured data collection point when transmitting data including a feature vector to a distributed message broker.

[0031] To this end, the system may include a communication unit (100), a processor (200), and a storage unit (300).

[0032] The communication unit (100) is equipped with a communication module connected to a network and can periodically acquire unstructured data for situation determination from CCTV, LiDAR sensor, external server, etc. based on a time set by the user.

[0033] Here, unstructured data may be video information (sample video) collected from CCTVs, etc., or point clouds acquired through LiDAR sensors.

[0034] The storage unit (300) is provided to store programs and data necessary for the operation of the processor (200). Specifically, the storage unit (300) may store knowledge information related to synchronization performance.

[0035] Here, knowledge information related to synchronization performance may include words related to compression, information on compression efficiency, words related to caching, and information on caching efficiency.

[0036] The processor (200) can process all matters for optimizing the resources of the message broker by referring to the situation information of the unstructured data collection point when transmitting data including a feature vector to a distributed message broker.

[0037] Specifically, the processor (200) can extract summary information in the form of natural language (text) for situational awareness from unstructured data obtained by utilizing a large language model (LLM), and calculate a similarity score by comparing the extracted summary information with previously stored knowledge information.

[0038] And the processor (200) can determine a policy of a message broker based on a calculated similarity score, determine a message broker to transmit to based on the determined policy, and transmit data containing summary information and feature vectors extracted from unstructured data to the determined message broker through the communication unit (100).

[0039] FIG. 4 is a drawing provided for a more detailed description of the configuration of a processor (200) according to one embodiment of the present invention.

[0040] Referring to FIG. 4, the processor (200) may include a summary information extraction unit (210), a similarity calculation unit (220), a policy decision unit (230), and a message broker decision unit (240).

[0041] The summary information extraction unit (210) can extract summary information in the form of natural language (text) for situational understanding from unstructured data obtained by utilizing a large language model (LLM).

[0042] The similarity calculation unit (220) can calculate a similarity score by comparing the previously stored knowledge information with the extracted summary information.

[0043] Specifically, the similarity calculation unit (220) can calculate a similarity score by comparing summary information extracted in natural language and knowledge information entered in natural language.

[0044] The policy decision unit (230) can determine the policy of the message broker based on the calculated similarity score.

[0045] Specifically, when the policy decision unit (230) determines the policy of the message broker, it can determine the policy for the compression rate of the data and the policy for the caching efficiency.

[0046] For example, when the policy decision unit (230) determines a policy for caching efficiency, it may determine a policy to adjust the individual size of data to be sent to the message broker and the size of the batch set according to the similarity score.

[0047] The message broker determination unit (240) can determine the message broker to transmit according to the determined policy.

[0048] For example, the message broker determination unit (240) can measure the resource usage of each message broker at a preset period (a period set by the user) and, if the resource usage of a specific message broker exceeds a preset maximum threshold, update a policy to adjust the individual size of the data and the size of the batch set so that the load of the message broker whose resource usage exceeds the maximum threshold is distributed to other message brokers.

[0049] And the message broker determination unit (240) measures the resource usage of each message broker at preset intervals and, if the resource usage of a specific message broker is below a preset minimum threshold, can update a policy to adjust the individual size of the data and the size of the batch set so that the load of other message brokers is distributed to the message broker whose resource usage is below the minimum threshold.

[0050] Meanwhile, the processor (200) may further include a similarity verification unit (not shown) in addition to the aforementioned summary information extraction unit (210), similarity calculation unit (220), policy decision unit (230), and message broker decision unit (240).

[0051] The similarity verification unit (not shown) can verify the similarity score calculated by comparing the previously stored knowledge information and the extracted summary information using a large language model (LLM), and can reflect the verification result as training data for the similarity calculation unit (220).

[0052] FIG. 5 is a diagram provided to explain a synchronization server balancing method in an unstructured data collection environment according to an embodiment of the present invention.

[0053] The synchronization server balancing method in an unstructured data collection environment according to the present embodiment can be executed by the system described above with reference to FIGS. 3 and 4.

[0054] Referring to FIG. 5, the system can periodically acquire unstructured data (e.g., sample images) for situational judgment based on a time set by the user (pre-set period).

[0055] In addition, the system can extract summary information in the form of natural language (text) for context understanding from unstructured data obtained using a large language model (LLM) (S520), and calculate a similarity score by comparing the extracted summary information with previously stored knowledge information.

[0056] Additionally, the system calculates a similarity score by comparing summary information extracted from natural language and knowledge information written in natural language (S530), and determines a policy for a message broker based on the calculated similarity score (S540). Subsequently, the system can determine a message broker to transmit to based on the determined policy (S550).

[0057] For example, the system can determine a policy for adjusting the data batch size, compression rate, and batch set size, and determine the message broker to transmit according to the determined policy.

[0058] So far, preferred embodiments of the synchronization server balancing method and system in an unstructured data collection environment have been described in detail.

[0059] While the technology for storing and managing feature vectors extracted from existing unstructured data and searching for similarity between feature vectors selects a static policy of a message broker based on binary or literal information of data transmitted by images, lidar, etc., according to an embodiment of the present invention, a policy is selected based on situational information that cannot be known solely from text information by utilizing contextual information that can be periodically extracted from images and lidar, and a policy related to compression and batch transmission of a message broker can be derived by comparing contextual information extracted in natural language with expert knowledge written in natural language.

[0060] Through this, high-level policy setting becomes possible by utilizing contextual information, and expert knowledge written in a natural language can be used as a policy standard, allowing the basis for policy setting to be understood and enabling flexible policy setting without experts having to enter detailed system information.

[0061] Meanwhile, it goes without saying that the technical concept of the present invention may also be applied to a computer-readable recording medium containing a computer program that enables the device and method according to the present embodiment to perform their functions. Furthermore, the technical concept according to various embodiments of the present invention may be implemented in the form of computer-readable code recorded on a computer-readable recording medium. A computer-readable recording medium may be any data storage device that can be read by a computer and store data. For example, a computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, etc. Additionally, computer-readable code or a program stored on a computer-readable recording medium may be transmitted through a network connected between computers.

[0062] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention. Explanation of the symbols

[0064] 100 : Communications Department 200 : Processor 210 : Summary Information Extraction Unit 220 : Similarity Calculation Unit 230 : Policy-making department 240: Message broker decision unit 300 : Storage unit

Claims

Claim 1 The system comprises: a step of acquiring unstructured data for situational judgment; a step of the system extracting summary information in the form of natural language (text) for situational understanding from the acquired unstructured data using a large language model (LLM); a step of the system calculating a similarity score by comparing previously stored knowledge information with the extracted summary information; a step of the system determining a message broker to transmit among a plurality of distributed message brokers based on the calculated similarity score; and a step of transmitting data containing the summary information and feature vectors extracted from the unstructured data to the determined message broker; wherein the knowledge information is knowledge information related to synchronization performance, the step of calculating the similarity score calculates a similarity score by comparing the summary information extracted in natural language with the knowledge information written in natural language, and the step of determining the message broker to transmit includes a step of determining a policy of the message broker based on the calculated similarity score; and a step of determining a message broker to transmit based on the determined policy.The step of determining the policy of a message broker includes determining a policy for data compression rate and a policy for caching efficiency; the step of determining the policy of a message broker, when determining a policy for caching efficiency, determines a policy for adjusting the individual size of data to be transmitted to the message broker and the size of the batch set according to the similarity score; the step of determining the message broker to transmit according to the determined policy measures the resource usage of each message broker at preset intervals, and if the resource usage of a specific message broker exceeds a preset maximum threshold, updates a policy for adjusting the individual size of data and the size of the batch set so that the load of the message broker whose resource usage exceeds the maximum threshold is distributed to other message brokers; the step of determining the message broker to transmit according to the determined policy measures the resource usage of each message broker at preset intervals, and if the resource usage of a specific message broker is below a preset minimum threshold, updates a policy for adjusting the individual size of data and the size of the batch set so that the load of other message brokers is distributed to message brokers whose resource usage is below the minimum threshold; and the system utilizes a Large Language Model (LLM) to calculate the similarity score A synchronization server balancing method in an unstructured data collection environment characterized by verifying and utilizing the verification results as training data when calculating similarity. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 delete Claim 8 delete Claim 9 A synchronization server balancing method in an unstructured data collection environment, characterized in that, in claim 1, the unstructured data is a point cloud acquired through image information or a LiDAR sensor. Claim 10 A communication unit for acquiring unstructured data for situational judgment; and a processor that utilizes a large language model (LLM) to extract summary information in the form of natural language (text) for situational understanding from the acquired unstructured data, calculates a similarity score by comparing the extracted summary information with previously stored knowledge information, determines a message broker to transmit among a plurality of distributed message brokers based on the calculated similarity score, and transmits data containing the summary information and feature vectors extracted from the unstructured data to the determined message broker;...including, and the knowledge information is knowledge information related to synchronization performance, and the processor calculates a similarity score by comparing summary information extracted from natural language and knowledge information written in natural language, and when the processor determines the message broker to transmit, determines the policy of the message broker according to the calculated similarity score, and determines the message broker to transmit according to the determined policy, and when the processor determines the policy of the message broker, determines a policy regarding the data compression rate and a policy regarding caching efficiency, and when the processor determines the policy regarding caching efficiency, determines a policy for adjusting the individual size of data to be transmitted to the message broker and the size of the batch set according to the similarity score, and when the processor determines the message broker to transmit according to the determined policy, measures the resource usage of each message broker at preset intervals, and if the resource usage of a specific message broker exceeds a preset maximum threshold, updates the policy for adjusting the individual size of data and the size of the batch set so that the load of the message broker whose resource usage exceeds the maximum threshold is distributed to other message brokers, and when the processor determines the message broker to transmit according to the determined policy, the preset A synchronization server balancing system in an unstructured data collection environment, characterized by measuring the resource usage of each message broker at regular intervals, and if the resource usage of a specific message broker is below a preset minimum threshold, updating a policy to adjust the individual size of data and the size of batch sets so that the load of other message brokers is distributed to the message broker whose resource usage is below the minimum threshold, and the processor verifying the calculated similarity score using a large language model (LLM) and using the verification result as training data when calculating similarity. Claim 11 A step in which the system extracts summary information in the form of natural language (text) for context understanding from unstructured data by utilizing a large language model (LLM); a step in which the system calculates a similarity score by comparing the extracted summary information with knowledge information related to synchronization performance that is stored in advance; a step in which the system determines a policy for determining a message broker to transmit among a plurality of distributed message brokers based on the calculated similarity score; and a step in which the system determines a message broker to transmit according to the determined policy....including, and knowledge information is knowledge information related to synchronization performance, and the step of calculating a similarity score calculates a similarity score by comparing summary information extracted from natural language and knowledge information written in natural language, and the step of determining a policy determines a policy regarding the data compression rate and a policy regarding caching efficiency, and when determining a policy regarding caching efficiency, determines a policy to adjust the individual size of data to be transmitted to the message broker and the size of the batch set according to the similarity score, and the step of determining the message broker to transmit according to the determined policy measures the resource usage of each message broker at preset intervals, and if the resource usage of a specific message broker exceeds a preset maximum threshold, updates a policy to adjust the individual size of data and the size of the batch set so that the load of the message broker whose resource usage exceeds the maximum threshold is distributed to other message brokers, and the step of determining the message broker to transmit according to the determined policy measures the resource usage of each message broker at preset intervals, and if the resource usage of a specific message broker is below a preset minimum threshold, the load of other message brokers on the message broker whose resource usage is below the minimum threshold A synchronization server balancing method in an unstructured data collection environment, characterized by updating a policy to adjust the individual size of data and the size of batch sets so as to be distributed, and the system verifying the calculated similarity score using a large language model (LLM) and using the verification result as training data when calculating similarity. Claim 12 A communication unit for acquiring unstructured data for situational judgment; and a processor that utilizes a large language model (LLM) to extract summary information in the form of natural language (text) for situational understanding from unstructured data, calculates a similarity score by comparing the extracted summary information with knowledge information related to synchronization performance stored in advance, determines a policy for determining a message broker to transmit among a plurality of distributed message brokers based on the calculated similarity score, and determines a message broker to transmit based on the determined policy;...including, and the knowledge information is knowledge information related to synchronization performance; the processor calculates a similarity score by comparing summary information extracted from natural language and knowledge information written in natural language; when determining a policy, the processor determines a policy regarding the data compression rate and a policy regarding caching efficiency; when determining a policy regarding caching efficiency, the processor determines a policy for adjusting the individual size of data to be transmitted to the message broker and the size of the batch set according to the similarity score; when determining the message broker to transmit according to the determined policy, the processor measures the resource usage of each message broker at preset intervals, and if the resource usage of a specific message broker exceeds a preset maximum threshold, updates the policy for adjusting the individual size of data and the size of the batch set so that the load of the message broker whose resource usage exceeds the maximum threshold is distributed to other message brokers; and when determining the message broker to transmit according to the determined policy, the processor measures the resource usage of each message broker at preset intervals, and if the resource usage of a specific message broker is below a preset minimum threshold, sends another message to the message broker whose resource usage is below the minimum threshold A synchronization server balancing system in an unstructured data collection environment, characterized by updating a policy to adjust the individual size of data and the size of batch sets so that the load on the broker is distributed, and the processor verifying the calculated similarity score using a large language model (LLM) and using the verification result as training data when calculating similarity.

Images