A robot real-time state log storage and retrieval method under mass data

Abstract

The application provides a robot real-time state log storage and retrieval method under mass data, and the design method comprises the following steps: using ElasticSearch to store key fields in a log file, reducing resource consumption caused by the system when saving the file; using HBase to store all fields in the log file, using the advantage of the underlying HDFS file system to reduce the retrieval time; using Java language to develop a background system, processing log file information reported by the robot in real time, and providing Restful API to the outside, so as to facilitate users to customize search conditions, retrieve and download log files. Through the technical scheme of the application, the robot real-time state log under the background of mass data can be stored and retrieved more quickly.

Description

Technical Field

[0001] This invention relates to the field of big data storage technology, and in particular to a method for storing and retrieving real-time status logs of robots under massive data conditions. Background Technology

[0002] Data retrieval is a common method for querying data, allowing users to retrieve desired data resources from large datasets based on specific criteria. The core of this method is storing certain data resources in a database, allowing users to input search criteria for analysis. With the continuous development of artificial intelligence technology, robots have been widely applied in various fields, such as industrial production, medical services, and household life. To ensure the safe and stable operation of robots, it is necessary to monitor and record their real-time status. However, as robot operating time increases, the amount of status log data generated also increases dramatically. Each robot generates hundreds of thousands of real-time status logs per day, making the effective storage and retrieval of this massive amount of data a pressing problem.

[0003] Existing storage and retrieval methods often encounter problems when processing massive amounts of data, such as low data storage efficiency and slow retrieval speed. Furthermore, some existing storage and retrieval methods also frequently face challenges such as complex data management and inefficient use of storage space when handling large volumes of data. Summary of the Invention

[0004] This invention proposes a method for storing and retrieving real-time status logs of robots under massive data conditions, which can effectively solve the problems of low data storage efficiency and slow retrieval speed in the context of big data.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for storing and retrieving real-time status logs of robots under massive data conditions employs the distributed search and analysis engine ElasticSearch for storing and retrieving key data, and the distributed and scalable non-relational database HBase for storing and retrieving complete data. The method is characterized by the following specific steps:

[0007] Step 1: Create a source data log file description object. Elasticsearch only stores key fields to reduce the resource consumption of Elasticsearch when saving files. HBase stores all fields and takes advantage of its underlying HDFS file system to reduce retrieval time. Since most retrieval operations use the device unique code and time as retrieval conditions, the combination of device unique code and creation time is selected as RowKey.

[0008] Step 2: Parse the log files reported by the machine, and develop a backend service project using Java and the Spring Boot framework to receive messages sent by the robot;

[0009] Step 3: Retrieve logs based on the conditions and download the log files. The backend service mentioned in Step 2 will provide a RESTful API to the user. The user can enter the relevant conditions in the API and download the log files.

[0010] As a further aspect of the present invention, in step 2, the robot reports logs via the MQTT protocol, and the background stores the data in ElasticSearch and HBase.

[0011] As a further solution of the present invention, in order to prevent data inconsistency in single-machine mode, a cluster deployment is adopted. The master node is responsible for writing data and the slave nodes are responsible for reading data. At the same time, load balancing is adopted to reduce the pressure of reading data on a single slave node. If the master node fails due to network or excessive writing pressure, another master node can be selected from multiple slave nodes.

[0012] As a further aspect of the present invention, the retrieval conditions in step 3 are as follows: the device unique code device_id is a required field, and other conditions can be omitted. The log level has the following options: ERROR, DEBUG, WARN, and INFO. By default, logs of all levels will be retrieved. The start time and end time are a range query, and by default, log files from the most recent seven days will be retrieved.

[0013] As a further aspect of the present invention, specifically, during retrieval, the key information of the log file, namely the device unique code and creation time, is first retrieved from ElasticSearch based on keywords. Then, all the information of the log file is retrieved from HBase by combining RowKey. Some log information will be displayed in a table format on the cloud platform. Users can also click the export file button on the page to obtain all the retrieved log files.

[0014] As a further aspect of the present invention, the analysis engine ElasticSearch is based on Lucene, and the basic unit for storing data is an index. It consists of an architecture of three nodes, and the three nodes are mutually redundant.

[0015] As a further aspect of the present invention, the writing process of the analysis engine ElasticSearch is as follows: the client randomly selects a Node, which is called a proxy node. Then, the proxy node routes to the corresponding node according to the document, waits for the main shard to finish writing, and the slave shard to finish writing synchronously, that is, the CP model is completed and returned to the client.

[0016] As a further aspect of the present invention, the reading process of the analysis engine ElasticSearch is as follows: the client randomly selects a Node, which is called the proxy node, and then obtains the main shard and all slave shards through hash routing by document ID. The round-robin random polling algorithm is then used to load the document onto any shard, which is then returned to the proxy node and then back to the client.

[0017] The beneficial effects of this invention are as follows:

[0018] The method of this invention uses the distributed search and analysis engine ElasticSearch to store and retrieve key data, and the distributed scalable non-relational database HBase to store and retrieve complete data. Through the technical solution of this invention, the real-time status logs of robots in the context of massive data can be stored and retrieved more quickly. Attached Figure Description

[0019] Figure 1 The flowchart illustrates a method for storing and retrieving real-time status logs of a robot under massive data conditions, as provided in this embodiment of the invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0021] Reference Figure 1 This invention provides a method for storing and retrieving real-time status logs of robots under massive data conditions. It employs the distributed search and analysis engine ElasticSearch for storing and retrieving key data, and the distributed, scalable, non-relational database HBase for storing and retrieving complete data. The method includes the following steps:

[0022] Step 1: Create source data log file description objects. Elasticsearch only stores key fields to reduce resource consumption when saving files. The structure of the Elasticsearch file description object is shown in Table 1 below. HBase stores all fields, taking advantage of its underlying HDFS file system to reduce retrieval time. The HBase file description object is shown in Table 2 below. In practice, since most retrieval operations use the device unique code and time as retrieval conditions, this invention selects the combination of device unique code and creation time as RowKey.

[0023] Table 1 Elasticsearch Storage Fields

[0024] Data Items Remark device_id Unique device code level Log levels create_time Creation time

[0025] Table 2 HBase Storage Fields

[0026] Data Items Remark id Log file unique ID device_id Unique device code device_status Equipment status product_id Equipment Model name Log name module Log module message Log content IP Device IP level Log levels map_id Equipment location map ID map_name Equipment location map name create_time Creation time

[0027] Step 2: Parse the log files reported by the machine. This invention uses Java and the Spring Boot framework to develop a backend service project to receive messages sent by the robot. Specifically, the robot reports logs via the MQTT protocol. Its development team creates a fixed log reporting topic, and the backend service subscribes to the relevant topic. After receiving the load, it stores the data in ElasticSearch and HBase. Preferably, to prevent data inconsistency caused by service crashes in single-machine mode, both ElasticSearch and HBase are deployed in a master-slave configuration. The master node is responsible for writing data, and the slave nodes are responsible for reading data. At the same time, load balancing is used to reduce the pressure on a single slave node to read data. If the master node fails due to network issues or excessive write pressure, another master node can be selected from multiple slave nodes.

[0028] Step 3: Retrieve logs based on conditions and download log files. The backend service described in Step 2 will provide a RESTful API to the user. The user can enter the relevant conditions in the API to download the log files. The retrieval conditions are shown in Table 3. Among them, the device unique code device_id is a required field, and other conditions can be omitted. The log level has the following options: ERROR, DEBUG, WARN, and INFO. By default, logs of all levels will be retrieved. The start time and end time are a range query. By default, log files from the most recent seven days will be retrieved. In particular, during the retrieval, the key information of the log file, namely the device unique code and creation time, will be retrieved from Elasticsearch based on keywords. Then, the complete information of the log file will be retrieved from HBase by combining RowKey. Some log information will be displayed in a table on the cloud platform. Users can also click the export file button on the page to obtain all retrieved log files.

[0029] Table 3 Search criteria

[0030] Search criteria Remark device_id Unique device code level Log levels start_time Start time end_time End time

[0031] The Elasticsearch analytics engine is based on Lucene. Its basic data storage unit is the index, and it consists of a three-node architecture, with each node mutually backing up the others. The Elasticsearch write process works as follows: the client randomly selects a node, called the proxy (assistant) node. The proxy node then routes data to the corresponding node based on the document, waiting for the main shard to finish writing and for the secondary shard to also complete synchronously (CP model completion), before returning the result to the client. During this process, as data continuously enters the buffer and translog, buffer data is continuously written to new segment files (memory). Each refresh clears the buffer, but the translog is retained. As this process continues, the translog grows larger and larger. When the translog reaches a certain length, a commit operation is triggered.

[0032] The first step of a commit operation is to refresh the existing data in the buffer to the OS cache and clear the buffer. Then, a commit point is written to a disk file, which identifies all the segment files corresponding to this commit point. At the same time, all the data currently in the OS cache is forcibly fsynced to the disk file. Finally, the existing translog file is cleared and a new translog is restarted. At this point, the commit operation is complete. If the server crashes, only 5 seconds of translog logs are lost, and the segment file cache can be recovered through the translog.

[0033] The reading process of the analysis engine ElasticSearch: The client randomly selects a Node, which is called the proxy node. Then, it uses the document ID to perform hash routing to obtain the main shard and all the slave shards. It then performs a round-robin random polling algorithm to load the document onto any shard, retrieves the document, returns it to the proxy node, and then returns it to the client.

[0034] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for storing and retrieving real-time status logs of a robot under massive data conditions, characterized in that, The distributed search and analysis engine ElasticSearch is used for storing and retrieving key data, while the distributed and scalable NoSQL database HBase is used for storing and retrieving complete data. The specific steps are as follows: Step 1: Create a source data log file description object. Elasticsearch only stores key fields to reduce the resource consumption of Elasticsearch when saving files. HBase stores all fields and takes advantage of its underlying HDFS file system to reduce retrieval time. Since most retrieval operations use the device unique code and time as retrieval conditions, the combination of device unique code and creation time is selected as RowKey. Step 2: Parse the log files reported by the machine, and develop a backend service project using Java and the Spring Boot framework to receive messages sent by the robot; Step 3: Retrieve logs based on conditions and download log files. The backend service mentioned in Step 2 will provide a RESTful API to the user. The user can enter the relevant conditions in the API to download the log files. In step 3, the following search conditions are required: the device unique code device_id is a required field, and other conditions can be omitted. The log level has the following options: ERROR, DEBUG, WARN, and INFO. By default, logs of all levels will be retrieved. The start time and end time are a range query. By default, log files from the last seven days will be retrieved. During the retrieval process, the system first retrieves key information about the log file, namely the device unique code and creation time, from ElasticSearch based on keywords. Then, it retrieves all the log file information from HBase by combining the RowKey. Some log information will be displayed in a table format on the cloud platform. Users can also click the "Export File" button on the page to obtain all the retrieved log files.

2. The method for storing and retrieving real-time robot status logs under massive data as described in claim 1, characterized in that, In step 2, the robot reports logs via the MQTT protocol, and the background stores the data in ElasticSearch and HBase.

3. The method for storing and retrieving real-time robot status logs under massive data as described in claim 2, characterized in that, To prevent data inconsistency in single-machine mode, a cluster deployment is adopted. The master node is responsible for writing data, and the slave nodes are responsible for reading data. At the same time, load balancing is used to reduce the pressure on a single slave node to read data. If the master node fails due to network or excessive write pressure, a new master node can be selected from multiple slave nodes.

4. The method for storing and retrieving real-time robot status logs under massive data as described in claim 1, characterized in that, The analysis engine, ElasticSearch, is based on Lucene. The basic unit for storing data is the index, which consists of an architecture of three nodes, and each of the three nodes is mutually redundant.

5. The method for storing and retrieving real-time robot status logs under massive data as described in claim 4, characterized in that, The writing process of the ElasticSearch analytics engine is as follows: The client randomly selects a Node, which is called the proxy node. The proxy node then routes to the corresponding node based on the document, waits for the main shard to finish writing, and for the slave shard to also finish writing synchronously. That is, the CP model is completed and returned to the client.

6. The method for storing and retrieving real-time robot status logs under massive data as described in claim 4, characterized in that, The reading process of the ElasticSearch analytics engine is as follows: The client randomly selects a Node, which is called the proxy node. Then, it uses the document ID to perform hash routing to obtain the main shard and all the slave shards. A round-robin random polling algorithm is then used to load the document onto any shard, which is then returned to the proxy node and then back to the client.

Images