Robot log data analysis method based on big data and storage medium

Abstract

The invention relates to a robot log data analysis method based on big data and a storage medium. The method comprises the following steps: collecting log data of a robot; log data in the message queue Kafka is transmitted to a data paste source layer of the distributed file system to be stored; cleaning and processing log data in a data source layer of the distributed file system, and extracting detail data of a robot to a data detail layer for storage; analyzing and applying the detail data in the data detail layer, and extracting related indexes into the MySQL database; and carrying out visual display on the stored indexes by adopting Superset. According to the scheme, an operator can conveniently and quickly gather, store and analyze the log data of the robot, large-scale and unstructured log data of the robot can be processed, and the log data can be quickly analyzed and converted into indexes to be displayed.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot log data analysis method and storage medium based on big data. Background Art

[0002] With the rapid development of artificial intelligence and robotics technology, robots are increasingly used in industry, medical care, services and other fields, and the number of robots is increasing day by day. During the operation of the robot system, a large amount of log data will be generated. These data record the robot's operating status, data collected by sensors, issued control instructions and other information. How to analyze this data and provide guidance for the optimization direction of the robot is a concern of robot companies.

[0003] The currently widely used data analysis architecture mainly uses traditional structured databases for storage and analysis, but this is difficult to handle robot log data with complex structures.

[0004] Another way is to analyze through the unstructured database MongoDB. Write a script to import log files into MongoDB and analyze them through MongoDB's aggregation framework. However, this analysis method has limited performance for distributed and large-volume data, and the processing speed is greatly limited by machine performance. Summary of the invention

[0005] In view of the above problems, the present application provides a robot log data analysis method and storage medium based on big data to solve the problem that the existing method of storing and analyzing robot log data through structured databases is difficult to process complex robot log data.

[0006] To achieve the above purpose, the inventor provides a robot log data analysis method based on big data, comprising the following steps:

[0007] Use Flume+Kafka architecture to collect robot log data;

[0008] Transfer the log data in the message queue Kafka to the data source layer of the distributed file system for storage;

[0009] The Spark / Flink+regularization architecture is used to clean and process the log data in the data source layer of the distributed file system, and the robot's detailed data is extracted and stored in the data detail layer;

[0010] Analyze and apply the detailed data in the data detail layer, and extract relevant indicators into the MySQL database;

[0011] Superset is used to visualize the indicators stored in the MySQL database.

[0012] In some embodiments, the use of the Flume+Kafka architecture to collect robot log data specifically includes the following steps:

[0013] Configure the relevant environment of the log collection system in the log server according to the custom Flume_Config;

[0014] Create a Topic in Kafka Server based on the custom Flume_Config;

[0015] Generate data collection conf file based on custom Flume_Config loading template;

[0016] Create a data collection execution script based on the custom Flume_Config;

[0017] Send the data collection conf file and data collection execution script to the server that needs to be collected according to the custom Flume_Config;

[0018] Configure sensitive policies to filter sensitive information;

[0019] Start the robot log collection task to collect robot log data.

[0020] In some embodiments, the step of transferring the log data in the message queue Kafka to the data source layer of the distributed file system for storage specifically includes the following steps:

[0021] Generate data storage conf file by customizing Flume_Config loading template;

[0022] Generate data storage execution script by customizing Flume_Config loading template;

[0023] Send the generated data storage conf file and data storage execution script to the server that needs to be collected;

[0024] Start the robot log data storage task;

[0025] Call the quality inspection task to compare the source file and storage file of the log collection, and issue an alarm when the log generation is abnormal.

[0026] In some embodiments, the Spark / Flink+regularization architecture is used to clean and process the log data in the data source layer of the distributed file system, and extract the robot's detailed data to the data detail layer for storage, specifically including the following steps:

[0027] Storing the robot's dimensional data in the form of slowly changing dimensions, the dimensional data including device information and spatial information;

[0028] Use Spark SQL + regular expressions to clean the log data and extract the gateway protocol data.

[0029] Parse the robot gateway protocol data and extract relevant data, including date, equipment and sensor;

[0030] The extracted relevant data is collided with the stored dimension data to generate a gateway data detail table;

[0031] Save the cleaned data to HDFS.

[0032] In some embodiments, the analysis and application of the detailed data in the data detail layer and the extraction of relevant indicators into the MySQL database specifically include the following steps:

[0033] Based on the business, build a demand bus matrix and design indicator types and models;

[0034] Based on the demand bus matrix, the cleaned data is analyzed to obtain statistical indicator information of relevant dimensions;

[0035] Use Spark to build models by combining relevant dimension statistical indicator information;

[0036] Output model results to MySQL database.

[0037] In another embodiment, a storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are performed:

[0038] Use Flume+Kafka architecture to collect robot log data;

[0039] Transfer the log data in the message queue Kafka to the data source layer of the distributed file system for storage;

[0040] The Spark / Flink+regularization architecture is used to clean and process the log data in the data source layer of the distributed file system, and the robot's detailed data is extracted and stored in the data detail layer;

[0041] Analyze and apply the detailed data in the data detail layer, and extract relevant indicators into the MySQL database;

[0042] Superset is used to visualize the indicators stored in the MySQL database.

[0043] In some embodiments, the use of the Flume+Kafka architecture to collect robot log data specifically includes the following steps:

[0044] Configure the relevant environment of the log collection system in the log server according to the custom Flume_Config;

[0045] Create a Topic in Kafka Server based on the custom Flume_Config;

[0046] Generate data collection conf file based on custom Flume_Config loading template;

[0047] Create a data collection execution script based on the custom Flume_Config;

[0048] Send the data collection conf file and data collection execution script to the server that needs to be collected according to the custom Flume_Config;

[0049] Configure sensitive policies to filter sensitive information;

[0050] Start the robot log collection task to collect robot log data.

[0051] In some embodiments, the step of transferring the log data in the message queue Kafka to the data source layer of the distributed file system for storage specifically includes the following steps:

[0052] Generate data storage conf file by customizing Flume_Config loading template;

[0053] Generate data storage execution script by customizing Flume_Config loading template;

[0054] Send the generated data storage conf file and data storage execution script to the server that needs to be collected;

[0055] Start the robot log data storage task;

[0056] Call the quality inspection task to compare the source file and storage file of the log collection, and issue an alarm when the log generation is abnormal.

[0057] In some embodiments, the Spark / Flink+regularization architecture is used to clean and process the log data in the data source layer of the distributed file system, and extract the robot's detailed data to the data detail layer for storage, specifically including the following steps:

[0058] Storing the robot's dimensional data in the form of slowly changing dimensions, the dimensional data including device information and spatial information;

[0059] Use Spark SQL + regular expressions to clean the log data and extract the gateway protocol data.

[0060] Parse the robot gateway protocol data and extract relevant data, including date, equipment and sensor;

[0061] The extracted relevant data is collided with the stored dimension data to generate a gateway data detail table;

[0062] Save the cleaned data to HDFS.

[0063] In some embodiments, the analysis and application of the detailed data in the data detail layer and the extraction of relevant indicators into the MySQL database specifically include the following steps:

[0064] Based on the business, build a demand bus matrix and design indicator types and models;

[0065] Based on the demand bus matrix, the cleaned data is analyzed to obtain statistical indicator information of relevant dimensions;

[0066] Use Spark to build models by combining relevant dimension statistical indicator information;

[0067] Output model results to MySQL database.

[0068] Different from the existing technology, the above technical solution collects the robot's log data through the Flume+Kafka-based architecture, then transmits the log data in the message queue Kafka to the data source layer of the distributed file system, and then uses the Spark / Flink+regular architecture to clean and process the log data in the data source layer of the distributed file system, extracts the robot's detailed data to the data detail layer for storage, analyzes and applies the detailed data in the data detail layer, extracts related indicators to the MySQL database, and then uses Superset to visualize the indicators stored in the MySQL database. This solution allows operators to quickly aggregate, store and analyze the robot's log data, can process large-scale, unstructured robot log data, and can quickly analyze and convert it into indicators for display.

[0069] The above-mentioned records related to the invention content are only an overview of the technical solution of the present application. In order to enable ordinary technicians in the field to more clearly understand the technical solution of the present application, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned purpose and other purposes, features and advantages of the present application easier to understand, the following is an explanation in combination with the specific implementation mode and drawings of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The drawings are only used to illustrate the principles, implementation methods, applications, characteristics and effects of the specific embodiments of the present application and other related contents, and shall not be considered as limitations of the present application.

[0071] In the drawings of the specification:

[0072] Figure 1 A schematic diagram of a flow chart of a robot log data analysis method based on big data according to a specific implementation method;

[0073] Figure 2 A schematic diagram of a process of collecting robot logs according to a specific implementation method;

[0074] Figure 3 A schematic diagram of a process of storing robot logs according to a specific implementation method;

[0075] Figure 4 A schematic diagram of a process of cleaning robot logs according to a specific implementation method;

[0076] Figure 5 A schematic diagram of a process of robot log analysis described in a specific implementation method;

[0077] Figure 6 Another flowchart of the robot log data analysis method based on big data described in the specific implementation method is shown;

[0078] Figure 7 A schematic diagram of the structure of the storage medium described in the specific implementation method.

[0079] The reference numerals in the above drawings are described as follows:

[0080] 710. Storage medium,

[0081] 720. Processor. DETAILED DESCRIPTION

[0082] In order to explain in detail the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0083] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.

[0084] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.

[0085] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.

[0086] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.

[0087] Without further limitations, in this application, the words "include", "comprises", "has" or other similar expressions used in the sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0088] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.

[0089] In the description of the embodiments of the present application, space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the referred device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0090] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms such as "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0091] The explanations of some important terms involved in the following embodiments are as follows:

[0092] 1.Flume: A log collection system that provides high availability, high reliability, and distributed log transmission capabilities.

[0093] 2.HDFS: A distributed file system that provides the ability to uniformly manage file systems distributed on a cluster.

[0094] 3.Kafka: A distributed messaging system that provides the ability to store large amounts of data in real time.

[0095] 4.MySQL: An open source relational database management system (RDBMS).

[0096] See also Figure 1, this embodiment provides a robot log data analysis method based on big data, comprising the following steps:

[0097] Step S110 of collecting robot log data: using the Flume+Kafka architecture to collect the robot log data;

[0098] Robot log data storage step S120: transferring the log data in the message queue Kafka to the data source layer of the distributed file system for storage;

[0099] Step S130 of cleaning robot log data: using the Spark / Flink+regularization architecture to clean and process the log data in the data source layer of the distributed file system, extracting the robot's detailed data to the data detail layer for storage;

[0100] Robot log data analysis step S140: Analyze and apply the detailed data in the data detail layer, and extract relevant indicators into the MySQL database;

[0101] Step S150 of displaying robot indicators: using Superset to visualize the indicators stored in the MySQL database.

[0102] The robot's log data is collected through the Flume+Kafka-based architecture, and then the log data in the message queue Kafka is transmitted to the data source layer of the distributed file system. Then, the Spark / Flink+regular architecture is used to clean and process the log data in the data source layer of the distributed file system, extract the robot's detailed data to the data detail layer for storage, analyze and apply the detailed data in the data detail layer, extract related indicators to the MySQL database, and then use Superset to visualize the indicators stored in the MySQL database. This solution allows operators to quickly aggregate, store and analyze the robot's log data.

[0103] In some embodiments, the use of the Flume+Kafka architecture to collect robot log data specifically includes the following steps:

[0104] Configure the relevant environment of the log collection system in the log server according to the custom Flume_Config;

[0105] Create a Topic in Kafka Server based on the custom Flume_Config;

[0106] Generate data collection conf file based on custom Flume_Config loading template;

[0107] Create a data collection execution script based on the custom Flume_Config;

[0108] Send the data collection conf file and data collection execution script to the server that needs to be collected according to the custom Flume_Config;

[0109] Configure sensitive policies to filter sensitive information;

[0110] Start the robot log collection task to collect robot log data.

[0111] The robot log collection process is as follows Figure 2 As shown, the specific steps include:

[0112] Step S10: Robot log collection process, please refer to Figure 2 The steps specifically include the following steps:

[0113] Step S11: Specify the server configuration by customizing Flume_Config and install the related environment of Flume with one click.

[0114] Step S12: By customizing Flume_Config, a Topic is automatically created in the Kafka Server to store log messages.

[0115] Step S13: Generate a conf file by customizing the Flume_Config loading template. The default is File-Kafka, and the source_type used is spooldir. It can realize real-time monitoring of log files and real-time monitoring of polling log files. The default channel is memory mode.

[0116] Step S14: Automatically create an execution script by customizing Flume_Config;

[0117] Step S15: Send the conf and script files to the server that needs to be collected through the custom Flume_Config;

[0118] Step S16: The collection strategy is set to not include sensitive information. If there is any sensitive information, desensitization is performed;

[0119] Step S17: Start the robot log collection task.

[0120] In some embodiments, the step of transferring the log data in the message queue Kafka to the data source layer of the distributed file system for storage specifically includes the following steps:

[0121] Generate data storage conf file by customizing Flume_Config loading template;

[0122] Generate data storage execution script by customizing Flume_Config loading template;

[0123] Send the generated data storage conf file and data storage execution script to the server that needs to be collected;

[0124] Start the robot log data storage task;

[0125] Call the quality inspection task to compare the source file and storage file of the log collection, and issue an alarm when the log generation is abnormal.

[0126] Robot logs are stored as Figure 3 As shown, the specific steps include:

[0127] Step S21: Generate a conf file by customizing the Flume_Config loading template. The default is Kafka-HDFS. The partitioning strategy of data in HDFS uses the time field for directory division by default. In order to process log data faster in the cluster, the consistent hashing algorithm hash(key)=((a*key+b)%p)%m is used here, where a and b are random numbers, p is a large prime number, and m is the number of slots. This method can load balance the log data.

[0128] Step S22: Automatically create an execution script by customizing Flume_Config.

[0129] Step S23: Send the conf and script files to the server that needs to be collected through the custom Flume_Config.

[0130] Step S24: Start the robot log storage task.

[0131] Step S25: call the quality inspection task, compare the size and records of the source file and storage file of the log collection, set the alarm threshold, and notify in time when the log generation is abnormal.

[0132] In some embodiments, the Spark / Flink+regularization architecture is used to clean and process the log data in the data source layer of the distributed file system, and extract the robot's detailed data to the data detail layer for storage, specifically including the following steps:

[0133] Storing the robot's dimensional data in the form of slowly changing dimensions, the dimensional data including device information and spatial information;

[0134] Use Spark SQL + regular expressions to clean the log data and extract the gateway protocol data.

[0135] Parse the robot gateway protocol data and extract relevant data, including date, equipment and sensor;

[0136] The extracted relevant data is collided with the stored dimension data to generate a gateway data detail table;

[0137] Save the cleaned data to HDFS.

[0138] Robot log cleaning Figure 4 As shown, the specific steps include:

[0139] Step S31: Store robot-related dimensional data, such as device information and space information, in the form of BCD (slowly changing dimension)

[0140] Step S32: Clean the log data and extract the gateway protocol data by using Spark SQL+regular expression.

[0141] Step S33: Parse the robot gateway protocol data and extract relevant data such as date, equipment, and sensor.

[0142] Step S34: The data processed in step S33 is collided with the dimension data in step S31 to generate a gateway data list.

[0143] Step S35: Save the cleaned data to HDFS.

[0144] In some embodiments, the analysis and application of the detailed data in the data detail layer and the extraction of relevant indicators into the MySQL database specifically include the following steps:

[0145] Based on the business, build a demand bus matrix and design indicator types and models;

[0146] Based on the demand bus matrix, the cleaned data is analyzed to obtain statistical indicator information of relevant dimensions;

[0147] Use Spark to build models by combining relevant dimension statistical indicator information;

[0148] Output model results to MySQL database.

[0149] Robot log analysis Figure 5 As shown, the specific steps include:

[0150] Step S41: Based on the business, build a demand bus matrix and design indicator types and models

[0151] Step S42: Based on the bus matrix of step S41, the data cleaned in step S35 is analyzed, and statistical indicator information is obtained by combining relevant dimensions such as time, space, and equipment.

[0152] Step S43: Use Spark to combine indicator information to build a model

[0153] Step S44: Output the model results to the MySQL database.

[0154] In some embodiments, the robot indicator display specifically includes the following steps:

[0155] Step S51: Configure the indicator chart in SuperSet, and configure the dimension fields and indicator fields.

[0156] Step S52: Configure filter items and dimension fields in SuperSet.

[0157] Step S53: The chart of step S51 and the filter items of step S52 are configured in the dashboard for display.

[0158] like Figure 6 As shown, a robot log data analysis method based on big data facilitates operators to quickly aggregate, store and analyze robot log data. The technical solution of the invention can be found in Figure 1 , which includes the following key steps:

[0159] Step S10: Robot log collection, using the Flume+Kafka architecture for data collection, creating the Flume_Config method to generate the Flume config file and shell run script

[0160] Step S20: Robot log storage, transfer the data of the message queue Kafka to the ODS (data source layer) in HDFS through Flume for storage.

[0161] Step S30: Robot log cleaning, using the Spark / Flink+regularization architecture to clean and process log data, extracting valuable data to DWD (data detail layer) for storage.

[0162] Step S40: Robot log analysis, analyze and apply the data cleaned and processed in step S30, and extract relevant indicators into the MySQL database

[0163] Step S50: Robot indicator display, using Superset to visualize the indicators stored in MySQL for decision-making.

[0164] Robot log file analysis is of great significance to the stability and reliability of the robot system, mainly in providing strong support for robot troubleshooting, performance optimization, and safety monitoring.

[0165] This application is a robot log analysis method based on big data technology, which covers the collection and synchronization of robot log data, data storage and calculation analysis, and the final presentation of analysis results, reflecting the typical process of big data technology in robot log analysis.

[0166] See also Figure 7 , a storage medium 710, wherein the storage medium 710 stores a computer program, and the computer program is executed by a processor 720 to perform the following steps:

[0167] Use Flume+Kafka architecture to collect robot log data;

[0168] Transfer the log data in the message queue Kafka to the data source layer of the distributed file system for storage;

[0169] The Spark / Flink+regularization architecture is used to clean and process the log data in the data source layer of the distributed file system, and the robot's detailed data is extracted and stored in the data detail layer;

[0170] Analyze and apply the detailed data in the data detail layer, and extract relevant indicators into the MySQL database;

[0171] Superset is used to visualize the indicators stored in the MySQL database.

[0172] In some embodiments, the use of the Flume+Kafka architecture to collect robot log data specifically includes the following steps:

[0173] Configure the relevant environment of the log collection system in the log server according to the custom Flume_Config;

[0174] Create a Topic in Kafka Server based on the custom Flume_Config;

[0175] Generate data collection conf file based on custom Flume_Config loading template;

[0176] Create a data collection execution script based on the custom Flume_Config;

[0177] Send the data collection conf file and data collection execution script to the server that needs to be collected according to the custom Flume_Config;

[0178] Configure sensitive policies to filter sensitive information;

[0179] Start the robot log collection task to collect robot log data.

[0180] In some embodiments, the step of transferring the log data in the message queue Kafka to the data source layer of the distributed file system for storage specifically includes the following steps:

[0181] Generate data storage conf file by customizing Flume_Config loading template;

[0182] Generate data storage execution script by customizing Flume_Config loading template;

[0183] Send the generated data storage conf file and data storage execution script to the server that needs to be collected;

[0184] Start the robot log data storage task;

[0185] Call the quality inspection task to compare the source file and storage file of the log collection, and issue an alarm when the log generation is abnormal.

[0186] In some embodiments, the Spark / Flink+regularization architecture is used to clean and process the log data in the data source layer of the distributed file system, and extract the robot's detailed data to the data detail layer for storage, specifically including the following steps:

[0187] Storing the robot's dimensional data in the form of slowly changing dimensions, the dimensional data including device information and spatial information;

[0188] Use Spark SQL + regular expressions to clean the log data and extract the gateway protocol data.

[0189] Parse the robot gateway protocol data and extract relevant data, including date, equipment and sensor;

[0190] The extracted relevant data is collided with the stored dimension data to generate a gateway data detail table;

[0191] Save the cleaned data to HDFS.

[0192] In some embodiments, the analysis and application of the detailed data in the data detail layer and the extraction of relevant indicators into the MySQL database specifically include the following steps:

[0193] Based on the business, build a demand bus matrix and design indicator types and models;

[0194] Based on the demand bus matrix, the cleaned data is analyzed to obtain statistical indicator information of relevant dimensions;

[0195] Use Spark to build models by combining relevant dimension statistical indicator information;

[0196] Output model results to MySQL database.

[0197] The robot's log data is collected through the Flume+Kafka-based architecture, and then the log data in the message queue Kafka is transmitted to the data source layer of the distributed file system. Then, the Spark / Flink+regular architecture is used to clean and process the log data in the data source layer of the distributed file system, extract the robot's detailed data to the data detail layer for storage, analyze and apply the detailed data in the data detail layer, extract related indicators to the MySQL database, and then use Superset to visualize the indicators stored in the MySQL database. This solution allows operators to quickly aggregate, store and analyze the robot's log data.

[0198] Robot log file analysis is of great significance to the stability and reliability of the robot system, mainly in providing strong support for robot troubleshooting, performance optimization, and safety monitoring.

[0199] This application is a robot log analysis method based on big data technology, which covers the collection and synchronization of robot log data, data storage and calculation analysis, and the final presentation of analysis results, reflecting the typical process of big data technology in robot log analysis.

[0200] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concept of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.

Claims

1. A robot log data analysis method based on big data, characterized in that: The following steps are involved: Use Flume+Kafka architecture to collect robot log data; Transfer the log data in the message queue Kafka to the data source layer of the distributed file system for storage; The Spark / Flink+regularization architecture is used to clean and process the log data in the data source layer of the distributed file system, and the robot's detailed data is extracted and stored in the data detail layer; Analyze and apply the detailed data in the data detail layer, and extract relevant indicators into the MySQL database; Superset is used to visualize the indicators stored in the MySQL database.

2. The robot log data analysis method based on big data according to claim 1 is characterized in that: The use of the Flume+Kafka architecture to collect robot log data specifically includes the following steps: Configure the relevant environment of the log collection system in the log server according to the custom Flume_Config; Create a Topic in Kafka Server based on the custom Flume_Config; Generate data collection conf file based on custom Flume_Config loading template; Create a data collection execution script based on the custom Flume_Config; Send the data collection conf file and data collection execution script to the server that needs to be collected according to the custom Flume_Config; Configure sensitive policies to filter sensitive information; Start the robot log collection task to collect robot log data.

3. The robot log data analysis method based on big data according to claim 1 is characterized in that: The method of transferring the log data in the message queue Kafka to the data source layer of the distributed file system for storage specifically includes the following steps: Generate data storage conf file by customizing Flume_Config loading template; Generate data storage execution script by customizing Flume_Config loading template; Send the generated data storage conf file and data storage execution script to the server that needs to be collected; Start the robot log data storage task; Call the quality inspection task to compare the source file and storage file of the log collection, and issue an alarm when the log generation is abnormal.

4. The robot log data analysis method based on big data according to claim 1 is characterized in that: The Spark / Flink+regularization architecture is used to clean and process the log data in the data source layer of the distributed file system, and extract the robot's detailed data to the data detail layer for storage, which specifically includes the following steps: Storing the robot's dimensional data in the form of slowly changing dimensions, the dimensional data including device information and spatial information; Use Spark SQL + regular expressions to clean the log data and extract the gateway protocol data. Parse the robot gateway protocol data and extract relevant data, including date, equipment and sensor; The extracted relevant data is collided with the stored dimension data to generate a gateway data detail table; Save the cleaned data to HDFS.

5. The robot log data analysis method based on big data according to claim 1 is characterized in that: The detailed data in the data detail layer are analyzed and applied to extract relevant indicators into the MySQL database, which specifically includes the following steps: Based on the business, build a demand bus matrix and design indicator types and models; Based on the demand bus matrix, the cleaned data is analyzed to obtain statistical indicator information of relevant dimensions; Use Spark to build models by combining relevant dimension statistical indicator information; Output model results to MySQL database.

6. A storage medium storing a computer program, characterized in that: The computer program is executed by a processor to perform the following steps: Use Flume+Kafka architecture to collect robot log data; Transfer the log data in the message queue Kafka to the data source layer of the distributed file system for storage; The Spark / Flink+regularization architecture is used to clean and process the log data in the data source layer of the distributed file system, and the robot's detailed data is extracted and stored in the data detail layer; Analyze and apply the detailed data in the data detail layer, and extract relevant indicators into the MySQL database; Superset is used to visualize the indicators stored in the MySQL database.

7. The storage medium according to claim 6, characterized in that The use of the Flume+Kafka architecture to collect robot log data specifically includes the following steps: Configure the relevant environment of the log collection system in the log server according to the custom Flume_Config; Create a Topic in Kafka Server based on the custom Flume_Config; Generate data collection conf file based on custom Flume_Config loading template; Create a data collection execution script based on the custom Flume_Config; Send the data collection conf file and data collection execution script to the server that needs to be collected according to the custom Flume_Config; Configure sensitive policies to filter sensitive information; Start the robot log collection task to collect robot log data.

8. The storage medium according to claim 6, characterized in that The method of transferring the log data in the message queue Kafka to the data source layer of the distributed file system for storage specifically includes the following steps: Generate data storage conf file by customizing Flume_Config loading template; Generate data storage execution script by customizing Flume_Config loading template; Send the generated data storage conf file and data storage execution script to the server that needs to be collected; Start the robot log data storage task; Call the quality inspection task to compare the source file and storage file of the log collection, and issue an alarm when the log generation is abnormal.

9. The storage medium according to claim 6, characterized in that The Spark / Flink+regularization architecture is used to clean and process the log data in the data source layer of the distributed file system, and extract the robot's detailed data to the data detail layer for storage, which specifically includes the following steps: Storing the robot's dimensional data in the form of slowly changing dimensions, the dimensional data including device information and spatial information; Use Spark SQL + regular expressions to clean the log data and extract the gateway protocol data. Parse the robot gateway protocol data and extract relevant data, including date, equipment and sensor; The extracted relevant data is collided with the stored dimension data to generate a gateway data detail table; Save the cleaned data to HDFS.

10. The storage medium according to claim 6, characterized in that The detailed data in the data detail layer are analyzed and applied to extract relevant indicators into the MySQL database, which specifically includes the following steps: Based on the business, build a demand bus matrix and design indicator types and models; Based on the demand bus matrix, the cleaned data is analyzed to obtain statistical indicator information of relevant dimensions; Use Spark to build models by combining relevant dimension statistical indicator information; Output model results to MySQL database.

Images