Data processing method, device, equipment and storage medium based on big data analysis

Abstract

The present invention provides a data processing method, device, computer equipment and storage medium based on big data analysis. The above method includes: obtaining the synchronization computing requirements of the storage node; wherein the storage node is a node for storing lighting data corresponding to the lighting task; constructing a time synchronization strategy corresponding to the storage node based on the synchronization computing requirements; configuring the time synchronization service of the storage node based on the time synchronization strategy; determining the time source corresponding to the time synchronization service, and constructing a network environment between the storage node and the time source; calculating the time synchronization coefficient of the storage node based on the time synchronization service and the network environment; determining whether the time synchronization coefficient meets the preset synchronization threshold; if so, installing and configuring a real-time stream processing system on the storage node to perform real-time processing on the received data stream through the real-time stream processing system. The present invention solves the technical problems of data processing delay and data processing failure caused by time synchronization problems.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing method, device, computer equipment and storage medium based on big data analysis. Background Art

[0002] With the rapid development of information technology and the deepening of global digital transformation, big data has become a key force in promoting innovation and optimization in all walks of life. In the field of civil aviation transportation, airports are the core of transportation hubs, and their operational efficiency and safety are directly related to passengers' travel experience and the smooth operation of the overall transportation system. As an important part of ensuring the safe operation of airports at night and under low visibility conditions, the demand for intelligent and refined management of airport lighting systems is becoming increasingly prominent.

[0003] However, in the actual application of airport lighting tasks, big data processing faces many challenges, especially time synchronization, which has become one of the key factors restricting data processing efficiency and accuracy. In the existing technology, airport lighting management systems often rely on complex data processing architectures and algorithms when collecting, processing and analyzing massive amounts of data from various sensors, monitoring equipment and external data sources. Although these systems take into account the real-time and accuracy requirements of data when designing, in actual operation, due to factors such as clock differences between devices, network delays, and uneven data processing speeds, time synchronization problems are difficult to completely avoid.

[0004] Specifically, time synchronization issues mainly lead to the following two impacts:

[0005] 1. Data processing delay: In airport lighting tasks, the real-time requirements for data are extremely high. For example, dynamically adjusting the lighting brightness and range according to weather changes, flight traffic, runway usage, etc. requires the system to respond quickly and make corresponding adjustments. However, deviations in time synchronization may cause unnecessary delays in data transmission and processing, which in turn affects the timeliness and effectiveness of lighting decisions.

[0006] 2. Data processing failure and data integrity loss: For the processing of high-speed data streams, the accuracy of time synchronization is particularly important. Once there is a large deviation in time synchronization, it may cause the disorder, loss or repeated processing of data packets, which in turn causes data processing failure, seriously affecting the integrity and validity of the data. In the airport lighting system, this data processing failure may directly lead to the incorrect execution of the lighting strategy, posing a potential threat to the safe operation of the airport. Summary of the invention

[0007] The main purpose of the present invention is to provide a data processing method, device, computer equipment and storage medium based on big data analysis, aiming to solve the technical problems in the prior art that time synchronization problems exist in the data processing process of airport lighting tasks, resulting in data processing delays, data processing failures and damaged data integrity.

[0008] To achieve the above object, the present invention provides a data processing method based on big data analysis, the method comprising:

[0009] Acquire the synchronization computing requirements of a preset storage node; wherein the storage node is a node for storing lighting data corresponding to the lighting task;

[0010] Building a time synchronization strategy corresponding to the storage node based on the synchronization computing requirement;

[0011] Configuring a time synchronization service of the storage node based on the time synchronization policy;

[0012] Determine a time source corresponding to the time synchronization service, and construct a network environment between the storage node and the time source;

[0013] Calculating a time synchronization coefficient of the storage node based on the time synchronization service and the network environment;

[0014] Determining whether the time synchronization coefficient meets a preset synchronization threshold;

[0015] If so, a preset real-time stream processing system is installed and configured on the storage node, so as to perform real-time processing on the received data stream through the real-time stream processing system.

[0016] Optionally, the calculating the time synchronization coefficient of the storage node based on the time synchronization service and the network environment includes:

[0017] In each preset monitoring period, the time difference between the storage node and a preset time server is calculated; wherein the time server is a server providing the time synchronization service;

[0018] Calculating the average deviation of the time differences included in all the monitoring periods;

[0019] Calculate the standard deviation of the time differences contained in all the monitoring periods;

[0020] A time synchronization coefficient of the storage node is generated based on the average deviation and the standard deviation.

[0021] Optionally, calculating the time difference between the storage node and a preset time server within each preset monitoring period includes:

[0022] Call the preset calculation processing script;

[0023] Deploy the computing processing script to the storage node;

[0024] The calculation processing script is run in the storage node to calculate the time difference between the storage node and the time server in each monitoring period.

[0025] Optionally, generating the time synchronization coefficient of the storage node based on the average deviation and the standard deviation includes:

[0026] Obtaining a target weight corresponding to the standard deviation;

[0027] Call the preset calculation formula;

[0028] Calculate the average deviation, the standard deviation and the target weight based on the calculation formula to obtain a corresponding calculation result;

[0029] The calculation result is used as the time synchronization coefficient of the storage node.

[0030] Optionally, after determining whether the time synchronization coefficient meets a preset synchronization threshold, the method further includes:

[0031] If the time synchronization coefficient does not meet the synchronization threshold, a corresponding performance analysis is performed on the storage node to obtain a corresponding performance analysis result;

[0032] Perform troubleshooting based on the performance analysis results to obtain corresponding troubleshooting results;

[0033] Based on the troubleshooting result, the storage node is optimized accordingly.

[0034] Optionally, after installing and configuring a preset real-time stream processing system on the storage node to perform real-time processing on the received data stream through the real-time stream processing system, the method further includes:

[0035] Calling a preset monitoring tool to collect performance indicator data of the real-time stream processing system;

[0036] Displaying the performance indicator data in a preset visualization tool;

[0037] Determining whether there is specified performance indicator data exceeding a corresponding performance threshold in the performance indicator data;

[0038] If so, filtering out the specified performance indicator data from the performance indicator data, and generating corresponding alarm notification information based on the specified performance indicator data;

[0039] The alarm notification information is sent to relevant processing personnel.

[0040] Optionally, after installing and configuring a preset real-time stream processing system on the storage node to perform real-time processing on the received data stream through the real-time stream processing system, the method further includes:

[0041] Collecting log files of the real-time stream processing system based on a preset log collection tool;

[0042] Storing the log file in a preset target storage medium;

[0043] Analyze and process the log file based on a preset log analysis tool to obtain corresponding log analysis results;

[0044] Execute corresponding exception processing based on the log analysis result.

[0045] In addition, to achieve the above-mentioned purpose, the present invention further provides a data processing device based on big data analysis, the data processing device based on big data analysis comprising:

[0046] An acquisition module, used to acquire a synchronization computing requirement of a preset storage node; wherein the storage node is a node used to store lighting data corresponding to a lighting task;

[0047] A first building module, configured to build a time synchronization strategy corresponding to the storage node based on the synchronization computing requirement;

[0048] A configuration module, configured to configure a time synchronization service of the storage node based on the time synchronization policy;

[0049] A second construction module, used to determine a time source corresponding to the time synchronization service, and to construct a network environment between the storage node and the time source;

[0050] A calculation module, configured to calculate a time synchronization coefficient of the storage node based on the time synchronization service and the network environment;

[0051] A first judgment module, used to judge whether the time synchronization coefficient meets a preset synchronization threshold;

[0052] The first processing module is used to install and configure a preset real-time stream processing system on the storage node when the time synchronization coefficient meets a preset synchronization threshold, so as to perform real-time processing on the received data stream through the real-time stream processing system.

[0053] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0054] The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the data processing method based on big data analysis proposed in any one of the embodiments of the present application are implemented.

[0055] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0056] The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the data processing method based on big data analysis proposed in any one of the embodiments of the present application are implemented.

[0057] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0058] The present invention provides a data processing method, device, computer equipment and storage medium based on big data analysis, the method comprising: first obtaining the synchronization computing requirements of a preset storage node; wherein the storage node is a node for storing lighting data corresponding to a lighting task; then constructing a time synchronization strategy corresponding to the storage node based on the synchronization computing requirements; then configuring the time synchronization service of the storage node based on the time synchronization strategy; subsequently determining the time source corresponding to the time synchronization service, and constructing a network environment between the storage node and the time source; further based on the time synchronization service and the network environment, calculating the time synchronization coefficient of the storage node; finally determining whether the time synchronization coefficient meets a preset synchronization threshold; if so, installing and configuring a preset real-time stream processing system on the storage node to perform real-time processing on the received data stream through the real-time stream processing system. The present application constructs a time synchronization strategy for storage nodes according to the synchronization computing requirements of the acquired storage nodes, configures the time synchronization service for the storage nodes based on the time synchronization strategy, and then constructs a network environment for the time source corresponding to the storage nodes and the time synchronization service, and then calculates the time synchronization coefficient of the storage nodes based on the time synchronization service and the network environment, and configures a real-time stream processing system in the storage node when it is detected that the time synchronization coefficient meets the preset synchronization threshold, thereby achieving real-time processing of the received data stream, effectively avoiding data processing delays caused by time synchronization problems, and meeting the real-time processing requirements of data. At the same time, it solves the problem that the prior art may fail to process data due to time synchronization problems when processing high-speed data streams, affecting the integrity and validity of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0061] Figure 2 is a flow chart of a data processing method based on big data analysis provided by an embodiment of the present invention;

[0062] Figure 3 is a structural schematic diagram of an embodiment of a data processing device based on big data analysis according to the present application;

[0063] Figure 4 This is a basic structural block diagram of the computer device in this embodiment. DETAILED DESCRIPTION

[0064] The data processing method based on big data analysis provided by the embodiment of the present invention is applied to the data processing device based on big data analysis. Unless otherwise defined, all technical and scientific terms used in this document have the same meaning as those generally understood by technicians in the technical field of this application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned drawings and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0065] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0066] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0067] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0068] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social online platform software, etc.

[0069] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), laptop computers, desktop computers, etc.

[0070] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0071] It should be noted that the data processing method based on big data analysis provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the data processing device based on big data analysis is generally arranged in the server / terminal device.

[0072] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0073] With the rapid development of information technology and the deepening of global digital transformation, big data has become a key force in promoting innovation and optimization in all walks of life. In the field of civil aviation transportation, airports are the core of transportation hubs, and their operational efficiency and safety are directly related to passengers' travel experience and the smooth operation of the overall transportation system. As an important part of ensuring the safe operation of airports at night and under low visibility conditions, the demand for intelligent and refined management of airport lighting systems is becoming increasingly prominent.

[0074] However, in the actual application of airport lighting tasks, big data processing faces many challenges, especially time synchronization, which has become one of the key factors restricting data processing efficiency and accuracy. In the existing technology, airport lighting management systems often rely on complex data processing architectures and algorithms when collecting, processing and analyzing massive amounts of data from various sensors, monitoring equipment and external data sources. Although these systems take into account the real-time and accuracy requirements of data when designing, in actual operation, due to factors such as clock differences between devices, network delays, and uneven data processing speeds, time synchronization problems are difficult to completely avoid.

[0075] Specifically, time synchronization issues mainly lead to the following two impacts:

[0076] 1. Data processing delay: In airport lighting tasks, the real-time requirements for data are extremely high. For example, dynamically adjusting the lighting brightness and range according to weather changes, flight traffic, runway usage, etc. requires the system to respond quickly and make corresponding adjustments. However, deviations in time synchronization may cause unnecessary delays in data transmission and processing, which in turn affects the timeliness and effectiveness of lighting decisions.

[0077] 2. Data processing failure and data integrity loss: For the processing of high-speed data streams, the accuracy of time synchronization is particularly important. Once there is a large deviation in time synchronization, it may cause the disorder, loss or repeated processing of data packets, which in turn causes data processing failure, seriously affecting the integrity and validity of the data. In the airport lighting system, this data processing failure may directly lead to the incorrect execution of the lighting strategy, posing a potential threat to the safe operation of the airport.

[0078] Continue to refer Figure 2 , shows a flow chart of an embodiment of a data processing method based on big data analysis proposed in this application. The embodiment of this application can acquire and process relevant data based on artificial intelligence technology.

[0079] The data processing method based on big data analysis provided by the embodiment of the present invention comprises the following steps:

[0080] S210, obtaining synchronization computing requirements of a preset storage node; wherein the storage node is a node for storing lighting data corresponding to a lighting task.

[0081] In this step, the present invention can be applied to business scenarios in which a large amount of data related to lighting tasks is processed, and the execution subject of the present invention can be specifically a data processing system based on big data analysis. The above-mentioned storage node is a node responsible for storing lighting data corresponding to the lighting task, and the number of the storage node may include multiple. The above-mentioned lighting tasks may include airport lighting tasks, mine lighting tasks, outdoor rescue lighting tasks, and the like. The setting process of the above-mentioned synchronization computing requirements includes: Demand analysis: First, communicate with the operation and maintenance team, network engineers and business team of the data center to understand their specific computing requirements for time synchronization. Formulate standards: Based on the communication results, clarify the time synchronization accuracy of the storage node (for example, it is required to reach the millisecond level, that is, the maximum error does not exceed 1 millisecond), synchronization frequency (for example, synchronization once per second) and reliability (for example, 99.9% reliability, that is, there is a maximum error time of 31.5 seconds per year). Documentation: The above-mentioned synchronization computing requirement standards are recorded as a basis for subsequent implementation and evaluation.

[0082] S220: Construct a time synchronization strategy corresponding to the storage node based on the synchronization computing requirement.

[0083] In this step, the process of building a time synchronization strategy corresponding to the storage node includes: Selecting a protocol: According to the synchronization accuracy and reliability requirements, select a suitable time synchronization protocol, such as the PTP (Precision Time Protocol) protocol, because it can provide sub-microsecond time synchronization accuracy, and thus can meet the synchronization calculation requirements of the above settings. Protocol configuration: Understand the working principle and configuration options of the PTP protocol, including the role allocation of the master clock (Master Clock) and the slave clock (SlaveClock), the transmission mechanism of message types (such as Sync, Delay_Req, Delay_Resp, Follow_Up, etc.). Strategy formulation: Based on the configuration options of the PTP protocol, formulate a detailed time synchronization strategy, including synchronization cycle, message priority, error handling, etc.

[0084] S230: Configure a time synchronization service of the storage node based on the time synchronization policy.

[0085] In this step, by installing a time synchronization client (such as a PTP client) on each storage node, it is ensured that the time synchronization information from the time source can be received to complete the configuration of the time synchronization service for the storage node. Among them, by ensuring that the software version is compatible with the PTP protocol version. And according to the time synchronization strategy, configure the parameters of the PTP client, such as the time source address, synchronization cycle, log level, etc. In addition, it can be verified by test commands or tools whether the PTP client can correctly receive and parse the time synchronization information.

[0086] S240, determining a time source corresponding to the time synchronization service, and constructing a network environment between the storage node and the time source.

[0087] In this step, the time source is the source from which the time server obtains accurate time information. It can be an atomic clock of the Global Positioning System (GPS), an atomic clock of an observatory, or other high-precision time reference equipment. The time source has very high time accuracy and is the basis for the time server to provide accurate time services. The time server receives time information from the time source and passes it to other devices, thereby achieving time synchronization of the entire network.

[0088] Among them, a high-speed, low-latency network connection (such as Gigabit Ethernet) can be established between the storage node and the time source to ensure the rapid transmission of time synchronization information. Specifically, the process of building a network environment includes: Network planning: According to the physical location of the storage node and the distribution of the time source, plan the network topology to ensure that each storage node can efficiently access the time source. Equipment selection: Select suitable network equipment and switches to ensure that they support the PTP protocol or have corresponding priority and delay control functions. Cabling construction: Carry out cabling construction according to the network plan, and use high-quality network cables or optical fibers to connect storage nodes and time sources. Network testing: Use network testing tools (such as ping, traceroute, iperf, etc.) to test the latency, bandwidth and stability of network connections.

[0089] S250: Calculate a time synchronization coefficient of the storage node based on the time synchronization service and the network environment.

[0090] In this step, the specific implementation process of calculating the time synchronization coefficient of the storage node based on the time synchronization service and the network environment will be further described in detail in subsequent specific embodiments of the present application and will not be elaborated on here.

[0091] S260: Determine whether the time synchronization coefficient meets a preset synchronization threshold.

[0092] In this step, there is no specific limitation on the selection of the above synchronization threshold, which can be set according to the actual business usage requirements, for example, it can be set to 1 millisecond. Among them, the time synchronization coefficient meets the preset synchronization threshold means that the time synchronization coefficient is less than the synchronization threshold. If the time synchronization coefficient is greater than or equal to the synchronization threshold, it is determined that the time synchronization coefficient does not meet the synchronization threshold.

[0093] S270: If yes, install and configure a preset real-time stream processing system on the storage node to perform real-time processing on the received data stream through the real-time stream processing system.

[0094] In this step, the above-mentioned real-time stream processing system can specifically adopt Apache Storm, which is an open source real-time stream processing system that can process high-speed data streams and perform complex calculations on data. The configuration process of the above-mentioned real-time stream processing system may specifically include: 1. Environment preparation: Ensure that the operating system and hardware resources (such as CPU, memory, and disk space) of the storage node meet the installation requirements of Apache Storm or other real-time stream processing systems. 2. Software installation: Download and install Apache Storm or other selected real-time stream processing systems. Follow the official documentation or installation guide for installation to ensure that all components are correctly configured and started. 3. Configure topology: Design and configure real-time stream processing topology according to business requirements. This includes defining data sources, processing logic (such as filtering, aggregation, conversion, etc.) and output targets (such as databases, message queues, file systems, etc.). 4. Testing and tuning: Use simulated data or actual data to test the real-time stream processing system to verify its performance and accuracy. Perform necessary tuning based on the test results to improve processing speed and stability. 5. Deployment and monitoring: Deploy the real-time stream processing system to the production environment and use monitoring tools to continuously monitor it. Ensure that the real-time stream processing system can run stably and meet business requirements.

[0095] In an embodiment of the present invention, first, a synchronization computing requirement of a preset storage node is obtained; wherein the storage node is a node for storing lighting data corresponding to a lighting task; then, a time synchronization strategy corresponding to the storage node is constructed based on the synchronization computing requirement; then, a time synchronization service of the storage node is configured based on the time synchronization strategy; subsequently, a time source corresponding to the time synchronization service is determined, and a network environment between the storage node and the time source is constructed; further, based on the time synchronization service and the network environment, a time synchronization coefficient of the storage node is calculated; finally, it is determined whether the time synchronization coefficient meets a preset synchronization threshold; if so, a preset real-time stream processing system is installed and configured on the storage node, so that the received data stream is processed in real time through the real-time stream processing system. The present application constructs a time synchronization strategy for storage nodes according to the synchronization computing requirements of the acquired storage nodes, configures the time synchronization service for the storage nodes based on the time synchronization strategy, and then constructs a network environment for the time source corresponding to the storage nodes and the time synchronization service, and then calculates the time synchronization coefficient of the storage nodes based on the time synchronization service and the network environment, and configures a real-time stream processing system in the storage node when it is detected that the time synchronization coefficient meets the preset synchronization threshold, thereby achieving real-time processing of the received data stream, effectively avoiding data processing delays caused by time synchronization problems, and meeting the real-time processing requirements of data. At the same time, it solves the problem that the prior art may fail to process data due to time synchronization problems when processing high-speed data streams, affecting the integrity and validity of the data.

[0096] Optionally, the calculating the time synchronization coefficient of the storage node based on the time synchronization service and the network environment includes:

[0097] In each preset monitoring period, the time difference between the storage node and a preset time server is calculated; wherein the time server is a server that provides the time synchronization service.

[0098] In this step, the above-mentioned time server is a server that provides the time synchronization service. Specifically, a time server is a network server that is used to provide accurate time information to other devices or computers. It is based on a high-precision time source (Time Source), such as the Universal Time (UTC), the Global Positioning System (GPS) or other atomic clocks. Time servers are usually used together with the Network Time Protocol (NTP) to ensure that devices in the network have accurate time data. Time servers play a vital role in network communications. Many applications and systems require accurate time data to ensure the accuracy of tasks such as logging, security authentication, and data synchronization. Among them, the specific implementation process of calculating the time difference between the storage node and the preset time server within each preset monitoring period will be further described in detail in the subsequent specific embodiments of this application, and will not be elaborated on here.

[0099] The average deviation of the time differences included in all the monitoring periods is calculated.

[0100] In this step, the average deviation can be obtained by calculating the average value of all time differences included in all monitoring cycles. The average deviation reflects the average offset of time synchronization.

[0101] The standard deviation of the time differences contained in all the monitoring periods is calculated.

[0102] In this step, the standard deviation of all time differences included in all the monitoring periods can be calculated by using a standard deviation calculation formula. The standard deviation reflects the stability or volatility of time synchronization.

[0103] A time synchronization coefficient of the storage node is generated based on the average deviation and the standard deviation.

[0104] In this step, the time synchronization coefficient is an indicator for comprehensively evaluating the time synchronization performance of the storage node. It can be defined according to actual needs, and specifically may include mean deviation and standard deviation as key factors. The specific implementation process of generating the time synchronization coefficient of the storage node based on the mean deviation and the standard deviation will be further described in detail in the subsequent specific embodiments of this application, and will not be elaborated on here.

[0105] In this embodiment, the time difference between the storage node and the preset time server is calculated in each preset monitoring cycle; wherein the time server is a server providing the time synchronization service; then the average deviation of the time difference included in all the monitoring cycles is calculated; and the standard deviation of the time difference included in all the monitoring cycles is calculated; and subsequently the time synchronization coefficient of the storage node is generated based on the average deviation and the standard deviation. The present application calculates the time difference between the storage node and the preset time server in each preset monitoring cycle, and then calculates the average deviation and the standard deviation of the time difference included in all the monitoring cycles, and then generates the time synchronization coefficient of the storage node based on the average deviation and the standard deviation, so as to realize intelligent and accurate generation of the time synchronization coefficient for comprehensively evaluating the time synchronization performance of the storage node based on the average deviation and the standard deviation, thereby improving the intelligence of the generation of the time synchronization coefficient and ensuring the reliability and accuracy of the obtained time synchronization coefficient.

[0106] Optionally, calculating the time difference between the storage node and a preset time server within each preset monitoring period includes:

[0107] Call the preset calculation processing script.

[0108] In this step, a suitable monitoring cycle is pre-set according to the actual processing requirements, such as recording the time difference once every minute, hour or day. This monitoring cycle can be determined according to the accuracy requirements of time synchronization and system characteristics. The above calculation processing script is an automated script written and generated according to the actual functional requirements of regularly querying the time server and calculating the time difference.

[0109] Deploy the calculation processing script to the storage node.

[0110] In this step, the node position of the storage node may be determined, and then the calculation processing script may be deployed to the storage node according to the node position.

[0111] The calculation processing script is run in the storage node to calculate the time difference between the storage node and the time server in each monitoring period.

[0112] In this step, after the calculation processing script is deployed in the storage node, the calculation processing script is run in the storage node, so as to calculate and record the time difference between the storage node and the time server in each monitoring cycle.

[0113] In this embodiment, a preset calculation processing script is called; the calculation processing script is then deployed to the storage node; and the calculation processing script is subsequently run in the storage node to calculate the time difference between the storage node and the time server in each monitoring cycle. Based on the use of the calculation processing script, the present application deploys the calculation processing script to the storage node and runs it, so that the time difference between the storage node and the time server can be automatically calculated in each monitoring cycle, which effectively improves the processing intelligence and processing efficiency of the calculation time difference.

[0114] Optionally, generating the time synchronization coefficient of the storage node based on the average deviation and the standard deviation includes:

[0115] Get a target weight corresponding to the standard deviation.

[0116] In this step, the target weight is used to adjust the influence of the standard deviation on the time synchronization coefficient, and the value of the target weight can be adjusted according to the actual situation.

[0117] Call the preset calculation formula.

[0118] In this step, the above calculation formula may specifically include: X=1 / (|X|+Y*a), wherein X is the time synchronization coefficient, X is the time difference contained in all the monitoring periods, |X| is the absolute value of the average deviation, Y is the standard deviation of the time difference contained in all the monitoring periods, and a is the target weight.

[0119] The average deviation, the standard deviation and the target weight are calculated based on the calculation formula to obtain corresponding calculation results.

[0120] In this step, the average deviation and standard deviation are indicators for evaluating the accuracy and stability of time synchronization. The average deviation, standard deviation and target weight can be substituted into the corresponding positions in the above calculation formula to obtain the corresponding calculation result.

[0121] The calculation result is used as the time synchronization coefficient of the storage node.

[0122] In this step, the time synchronization coefficient is a value that combines the average deviation, standard deviation and other relevant weights, and is used to quantify the performance of storage node time synchronization.

[0123] In this embodiment, the target weight corresponding to the standard deviation is obtained; then a preset calculation formula is called; then the average deviation, the standard deviation and the target weight are calculated based on the calculation formula to obtain a corresponding calculation result; and the calculation result is subsequently used as the time synchronization coefficient of the storage node. This application obtains the target weight corresponding to the standard deviation, and then calculates the average deviation, the standard deviation and the target weight based on the use of a calculation formula, thereby achieving rapid and accurate generation of the time synchronization coefficient of the storage node, effectively improving the generation efficiency of the time synchronization coefficient of the storage node, and ensuring the data accuracy and reliability of the obtained time synchronization coefficient of the storage node.

[0124] Optionally, after determining whether the time synchronization coefficient meets a preset synchronization threshold, the method further includes:

[0125] If the time synchronization coefficient does not meet the synchronization threshold, a corresponding performance analysis is performed on the storage node to obtain a corresponding performance analysis result.

[0126] In this step, if the time synchronization coefficient does not meet the synchronization threshold, that is, the time synchronization coefficient is greater than the synchronization threshold, it indicates that the time synchronization performance of the storage node has declined and the accuracy of time synchronization is low. At this time, the storage node is subjected to corresponding performance analysis to analyze the reasons that may cause the time synchronization performance of the storage node to decline, and obtain the above-mentioned performance analysis results. Among them, the performance analysis results may include unstable network environment, improper configuration of time synchronization service, hardware failure, etc.

[0127] Perform troubleshooting based on the performance analysis results to obtain corresponding troubleshooting results.

[0128] In this step, troubleshooting is performed based on the performance analysis results, specifically checking whether the network connection is stable, whether the time synchronization service is running normally, whether the hardware resources are sufficient, etc., and the corresponding troubleshooting results are recorded. Among them, the troubleshooting results may include unstable network connection, or the time synchronization service cannot run normally, or insufficient hardware resources, etc.

[0129] Based on the troubleshooting result, the storage node is optimized accordingly.

[0130] In this step, corresponding optimization measures can be taken for the storage node according to the troubleshooting results. Specifically, if the troubleshooting result is that the network connection is unstable, the network configuration is optimized to reduce delays and packet loss; if the troubleshooting result is that the time synchronization service cannot run normally, the parameters of the time synchronization service are adjusted to improve synchronization accuracy and stability; and if the troubleshooting result is that the hardware resources are insufficient, the hardware resources are increased to improve processing capabilities, etc. Among them, the process and results of performance evaluation, troubleshooting and optimization measures can be further recorded for subsequent reference and audit.

[0131] In this embodiment, if the time synchronization coefficient does not meet the synchronization threshold, a corresponding performance analysis is performed on the storage node to obtain a corresponding performance analysis result; then, a troubleshooting is performed based on the performance analysis result to obtain a corresponding troubleshooting result; and subsequently, the storage node is optimized accordingly based on the troubleshooting result. When the present application detects that the time synchronization coefficient does not meet the synchronization threshold, a corresponding performance analysis is intelligently performed on the storage node to obtain a corresponding performance analysis result, and then a troubleshooting is performed based on the performance analysis result to obtain a corresponding troubleshooting result, and then, based on the troubleshooting result, a corresponding optimization is performed on the storage node, so as to achieve automatic and intelligent performance analysis, troubleshooting, and optimization of the storage node when the time synchronization performance of the storage node decreases, thereby helping to improve the time synchronization performance of the storage node, and thereby improving the accuracy and stability of time synchronization.

[0132] Optionally, after installing and configuring a preset real-time stream processing system on the storage node to perform real-time processing on the received data stream through the real-time stream processing system, the method further includes:

[0133] A preset monitoring tool is called to collect performance indicator data of the real-time stream processing system.

[0134] In this step, the above-mentioned monitoring tool can specifically adopt Prometheus, which is an open source monitoring and alarm tool that supports time series database storage of monitoring indicator data and has flexible query language and efficient data collection capabilities. The above-mentioned performance indicator data may at least include throughput, latency, CPU usage, memory usage, etc. Among them, throughput: the amount of data that the system can process per unit time. Latency: the time interval from the data entering the system to the completion of processing. CPU usage: the occupancy of the processor when processing the data stream. Memory usage: the occupancy of the system memory when processing the data stream. In addition, the process of configuring the monitoring tool may include: Installing Prometheus: Downloading and installing the latest version from the official website of Prometheus. Configuring Prometheus: Editing the configuration file of Prometheus (such as prometheus.yml), specifying the target to be monitored (such as related services of the real-time stream processing system) and the data storage location. Start Prometheus: Start the Prometheus service through the command line to ensure that it can collect and process monitoring data normally.

[0135] The performance indicator data is displayed in a preset visualization tool.

[0136] In this step, the visualization tool mentioned above can be Grafana, which is a popular open source data visualization tool that supports multiple data sources and rich chart displays. Specifically, add Prometheus as a data source in Grafana and configure relevant parameters to ensure that Grafana can obtain monitoring data from Prometheus. Then create a dashboard in Grafana and add various charts to display performance indicator data such as throughput, latency, CPU usage, and memory usage.

[0137] It is determined whether there is any specified performance indicator data exceeding a corresponding performance threshold in the performance indicator data.

[0138] In this step, by setting the alarm rules in Grafana, when the performance indicator data exceeds the preset corresponding performance threshold, the alarm is automatically triggered to notify the relevant personnel. Among them, for each performance indicator, according to the actual monitoring and analysis requirements, performance thresholds corresponding to various performance indicators are preset. By comparing the values ​​of all the performance indicator data with the corresponding performance thresholds, it can be detected whether there is any specified performance indicator data in the performance indicator data that exceeds the corresponding performance threshold.

[0139] If so, the specified performance indicator data is filtered out from the performance indicator data, and corresponding alarm notification information is generated based on the specified performance indicator data.

[0140] In this step, the alarm notification information can be obtained by filling the specified performance indicator data into the preset alarm notification information template. There is no specific limitation on the content setting of the alarm notification information template, which can be set according to the actual alarm notification requirements.

[0141] The alarm notification information is sent to relevant processing personnel.

[0142] In this step, the alarm notification information may be sent to relevant processing personnel by using notification methods such as telephone, email, instant text message, etc. The above-mentioned relevant processing personnel may be personnel responsible for operation and maintenance of the real-time stream processing system.

[0143] In this embodiment, the performance indicator data of the real-time stream processing system is collected by calling a preset monitoring tool; then the performance indicator data is displayed in a preset visualization tool; then it is determined whether there is specified performance indicator data exceeding the corresponding performance threshold in the performance indicator data; if so, the specified performance indicator data is filtered out from the performance indicator data, and corresponding alarm notification information is generated based on the specified performance indicator data; and the alarm notification information is subsequently sent to relevant processing personnel. This application installs and configures a preset real-time stream processing system on the storage node, so that after the real-time stream processing system processes the received data stream in real time, it will also intelligently implement performance monitoring of the real-time stream processing system based on the use of monitoring tools and visualization tools, and will automatically trigger alarm notification information and send it to relevant processing personnel when it is detected that there is specified performance indicator data exceeding the corresponding performance threshold in the performance indicator data, so that the relevant processing personnel can promptly handle abnormal events occurring in the real-time stream processing system, thereby ensuring the stable operation and efficient processing of the real-time stream processing system.

[0144] Optionally, after installing and configuring a preset real-time stream processing system on the storage node to perform real-time processing on the received data stream through the real-time stream processing system, the method further includes:

[0145] The log files of the real-time stream processing system are collected based on a preset log collection tool.

[0146] In this step, log records are configured in advance in each component of the real-time stream processing system to ensure that all key operations corresponding to the real-time stream processing system have corresponding log records. The above-mentioned log collection tool can specifically use Fluentd, Logstash, etc., which is used to collect log files generated by each component of the real-time stream processing system.

[0147] The log file is stored in a preset target storage medium.

[0148] In this step, there is no specific limitation on the selection of the target storage medium. For example, storage systems such as Elasticsearch and Kafka may be used.

[0149] The log file is analyzed and processed based on a preset log analysis tool to obtain corresponding log analysis results.

[0150] In this step, the log analysis tool may be Kibana, Splunk, etc. The log analysis tool is used to search, filter, and aggregate the log files to identify potential problems and abnormal situations, and generate corresponding log analysis results.

[0151] Execute corresponding exception processing based on the log analysis result.

[0152] In this step, the above exception handling process specifically includes: Exception capture: Reasonably capture and handle exceptions in the code of the real-time stream processing system to ensure that the exception does not cause system crash or data loss. Exception reporting: Report the captured exception information to the exception management system or monitoring platform so that relevant personnel can discover and handle the exception in time. Exception analysis: Conduct in-depth analysis of the reported exception information to find out the causes and patterns of the exception, and take corresponding measures to avoid the recurrence of similar exceptions.

[0153] In addition, you can also set the log level and rotation strategy. Log level: Set different log levels (such as DEBUG, INFO, WARN, ERROR, etc.) according to the importance and urgency of the log, so that key information can be quickly filtered out when needed. Log rotation strategy: Set the rotation strategy of the log file, such as rotating by size, time or number of files, to avoid excessive storage space occupation by log files. You can use the system's log management tools (such as logrotate) or custom scripts to implement log rotation.

[0154] In this embodiment, the log files of the real-time stream processing system are collected based on a preset log collection tool; then the log files are stored in a preset target storage medium; then the log files are analyzed and processed based on a preset log analysis tool to obtain corresponding log analysis results; and corresponding exception processing is subsequently performed based on the log analysis results. The present application installs and configures a preset real-time stream processing system on the storage node, so that after the real-time stream processing system performs real-time processing on the received data stream, it will also intelligently implement log exception monitoring for the real-time stream processing system based on the use of log collection tools and log analysis tools, and will automatically perform corresponding exception processing based on the obtained log analysis results, so as to automatically handle abnormal events occurring in the real-time stream processing system, thereby ensuring the stable operation and efficient processing of the real-time stream processing system. In addition, the log files will also be stored based on the use of the target storage medium, thereby ensuring the security of the log files.

[0155] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a data processing device 300 based on big data analysis. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0156] An embodiment of the present invention provides a data processing device 300 based on big data analysis. The data processing device 300 based on big data analysis includes:

[0157] The acquisition module 310 is used to acquire the synchronization computing requirements of a preset storage node; wherein the storage node is a node for storing lighting data corresponding to the lighting task;

[0158] A first construction module 320, configured to construct a time synchronization strategy corresponding to the storage node based on the synchronization computing requirement;

[0159] A configuration module 330, configured to configure a time synchronization service of the storage node based on the time synchronization policy;

[0160] A second construction module 340 is used to determine a time source corresponding to the time synchronization service and to construct a network environment between the storage node and the time source;

[0161] A calculation module 350, configured to calculate a time synchronization coefficient of the storage node based on the time synchronization service and the network environment;

[0162] A first judgment module 360 ​​is used to judge whether the time synchronization coefficient meets a preset synchronization threshold;

[0163] The first processing module 370 is used to install and configure a preset real-time stream processing system on the storage node when the time synchronization coefficient meets a preset synchronization threshold, so as to perform real-time processing on the received data stream through the real-time stream processing system.

[0164] Optionally, the calculation module 350 includes:

[0165] A first calculation submodule, configured to calculate the time difference between the storage node and a preset time server within each preset monitoring period; wherein the time server is a server providing the time synchronization service;

[0166] A second calculation submodule, used for calculating the average deviation of the time differences included in all the monitoring cycles;

[0167] A third calculation submodule, used to calculate the standard deviation of the time differences contained in all the monitoring cycles;

[0168] A generating submodule is used to generate a time synchronization coefficient of the storage node based on the average deviation and the standard deviation.

[0169] Optionally, the first calculation submodule includes:

[0170] A first calling unit, used for calling a preset calculation processing script;

[0171] A deployment subunit, used to deploy the calculation processing script to the storage node;

[0172] The running subunit is used to run the calculation processing script in the storage node to calculate the time difference between the storage node and the time server in each monitoring cycle.

[0173] Optionally, the generating submodule includes:

[0174] An acquisition submodule, used to acquire a target weight corresponding to the standard deviation;

[0175] A second calling unit, used for calling a preset calculation formula;

[0176] A calculation unit, configured to calculate the average deviation, the standard deviation and the target weight based on the calculation formula to obtain a corresponding calculation result;

[0177] A determination unit is used to use the calculation result as a time synchronization coefficient of the storage node.

[0178] Optionally, the data processing device 300 based on big data analysis includes:

[0179] A first analysis module, configured to perform corresponding performance analysis on the storage node to obtain a corresponding performance analysis result if the time synchronization coefficient does not meet the synchronization threshold;

[0180] A troubleshooting module, used to perform troubleshooting based on the performance analysis results to obtain corresponding troubleshooting results;

[0181] The optimization module is used to perform corresponding optimization processing on the storage node based on the fault troubleshooting result.

[0182] Optionally, the data processing device 300 based on big data analysis further includes:

[0183] A calling module, used for calling a preset monitoring tool to collect performance indicator data of the real-time stream processing system;

[0184] A display module, used to display the performance indicator data in a preset visualization tool;

[0185] A second judgment module is used to judge whether there is specified performance indicator data exceeding the corresponding performance threshold in the performance indicator data;

[0186] A generating module, configured to filter out the specified performance indicator data from the performance indicator data when there is specified performance indicator data exceeding the corresponding performance threshold in the performance indicator data, and generate corresponding alarm notification information based on the specified performance indicator data;

[0187] The alarm notification information is sent to relevant processing personnel.

[0188] Optionally, the data processing device 300 based on big data analysis further includes:

[0189] A collection module, used to collect log files of the real-time stream processing system based on a preset log collection tool;

[0190] A storage module, used for storing the log file in a preset target storage medium;

[0191] The second analysis module is used to analyze and process the log file based on a preset log analysis tool to obtain a corresponding log analysis result;

[0192] The second processing module is used to perform corresponding exception processing based on the log analysis result.

[0193] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0194] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0195] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0196] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as the program code of the data processing method based on big data analysis, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0197] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the program code stored in the memory 41 or process data, such as running the program code of the data processing method based on big data analysis.

[0198] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0199] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores the application crash processing program, and the application crash processing program can be executed by at least one processor to enable the at least one processor to perform the steps of the data processing method based on big data analysis as described above.

[0200] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware online platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0201] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0202] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data processing method based on big data analysis, characterized in that: include: Acquire the synchronization computing requirements of a preset storage node; wherein the storage node is a node for storing lighting data corresponding to the lighting task; Building a time synchronization strategy corresponding to the storage node based on the synchronization computing requirement; Configuring a time synchronization service of the storage node based on the time synchronization policy; Determine a time source corresponding to the time synchronization service, and construct a network environment between the storage node and the time source; Calculating a time synchronization coefficient of the storage node based on the time synchronization service and the network environment; Determining whether the time synchronization coefficient meets a preset synchronization threshold; After determining whether the time synchronization coefficient meets a preset synchronization threshold, the method further includes: If the time synchronization coefficient does not meet the synchronization threshold, a corresponding performance analysis is performed on the storage node to obtain a corresponding performance analysis result; Perform troubleshooting based on the performance analysis results to obtain corresponding troubleshooting results; Performing corresponding optimization processing on the storage node based on the troubleshooting result; If so, a preset real-time stream processing system is installed and configured on the storage node, so as to perform real-time processing on the received data stream through the real-time stream processing system.

2. The method according to claim 1, characterized in that The calculating the time synchronization coefficient of the storage node based on the time synchronization service and the network environment includes: In each preset monitoring period, the time difference between the storage node and a preset time server is calculated; wherein the time server is a server providing the time synchronization service; Calculating the average deviation of the time differences included in all the monitoring periods; Calculate the standard deviation of the time differences contained in all the monitoring periods; A time synchronization coefficient of the storage node is generated based on the average deviation and the standard deviation.

3. The method according to claim 2, characterized in that The calculating, within each preset monitoring period, the time difference between the storage node and a preset time server comprises: Call the preset calculation processing script; Deploy the computing processing script to the storage node; The calculation processing script is run in the storage node to calculate the time difference between the storage node and the time server in each monitoring period.

4. The method according to claim 2, characterized in that: The generating the time synchronization coefficient of the storage node based on the average deviation and the standard deviation includes: Obtaining a target weight corresponding to the standard deviation; Call the preset calculation formula; Calculate the average deviation, the standard deviation and the target weight based on the calculation formula to obtain a corresponding calculation result; The calculation result is used as the time synchronization coefficient of the storage node.

5. The method according to claim 1, characterized in that After installing and configuring a preset real-time stream processing system on the storage node to perform real-time processing on the received data stream through the real-time stream processing system, the method further includes: Calling a preset monitoring tool to collect performance indicator data of the real-time stream processing system; Displaying the performance indicator data in a preset visualization tool; Determining whether there is specified performance indicator data exceeding a corresponding performance threshold in the performance indicator data; If so, filtering out the specified performance indicator data from the performance indicator data, and generating corresponding alarm notification information based on the specified performance indicator data; The alarm notification information is sent to relevant processing personnel.

6. The method according to claim 1, characterized in that After installing and configuring a preset real-time stream processing system on the storage node to perform real-time processing on the received data stream through the real-time stream processing system, the method further includes: Collecting log files of the real-time stream processing system based on a preset log collection tool; Storing the log file in a preset target storage medium; Analyze and process the log file based on a preset log analysis tool to obtain corresponding log analysis results; Execute corresponding exception processing based on the log analysis result.

7. A data processing device based on big data analysis, characterized in that: include: An acquisition module, used to acquire a synchronization computing requirement of a preset storage node; wherein the storage node is a node used to store lighting data corresponding to a lighting task; A first building module, configured to build a time synchronization strategy corresponding to the storage node based on the synchronization computing requirement; A configuration module, configured to configure a time synchronization service of the storage node based on the time synchronization policy; A second construction module, used to determine a time source corresponding to the time synchronization service, and to construct a network environment between the storage node and the time source; A calculation module, configured to calculate a time synchronization coefficient of the storage node based on the time synchronization service and the network environment; A first judgment module, used to judge whether the time synchronization coefficient meets a preset synchronization threshold; A first analysis module, configured to perform corresponding performance analysis on the storage node to obtain a corresponding performance analysis result if the time synchronization coefficient does not meet the synchronization threshold; A troubleshooting module, used to perform troubleshooting based on the performance analysis results to obtain corresponding troubleshooting results; An optimization module, used for performing corresponding optimization processing on the storage node based on the fault troubleshooting result; The first processing module is used to install and configure a preset real-time stream processing system on the storage node, so as to perform real-time processing on the received data stream through the real-time stream processing system.

8. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the data processing method based on big data analysis as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the data processing method based on big data analysis as described in any one of claims 1 to 6.

Images