A data monitoring and analysis method, system, device and medium for multiple data

Abstract

The present invention relates to the technical field of data processing, and in particular to a data monitoring and analysis method, system, device and medium for multiple data. The data monitoring and analysis method for multiple data includes receiving a data analysis and processing request for data processing of a queue of data to be processed in real time; preprocessing a pre-trained data analysis and processing framework according to the data analysis and processing request to obtain an initialized data analysis and processing rule that meets the data analysis and processing request; obtaining the data to be processed carried in the queue of data to be processed; mapping the data to be processed to a corresponding thread pool for data analysis processor adaptation according to the data analysis and processing rule, and selecting the data analysis processor with the best adaptation degree to perform data analysis and processing on the data to be processed. The present application has the effect of improving the efficiency and quality of data analysis and processing.

Description

Technical Field

[0001] The present invention relates to the technical field of data monitoring, and in particular to a data monitoring and analysis method, system, equipment and medium for multiple data. Background Art

[0002] With the promotion and development of informatization and digital transformation, more and more users will choose online informatization systems to handle business, resulting in a gradual increase in the number of business systems connected. Many schools and government departments are facing the challenges of rapid business growth and the operation and maintenance of IT assets in the cloud. The number of various IT resources such as network equipment, servers, storage, IT infrastructure, application systems, databases, etc. is increasing. In addition, the variety of IT resources and their scattered locations have greatly increased the difficulty of asset management and system maintenance.

[0003] At present, in order to solve the problem of multi-protocol parsing, some technical solutions adopt a hard-coded approach, that is, to write a dedicated parsing code for each protocol. However, the hard-coded multi-protocol parsing solution has problems such as poor flexibility, weak scalability and high maintenance cost. When a new protocol needs to be supported, a large amount of code often needs to be modified, and the entire parsing system may even need to be redesigned. Therefore, there is room for improvement. Summary of the invention

[0004] In order to improve the efficiency and quality of data analysis and processing, the present application provides a data monitoring and analysis method, system, device and medium for multiple data.

[0005] The above-mentioned invention objective of the present application is achieved through the following technical solutions: A method for monitoring and analyzing multiple data is provided, and the method for monitoring and analyzing multiple data includes: Receive data parsing processing requests for data processing on the data queue to be processed in real time; According to the data analysis and processing request, preprocessing is performed on the pre-trained data analysis and processing framework to obtain initialized data analysis and processing rules that meet the data analysis and processing request; Obtain the data to be processed in the queue of data to be processed; According to the data analysis and processing rules, the data to be processed is mapped to the corresponding thread pool for data analysis processor adaptation, and the data analysis processor with the best adaptation degree is selected to perform data analysis and processing on the data to be processed.

[0006] By adopting the above technical solution, since the data parsing queue service can only use the groups in the queue, it is impossible to dynamically create a data parsing queue according to the actual data to be parsed, and hard coding usually randomly pulls the data to be parsed to group the data parsing queue. The data parsing process is relatively messy and the processing pressure of the data parsing queue is large. Therefore, a pre-trained data parsing processing framework is initialized according to the data parsing processing request, so that the obtained data parsing processing rules meet the current data parsing processing requirements, improve the adaptability of the data parsing processing framework and the current data queue to be processed, and by mapping the data to be processed to the thread pool, the data parsing processor with the best adaptability performs data parsing processing on the data to be processed, so that each data to be processed is parsed by the corresponding thread pool and distributed to the corresponding data parsing processor for processing, so that each group of data in the data queue to be processed can be processed in parallel, thereby improving the orderliness of data parsing processing.

[0007] In a preferred example, the present application can be further configured as follows: according to the data analysis and processing request, the pre-trained data analysis and processing framework is pre-processed to obtain an initialized data analysis and processing rule that meets the data analysis and processing request, specifically including: According to the data parsing processing request, a target thread pool adapted to the queue of data to be processed is constructed; Re-dividing the data parsing and processing classes in the data parsing and processing framework to obtain a target data processing class that is compatible with the queue of data to be processed; Respectively obtain the mapping relationship between the to-be-processed data queue and the target thread pool and the target data processing class; According to the mapping relationship, the target data processing class is mapped to the target thread pool for execution, and a data analysis and processing rule that matches the data analysis and processing request is obtained, wherein the data analysis and processing rule is used to control the target thread pool to perform data analysis and processing on the target data analysis and processing class.

[0008] By adopting the above technical scheme, a corresponding target thread pool is constructed in the data parsing processing framework through the data in the data parsing processing request, so as to improve the degree of fit between the data to be parsed and the target thread pool, and the data processing class is re-divided according to the current data, so as to make the target data processing class fit with the data queue to be processed, and the data parsing processing class is mapped to the target thread pool for operation through the mapping relationship between the data queue to be processed and the target thread pool, and between the data queue to be processed and the target data parsing processing class, so as to mobilize the corresponding target thread pool to perform parallel processing on the data queue to be processed, thereby improving the orderliness of the data queue processing, and controlling the target thread pool to perform data parsing processing on the data queue to be processed by the data parsing processing rule, thereby improving the efficiency of data parsing processing in the data queue.

[0009] In a preferred example, the present application may be further configured as follows: according to the data analysis and processing request, the pre-trained data analysis and processing framework is pre-processed to obtain an initialized data analysis and processing rule that meets the data analysis and processing request, and further includes: Re-dividing the data capacity of the to-be-processed data queue according to the data parsing and processing request to obtain a data capacity threshold that meets the data parsing and processing request; Adjusting the data visibility time of each group of data queues so that the adjusted data visibility time meets the processing requirements of the data capacity threshold, thereby obtaining the target data visibility time; According to the target data visibility time, adjusting the data renewal time of each group of data queues to obtain a target data renewal time that meets the data processing request; According to the target data visible time and the target data renewal time, the data processing time rule of the data parsing and processing framework is updated to obtain the data processing time rule adapted to the queue of data to be processed.

[0010] By adopting the above technical scheme, the data capacity of each group in the data queue to be processed is dynamically divided according to the data parsing processing request, so as to obtain the data capacity threshold of the maximum data receiving quantity of each group of data, so as to control the division of each group of data to be processed to be within the optimal processing range of the data parsing processing framework, and adjust the data visibility time and data renewal time of the data queue respectively according to the data processing request, and update the adjusted target data visibility time and target data renewal time to the data processing time rule, so as to obtain the data parsing processing time rule adapted to the data queue to be processed, so as to quickly initialize the data of each group of data queue to be processed according to the data parsing processing time rule, thereby improving the processing efficiency of the data queue by the data parsing processing framework.

[0011] In a preferred example, the present application may be further configured as follows: according to the data analysis and processing rule, the data to be processed is mapped to a corresponding thread pool for data analysis processor adaptation, and the data analysis processor with the best adaptation degree is selected to perform data analysis processing on the data to be processed, specifically including: According to the data analysis and processing rules, data renewal processing is performed on the data to be processed to obtain renewal data that runs according to the target data renewal time; Perform thread pool adaptation processing on the renewal data, and construct a thread pool mapping relationship between the renewal data and the target thread pool; According to the thread pool mapping relationship, the renewal data is adapted to the data parsing processor in the target thread pool, and the optimal data parsing processor with the best adaptation degree is selected; A data distribution relationship between the renewal data and the optimal data parsing processor is established, and data parsing processing is performed on the renewal data according to the data distribution relationship.

[0012] By adopting the above technical scheme, data renewal processing is performed on the data to be processed, so that the data to be processed runs according to the target renewal time, the data to be processed is controlled for cyclic renewal, and the data to be processed is reduced from being divided into multiple groups for repeated processing at the same time. The renewal data is thread pool adapted, and a thread pool mapping relationship between the renewal data and the target thread pool is constructed according to the adaptation result, so that the renewal data can be directly placed in the target thread pool for processing, reducing the number of queries. The renewal data is adapted to the data parsing processor in the target thread pool, so as to obtain the optimal data parsing processor with the best degree of adaptation to the current renewal data. The renewal data is distributed through the data distribution relationship between the optimal data parsing processor and the renewal data. The data distribution process of the renewal data is dynamically operated and maintained by the data parsing processor, thereby improving the intelligence of the data parsing processing framework.

[0013] In a preferred example, the present application may be further configured as follows: after performing thread pool adaptation processing on the renewal data and constructing a thread pool mapping relationship between the renewal data and the target thread pool, and before performing adaptation processing on the renewal data and the data parsing processor in the target thread pool according to the thread pool mapping relationship and selecting the optimal data parsing processor with the best adaptation degree, it also includes: Performing data analysis and processing class classification processing on the renewal data, and establishing a data analysis and processing class mapping relationship between the renewal data and the target data analysis and processing class; According to the thread pool mapping relationship, a data parsing processing class thread pool mapping relationship is constructed between the target data parsing processing class and the target thread pool; According to the data parsing processing class thread pool mapping relationship, the target data parsing processing class is mapped to the target thread pool for batch data parsing processing.

[0014] By adopting the above technical solution, data parsing processing classes are allocated according to renewal data, thereby building a data parsing processing class mapping relationship between renewal data and target data parsing processing classes, which helps to map renewal data to corresponding data parsing processing classes according to the data parsing processing class mapping relationship, and build a data parsing processing class thread pool mapping relationship between the target data parsing processing class and the target thread pool according to the uniqueness of the renewal data, thereby facilitating indirect mapping of the target data parsing processing class to the corresponding target thread pool for batch parallel processing of data parsing, thereby improving the processing efficiency of the data parsing processing framework.

[0015] In a preferred example, the present application may be further configured as follows: according to the data analysis and processing rule, the data to be processed is mapped to a corresponding thread pool for data analysis processor adaptation, and the data analysis processor with the best adaptation degree is selected to perform data analysis processing on the data to be processed, and further includes: Acquire the data parsing and processing status of the renewal data in the target thread pool in real time; When the target data renewal time ends, determining whether the data analysis processing state is in a finished state; If not, the renewal data is marked as a data visible state, and a polling process is performed on the target data parsing processor so as to adapt to the data parsing processor in the idle state.

[0016] By adopting the above technical solution, through real-time monitoring of the data processing status of the renewal data in the target thread pool, it is helpful to know whether the renewal time of the target data is compatible with the data processing status. When the renewal time of the target data ends but the data processing has not ended, it means that the current renewal data has not been completely processed. Therefore, by marking the renewal data as a data visible state, other data parsing processors in an idle state can perform data parsing processing on the renewal data that has not been processed, thereby improving the renewal data. Can be discovered and automatically processed by the data parsing processing framework in a timely manner when the data visibility timeout state, reduce the incompleteness of data parsing processing.

[0017] In a preferred example, the present application may be further configured as follows: mapping the data to be processed to a corresponding thread pool for data analysis processor adaptation according to the data analysis processing rule, and selecting the data analysis processor with the best adaptation degree to perform data analysis processing on the data to be processed also includes: Obtaining the data analysis and processing progress of each of the to-be-processed data in real time; Determining whether a data analysis processing exception occurs in the data analysis processor according to the data analysis processing progress; If so, the abnormal data with data parsing processing exception is marked as a dead letter queue, and the data parsing processor is reallocated through the data parsing processing framework to re-drive the abnormal data for data parsing processing.

[0018] By adopting the above technical solution, it is determined whether data parsing exceptions occur according to the progress of data parsing, including abnormal interruption of data parsing, error in data parsing adaptation conditions, and loss of parsing protocol packets, etc. If the data to be processed in the abnormal state is not parsed and processed, it is easy to cause data queue congestion. The abnormal data is marked as a dead letter queue, and the abnormal data is isolated through the dead letter queue, and the cause of the abnormality is analyzed, so that the data parsing processing framework can reallocate the data parsing processor according to the cause of the abnormality, and drive the reallocated data parsing processor to perform data parsing on the abnormal data, thereby improving the overall operation efficiency of the data parsing processing framework, making each group of data in the entire data queue run in a parallel and orderly thread pool, and improving the orderliness of data parsing processing of the data parsing processing framework.

[0019] The second object of the invention is achieved by the following technical solutions: A multi-data data monitoring and analysis system is provided, the multi-data data monitoring and analysis system comprising: A data processing request receiving module, used for receiving in real time a data parsing request for data processing on a data queue to be processed; A data preprocessing module, used to preprocess the pre-trained data analysis processing framework according to the data analysis processing request, to obtain initialized data analysis processing rules that meet the data analysis processing request; A to-be-processed data acquisition module is used to acquire the to-be-processed data carried in the to-be-processed data queue; The data analysis and processing module is used to map the data to be processed to the corresponding thread pool for data analysis processor adaptation according to the data analysis and processing rules, and select the data analysis processor with the best adaptation degree to perform data analysis and processing on the data to be processed.

[0020] By adopting the above technical solution, since the data parsing queue service can only use the groups in the queue, it is impossible to dynamically create a data parsing queue according to the actual data to be parsed, and hard coding usually randomly pulls the data to be parsed to group the data parsing queue. The data parsing process is relatively messy and the processing pressure of the data parsing queue is large. Therefore, a pre-trained data parsing processing framework is initialized according to the data parsing processing request, so that the obtained data parsing processing rules meet the current data parsing processing requirements, improve the adaptability of the data parsing processing framework and the current data queue to be processed, and by mapping the data to be processed to the thread pool, the data parsing processor with the best adaptability performs data parsing processing on the data to be processed, so that each data to be processed is parsed by the corresponding thread pool and distributed to the corresponding data parsing processor for processing, so that each group of data in the data queue to be processed can be processed in parallel, thereby improving the orderliness of data parsing processing.

[0021] The third objective of the present application is achieved through the following technical solutions: A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned multi-data data monitoring and analysis method when executing the computer program.

[0022] The fourth objective of the present application is achieved through the following technical solutions: A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned multi-data data monitoring and analysis method.

[0023] In summary, the present application includes at least one of the following beneficial technical effects: 1. Initialize the pre-trained data analysis and processing framework according to the data analysis and processing request, so that the obtained data analysis and processing rules meet the current data analysis processing requirements, improve the adaptability of the data analysis and processing framework to the current data queue to be processed, and map the data to be processed to the thread pool, and use the data analysis processor with the best adaptability to perform data analysis on the data to be processed, so that each data to be processed is analyzed by the corresponding thread pool and distributed to the corresponding data analysis processor for processing, so that each group of data in the data queue to be processed can be processed in parallel, and the orderliness of data analysis and processing is improved; 2. Construct a corresponding target thread pool in the data parsing processing framework through the data in the data parsing processing request, improve the degree of fit between the data to be parsed and the target thread pool, and re-divide the data processing class according to the current data, so that the target data processing class fits the data queue to be processed. Through the mapping relationship between the data queue to be processed and the target thread pool, and between the data queue to be processed and the target data parsing processing class, the data parsing processing class is mapped to the target thread pool for operation, thereby mobilizing the corresponding target thread pool to perform parallel processing on the data queue to be processed, improving the orderliness of data queue processing, and controlling the target thread pool to perform data parsing processing on the data queue to be processed by the data parsing processing rules, thereby improving the efficiency of data parsing processing in the data queue; 3. Dynamically divide the data capacity of each group in the data queue to be processed according to the data analysis and processing request, so as to obtain the data capacity threshold of the maximum data receiving quantity of each group of data, so as to control the division of each group of data to be processed within the optimal processing range of the data analysis and processing framework, and adjust the data visibility time and data renewal time of the data queue respectively according to the data processing request, and update the adjusted target data visibility time and target data renewal time to the data processing time rule, so as to obtain the data analysis and processing time rule adapted to the data queue to be processed, so as to quickly initialize the data of each group of data queue to be processed according to the data analysis and processing time rule, and improve the processing efficiency of the data queue by the data analysis and processing framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart for implementing a data monitoring and analysis method for multiple data according to an embodiment of the present application.

[0025] Figure 2 This is a flowchart for implementing step S20 of the multi-data data monitoring and analysis method according to an embodiment of the present application.

[0026] Figure 3 This is another implementation flowchart of step S20 of the multi-data data monitoring and analysis method according to an embodiment of the present application.

[0027] Figure 4 It is a flowchart for implementing step S40 of the data monitoring and analysis method for multiple data according to an embodiment of the present application.

[0028] Figure 5 It is a flowchart for implementing batch processing of data analysis by a data monitoring and analysis method for multiple data in one embodiment of the present application.

[0029] Figure 6 It is an implementation flow chart of a data monitoring and analysis method for monitoring data analysis processing status of multiple data in one embodiment of the present application.

[0030] Figure 7 This is a flowchart of an implementation of a multi-data data monitoring and analysis method for reducing data queue congestion according to an embodiment of the present application.

[0031] Figure 8 It is a structural diagram of a multi-data data monitoring and analysis system according to an embodiment of the present application.

[0032] Fig. 9 It is a schematic diagram of the internal structure of a computer device used to implement a data processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] The present application is further described in detail below in conjunction with the accompanying drawings.

[0034] In one embodiment, if Figure 1 As shown, the present application discloses a multi-data data monitoring and analysis method, which specifically includes the following steps: S10: receiving a data parsing request for processing data in a queue of data to be processed in real time.

[0035] Specifically, a data connection channel is established between each user terminal and the data parsing and processing framework through the registration information of each user. When the user sends the data to be parsed to the gateway, the data parsing and processing framework pulls the user's data to be parsed from the gateway and stores each data to be parsed in the queue of data to be processed according to the user's unique identification code, and generates a data parsing processing request for data processing on the queue of data to be processed.

[0036] S20: Preprocessing a pre-trained data analysis and processing framework according to the data analysis and processing request to obtain initialized data analysis and processing rules that meet the data analysis and processing request.

[0037] Specifically, Figure 2 As shown, step S20 includes the following steps: S101: According to the data parsing processing request, a target thread pool adapted to the queue of data to be processed is constructed.

[0038] Specifically, according to the data parsing processing request carrying the information of the data queue to be processed, the initial thread pool of the data parsing processing framework is initialized. For example, according to the type of each group of data analysis in the data queue to be processed, the parameters of the initial thread pool in the data parsing processing framework are reallocated so that the allocated target thread pool matches the data queue to be processed.

[0039] S102: Re-dividing the data parsing and processing classes in the data parsing and processing framework to obtain a target data processing class that is compatible with the queue of data to be processed.

[0040] Specifically, according to the data parsing processing request, the data parsing processing classes in the data parsing processing framework are re-divided. For example, according to the parameter type of each group of data in the data queue to be processed, the parameters of the initial data parsing processing class in the data parsing processing framework are reset, so that the re-divided target data parsing processing class in the data parsing processing framework is adapted to the data queue to be processed.

[0041] S103: respectively obtaining mapping relationships between the to-be-processed data queue and the target thread pool and the target data processing class.

[0042] Specifically, according to the matching degree between the data queue to be processed and the target thread pool, a one-to-one mapping relationship between each queue group in the data queue to be processed and the corresponding target thread pool is constructed; according to the matching degree between the data queue group and the target data parsing processing class, a one-to-one mapping relationship between each data queue group in the data queue to be processed and the target data parsing processing class is constructed.

[0043] S104: According to the mapping relationship, the target data processing class is mapped to the target thread pool for running, and a data analysis and processing rule adapted to the data analysis and processing request is obtained, wherein the data analysis and processing rule is used to control the target thread pool to perform data analysis and processing on the target data analysis and processing class.

[0044] Specifically, according to the mapping relationship between the data queue to be processed and the target thread pool and the target data parsing processing class, the target data parsing processing class is mapped to the corresponding target thread pool for operation. For example, taking the data queue created by the product as the middleware, the target data parsing processing class responsible for product creation is mapped to the corresponding target thread pool created by the product, thereby using the mapping and operation relationship between the target data parsing processing class and the target thread pool as a data parsing processing rule, so as to control the target thread pool to perform data processing on the mapped target data parsing processing class according to the data parsing processing rule. The data parsing processing rule includes the mapping relationship between the target data parsing processing class and the target thread pool and the relationship for performing data parsing processing on the data queue to be processed.

[0045] In this embodiment, in order to better allocate each group of to-be-parsed data in the to-be-processed data queue and reduce the repeated processing of the to-be-processed data by multiple data parsing processor recipients, as shown in FIG. Figure 3 As shown, step S20 also includes: S201: Re-dividing the data capacity of the to-be-processed data queue according to the data parsing and processing request to obtain a data capacity threshold that meets the data parsing and processing request.

[0046] Specifically, according to the data parsing processing request, the configuration file in the data parsing processing framework is initialized. For example, the data capacity of the data queue to be processed is redivided according to the capacity of the received data to be parsed, and the maximum number of data received by each data queue group is set, so that the grouping of data in the data queue to be processed meets the optimal operating efficiency of the data parsing processing framework, thereby obtaining a data capacity threshold that meets the data parsing processing request.

[0047] S202: Adjust the data visibility time of each group of data queues so that the adjusted data visibility time meets the processing requirement of the data capacity threshold, thereby obtaining the target data visibility time.

[0048] Specifically, the data visibility time refers to the length of time that the data analysis processor holds the data. During the data visibility time, the received data is only visible to the data analysis processor, reducing the probability of the same data being repeatedly received by multiple data analysis processors. For example, the duration of the data analysis processing is determined based on the data capacity, and this duration is used as the data visibility time, such as 20 seconds, so that the adjusted data visibility time meets the processing requirements of the data capacity threshold, thereby obtaining the target data visibility time.

[0049] S203: According to the target data visible time, the data renewal time of each group of data queues is adjusted to obtain a target data renewal time that meets the data processing request.

[0050] Specifically, the data renewal time of each group of data queues is dynamically adjusted according to the target data visibility time. For example, the target data visibility time of each group of data queues is monitored separately. When the target data visibility time ends, the target data renewal time is called to automatically renew the information of the data queue, such as renewing it once every 30 seconds. During the data renewal time, the data visibility is changed through the data analysis and processing framework, and the change result is returned to the data queue and the corresponding data visibility time is updated, thereby completing the automatic cyclic renewal of the data queue.

[0051] S204: According to the target data visible time and the target data renewal time, the data processing time rule of the data parsing and processing framework is updated to obtain the data processing time rule adapted to the queue of data to be processed.

[0052] Specifically, the target data visible time and the target data renewal time are updated to the initial data processing time rule of the data queue to be processed, thereby obtaining a data analysis and processing time rule that can automatically and cyclically renew the data queue to be processed, wherein the data analysis and processing time rule includes the data renewal time and the data visible time.

[0053] S30: Acquire the data to be processed carried in the queue of data to be processed.

[0054] Specifically, according to the group type of the queue of data to be processed, the data to be processed of the corresponding group is obtained, such as product creation data, product listing data, and product delisting data.

[0055] S40: According to the data analysis and processing rules, the data to be processed is mapped to a corresponding thread pool for data analysis processor adaptation, and the data analysis processor with the best adaptation degree is selected to perform data analysis and processing on the data to be processed.

[0056] Specifically, Figure 4 As shown, step S40 includes the following steps: S301: Performing data renewal processing on the data to be processed according to the data analysis and processing rules to obtain renewal data that runs according to the target data renewal time.

[0057] Specifically, according to the data analysis and processing rules, data analysis and renewal processing is performed on the data to be processed. For example, when the data analysis and processing framework receives the data to be analyzed for product listings, the data analysis and processing rules are called to renew the data to be analyzed for product listings, and the renewal is performed every 30 seconds, so that the data to be processed is cyclically renewed according to the target renewal time to obtain the renewal data.

[0058] S302: Perform thread pool adaptation processing on the renewal data, and construct a thread pool mapping relationship between the renewal data and the target thread pool.

[0059] Specifically, the renewal data is matched to the target thread pool through the data uniqueness of the renewal data, such as data classification and data unique identification code, to obtain a thread pool mapping relationship between the renewal data and the target thread pool.

[0060] S303: According to the thread pool mapping relationship, the renewal data is adapted to the data parsing processor in the target thread pool, and an optimal data parsing processor with the best adaptation degree is selected.

[0061] Specifically, according to the thread pool mapping relationship, the renewal data is put into the thread pool, and the data parsing distributor is called to find an adapted data parsing processor in the thread pool. The optimal data parsing processor with the best adaptation degree to the renewal data is recommended according to the adaptation result, such as adapting the renewal data created by the product to the data parsing processor created by the product, and adapting the renewal data of the product on the shelf to the data parsing processor for the product on the shelf.

[0062] S304: Establishing a data distribution relationship between the renewal data and the optimal data analysis processor, and performing data analysis processing on the renewal data according to the data distribution relationship.

[0063] Specifically, according to the degree of adaptation between the renewal data and the optimal data analysis processor, a data distribution relationship between the renewal data and the optimal data analysis processor is established, and the renewal data is analyzed and processed according to the data distribution relationship, such as distributing the renewal data to the corresponding data analysis receiving end through the corresponding data analysis processor.

[0064] In this embodiment, in order to better parse and process the data in batches, Figure 5 As shown, after performing thread pool adaptation processing on the renewal data and constructing a thread pool mapping relationship between the renewal data and the target thread pool, and before adapting the renewal data to the data parsing processor in the target thread pool according to the thread pool mapping relationship and selecting the optimal data parsing processor with the best adaptation degree, it also includes: S401: Perform data analysis and processing class classification processing on the renewal data, and establish a data analysis and processing class mapping relationship between the renewal data and a target data analysis and processing class.

[0065] Specifically, according to the mapping relationship between the renewal data and the target data parsing and processing class, the renewal data is classified and processed by the data parsing and processing class. For example, the renewal data to be parsed which is responsible for putting goods on the shelves is divided into the data parsing and processing class responsible for the product putting on the shelves, thereby obtaining the data parsing and processing class mapping relationship between the renewal data and the target data parsing and processing class.

[0066] S402: Constructing a data parsing processing class thread pool mapping relationship between the target data parsing processing class and the target thread pool according to the thread pool mapping relationship.

[0067] Specifically, according to the thread pool mapping relationship between the renewal data and the target thread pool, the unique identification code of the renewal data is used as the middleware to construct a data parsing processing class thread pool relationship between the target data parsing processing class and the target thread pool, that is, to build an indirect association relationship between the target thread pool and the target data parsing processing class, so as to facilitate mapping the target data parsing processing class of the same processing function, such as product listing processing, to the target thread pool for operation, thereby improving the parallel data parsing processing capability of the data parsing processing framework.

[0068] S403: According to the data parsing processing class thread pool mapping relationship, the target data parsing processing class is mapped to the target thread pool for batch data parsing processing.

[0069] Specifically, according to the mapping relationship of the data analysis processing class thread pool, the target data analysis processing class is mapped to the target thread pool of the corresponding processing function through the relevance of the renewal data, and the target data analysis processing class is controlled in the target thread pool to perform parallel processing of data analysis, so that the target data analysis processing class in the same target thread pool can perform batch parallel analysis on the data to be analyzed, thereby improving the data processing efficiency of the data analysis processing framework.

[0070] In this embodiment, in order to better monitor the processing status of the data to be parsed, such as Figure 6 As shown, according to the data parsing and processing rules, the data to be processed is mapped to the corresponding thread pool for data parsing processor adaptation, and the data parsing processor with the best adaptation degree is selected to perform data parsing and processing on the data to be processed, and further includes: S501: Acquire the data parsing and processing status of the renewal data in the target thread pool in real time.

[0071] Specifically, in the process of the thread pool parsing the renewal data, the data parsing and processing status of the renewal data is monitored in real time through the heartbeat packet. For example, in the data parsing and processing framework, when the target thread pool starts to parse the renewal data, a parsing start signal is sent to the server of the data parsing and processing framework to start the heartbeat monitoring mechanism of the server. The heartbeat monitoring mechanism is stopped until the heartbeat packet receiving link is interrupted or the parsing bundle signal is received, and the data parsing and processing status is judged whether it is in an abnormal state based on the monitoring data of the heartbeat packet.

[0072] S502: When the target data renewal time ends, determine whether the data analysis processing status is in a finished state.

[0073] Specifically, when the target data renewal time ends, if the target renewal time is set to be renewed every 30 seconds, when the target renewal time arrives, the data parsing and processing status is judged to be in the end state through the monitoring data of the heartbeat packet. If the heartbeat packet monitoring data shows that the link is interrupted, it is judged that the data parsing and processing is abnormal. Or before the target data renewal time ends, a parsing end signal is received in the monitoring data of the heartbeat packet, then it is judged that the data parsing and processing is abnormal. Within the error range, when the monitoring data of the heartbeat packet receives a parsing end signal at the end of the target data renewal time, it is judged that the data parsing and processing status is ended.

[0074] S503: If not, the renewal data is marked as a data visible state, and a polling process is performed on the target data parsing processor to facilitate adaptation with the data parsing processor in an idle state.

[0075] Specifically, if the data parsing processing status is not in the end state, it means that an exception occurs in the data parsing process, such as incomplete data parsing, etc., then the renewal data in the abnormal state is marked as a data visible state, and the data visible time of the renewal data in the abnormal state is modified, and the modified renewal data is sent to the target data parsing processor for polling processing, and the data parsing processor in the idle state is found. According to the mapping relationship between the renewal data and the target thread pool and the target data parsing processing class, the renewal data is adapted to the data parsing processor in the idle state, so as to facilitate timely parsing of the renewal data in the abnormal state.

[0076] S504: If yes, an end signal is sent to the heartbeat monitoring mechanism, and the corresponding data analysis processor is marked as idle to wait for the next data analysis processing request.

[0077] Specifically, if the data analysis and processing status is in the end state, this means that when the target data renewal time ends, a analysis end signal is received, and the data analysis process is judged to be normal. When the data analysis ends normally, an end signal is sent to the heartbeat monitoring mechanism, and the data analysis monitoring mechanism of the data analysis processing framework is adjusted to a sleep state to wait for the next wake-up signal, and the corresponding data analysis processor is marked as an idle state to wait for the next data analysis processing request.

[0078] In this embodiment, in order to reduce queue congestion caused by data parsing in abnormal conditions, Figure 7 As shown, according to the data parsing processing rule, mapping the data to be processed to the corresponding thread pool for data parsing processor adaptation, and selecting the data parsing processor with the best adaptation degree to perform data parsing processing on the data to be processed also includes: S601: Acquire the data analysis and processing progress of each of the to-be-processed data in real time.

[0079] Specifically, based on the data capacity of the data to be processed, the data parsing processing framework estimates the data parsing processing progress of the data to be processed. When a data parsing signal is received, the data parsing processing progress management mechanism is started, and the data parsing processing progress is judged as the ratio of parsed data to unparsed data in the target thread pool changes.

[0080] S602: Determine whether a data analysis process exception occurs in the data analysis processor according to the data analysis process progress.

[0081] Specifically, when the target renewal time is reached, the data parsing progress is judged based on the ratio of parsed data to unparsed data, and then the data parsing progress is used to judge whether a data parsing exception occurs in the data parsing processor. For example, when the target renewal time is reached, the data parsing progress is abnormally interrupted, and the parsed data to be processed has not yet been fully parsed, then it is judged that a data parsing exception occurs in the data parsing processor at this time, such as data corruption or loss.

[0082] S603: If yes, the abnormal data with data parsing processing exception is marked as a dead letter queue, and the data parsing processor is reallocated through the data parsing processing framework to re-drive the abnormal data for data parsing processing.

[0083] Specifically, when data parsing is abnormal, the abnormal data is marked as a dead letter queue and stored in the preset dead letter queue. When new data is stored in the dead letter queue, the re-driving strategy of the data parsing processing framework is started to access the data in the dead letter queue, analyze the cause of the data parsing abnormality, and reallocate the data parsing processor according to the cause of the data parsing abnormality, so as to re-drive the data parsing processor with processing authority to perform data parsing on the abnormal data.

[0084] S604: If not, continue to monitor the data analysis progress of the data to be processed, so as to obtain data with data analysis abnormalities in a timely manner.

[0085] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0086] In one embodiment, a multi-data data monitoring and analysis system is provided, and the multi-data data monitoring and analysis system corresponds one-to-one to the multi-data data monitoring and analysis method in the above embodiment. Figure 8 As shown, the multi-data data monitoring and analysis system includes a data processing request receiving module, a data preprocessing module, a to-be-processed data acquisition module and a data parsing and processing module. The detailed description of each functional module is as follows: A data processing request receiving module, used for receiving in real time a data parsing request for data processing on a data queue to be processed; A data preprocessing module, used to preprocess the pre-trained data analysis processing framework according to the data analysis processing request, to obtain initialized data analysis processing rules that meet the data analysis processing request; A to-be-processed data acquisition module is used to acquire the to-be-processed data carried in the to-be-processed data queue; The data analysis and processing module is used to map the data to be processed to the corresponding thread pool for data analysis processor adaptation according to the data analysis and processing rules, and select the data analysis processor with the best adaptation degree to perform data analysis and processing on the data to be processed.

[0087] For the specific definition of the multi-data data monitoring and analysis system, please refer to the definition of the multi-data data monitoring and analysis method above, which will not be repeated here. Each module in the above-mentioned multi-data data monitoring and analysis system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0088] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the data to be parsed in the queue of data to be processed and the intermediate file data generated during the data parsing process performed by the data parsing processing framework. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-data data monitoring and analysis method is implemented.

[0089] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned multi-data data monitoring and analysis method are implemented.

[0090] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0091] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

[0092] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data monitoring and analysis method for multiple data, characterized in that: The multi-data data monitoring and analysis method comprises: Receive data parsing processing requests for data processing on the data queue to be processed in real time; According to the data analysis and processing request, preprocessing is performed on the pre-trained data analysis and processing framework to obtain initialized data analysis and processing rules that meet the data analysis and processing request; Obtain the data to be processed in the queue of data to be processed; According to the data analysis and processing rules, the data to be processed is mapped to the corresponding thread pool for data analysis processor adaptation, and the data analysis processor with the best adaptation degree is selected to perform data analysis and processing on the data to be processed.

2. The multi-data data monitoring and analysis method according to claim 1, characterized in that: According to the data analysis and processing request, preprocessing the pre-trained data analysis and processing framework to obtain initialized data analysis and processing rules that meet the data analysis and processing request specifically includes: According to the data parsing processing request, a target thread pool adapted to the queue of data to be processed is constructed; Re-dividing the data parsing and processing classes in the data parsing and processing framework to obtain a target data processing class that is compatible with the queue of data to be processed; Respectively obtain the mapping relationship between the to-be-processed data queue and the target thread pool and the target data processing class; According to the mapping relationship, the target data processing class is mapped to the target thread pool for execution, and a data analysis and processing rule that matches the data analysis and processing request is obtained, wherein the data analysis and processing rule is used to control the target thread pool to perform data analysis and processing on the target data analysis and processing class.

3. The multi-data data monitoring and analysis method according to claim 2, characterized in that: The pre-processing of the pre-trained data analysis and processing framework according to the data analysis and processing request to obtain initialized data analysis and processing rules that meet the data analysis and processing request also includes: Re-dividing the data capacity of the to-be-processed data queue according to the data parsing and processing request to obtain a data capacity threshold that meets the data parsing and processing request; Adjusting the data visibility time of each group of data queues so that the adjusted data visibility time meets the processing requirements of the data capacity threshold, thereby obtaining the target data visibility time; According to the target data visibility time, adjusting the data renewal time of each group of data queues to obtain a target data renewal time that meets the data processing request; According to the target data visible time and the target data renewal time, the data processing time rule of the data parsing and processing framework is updated to obtain the data processing time rule adapted to the queue of data to be processed.

4. The multi-data data monitoring and analysis method according to claim 1, characterized in that: According to the data analysis and processing rules, mapping the data to be processed to the corresponding thread pool for data analysis processor adaptation, and selecting the data analysis processor with the best adaptation degree to perform data analysis and processing on the data to be processed specifically includes: According to the data analysis and processing rules, data renewal processing is performed on the data to be processed to obtain renewal data that runs according to the target data renewal time; Perform thread pool adaptation processing on the renewal data, and construct a thread pool mapping relationship between the renewal data and the target thread pool; According to the thread pool mapping relationship, the renewal data is adapted to the data parsing processor in the target thread pool, and the optimal data parsing processor with the best adaptation degree is selected; A data distribution relationship between the renewal data and the optimal data parsing processor is established, and data parsing processing is performed on the renewal data according to the data distribution relationship.

5. The multi-data data monitoring and analysis method according to claim 4, characterized in that: After performing thread pool adaptation processing on the renewal data and constructing a thread pool mapping relationship between the renewal data and the target thread pool, and before performing adaptation processing on the renewal data and the data parsing processor in the target thread pool according to the thread pool mapping relationship and selecting the optimal data parsing processor with the best adaptation degree, it also includes: Performing data analysis and processing class classification processing on the renewal data, and establishing a data analysis and processing class mapping relationship between the renewal data and the target data analysis and processing class; According to the thread pool mapping relationship, a data parsing processing class thread pool mapping relationship is constructed between the target data parsing processing class and the target thread pool; According to the data parsing processing class thread pool mapping relationship, the target data parsing processing class is mapped to the target thread pool for batch data parsing processing.

6. The multi-data data monitoring and analysis method according to claim 4, characterized in that: According to the data analysis and processing rules, the data to be processed is mapped to the corresponding thread pool for data analysis processor adaptation, and the data analysis processor with the best adaptation degree is selected to perform data analysis and processing on the data to be processed, and further includes: Acquire the data parsing and processing status of the renewal data in the target thread pool in real time; When the target data renewal time ends, determining whether the data analysis processing state is in a finished state; If not, the renewal data is marked as a data visible state, and a polling process is performed on the target data parsing processor so as to adapt to the data parsing processor in the idle state.

7. The multi-data data monitoring and analysis method according to claim 1, characterized in that: According to the data analysis and processing rule, mapping the data to be processed to the corresponding thread pool for data analysis processor adaptation, and selecting the data analysis processor with the best adaptation degree to perform data analysis and processing on the data to be processed also includes: Obtaining the data analysis and processing progress of each of the to-be-processed data in real time; Determining whether a data analysis processing exception occurs in the data analysis processor according to the data analysis processing progress; If so, the abnormal data with data parsing processing exception is marked as a dead letter queue, and the data parsing processor is reallocated through the data parsing processing framework to re-drive the abnormal data for data parsing processing.

8. A multi-data data monitoring and analysis system, characterized in that: The multi-data data monitoring and analysis system comprises: A data processing request receiving module, used for receiving in real time a data parsing request for data processing on a data queue to be processed; A data preprocessing module, used to preprocess the pre-trained data analysis processing framework according to the data analysis processing request, to obtain initialized data analysis processing rules that meet the data analysis processing request; A to-be-processed data acquisition module is used to acquire the to-be-processed data carried in the to-be-processed data queue; The data analysis and processing module is used to map the data to be processed to the corresponding thread pool for data analysis processor adaptation according to the data analysis and processing rules, and select the data analysis processor with the best adaptation degree to perform data analysis and processing on the data to be processed.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multi-data data monitoring and analysis method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the multi-data data monitoring and analysis method according to any one of claims 1 to 7 are implemented.

Images