A data processing method and device
By obtaining the set of pending texts, determining the processing node cluster, and performing sensitive word analysis, the problem of failure to effectively monitor non-functional service quality in the existing technology is solved, comprehensive monitoring of service quality is achieved, and monitoring accuracy is improved. Cost control.
Patent Information
- Application Number
- CN202011077971.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-10-10
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-10-10
AI Technical Summary
The existing service quality monitoring is mainly monitored from the functional aspect, and the non-functional service quality is not effectively monitored, resulting in the incomplete monitoring of equipment service quality.
By obtaining the set of pending texts, determining the processing node cluster, performing sensitive words analysis to obtain the analysis results of non-functional service quality, and performing early warning processing when abnormalities are found, so as to achieve monitoring of non-functional service quality.
Improve the accuracy of service quality monitoring, and automatically scale processing nodes, match processing capabilities and data volume, control monitoring costs, and ensure service stability.
Smart Images

Figure CN112183053B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of service quality monitoring, and in particular, relates to a data processing method and device. Background Art
[0002] Service quality monitoring is the basic operation and maintenance work of each Internet platform. Through service quality monitoring, each device (such as server) used to provide services in each Internet platform is monitored. Service quality monitoring includes hardware monitoring and software monitoring. If at least one of the monitoring methods of hardware monitoring and software monitoring is determined to have an abnormal trend, early warning processing will be carried out to effectively improve the service quality of the equipment.
[0003] At present, the monitoring of each device providing services is performed by the monitoring terminal in each Internet platform. The monitoring terminal obtains hardware data and software data from the specified location. The hardware data and software data are collected by the collection software deployed in each device providing services. For example, hardware data includes but is not limited to CPU (Central Processing Unit) usage rate, memory usage rate, IO (Input-Output) usage status, etc., and software data includes but is not limited to log data. The monitoring terminal monitors the hardware data and software data in real time. If it is determined that at least one of the hardware data and software data is abnormal, an early warning process is performed. However, the current service quality monitoring is carried out with the help of the hardware data and software data of each device providing services, so as to monitor the functional service quality of each device providing services. Summary of the invention
[0004] In view of this, the purpose of this application is to provide a data processing method and device for monitoring service quality from the perspective of non-functional service quality. The technical solution is as follows:
[0005] In one aspect, the present application provides a data processing method, the method comprising:
[0006] Acquire a set of texts to be processed at the current moment, wherein the set of texts to be processed includes at least one text to be processed, and the text to be processed is a text used for monitoring service quality from the aspect of non-functional service quality;
[0007] Determining, from a node cluster, a processing node corresponding to the text to be processed according to the data volume of the text to be processed in the text set to be processed;
[0008] Performing sensitive word analysis on each of the to-be-processed texts through a determined processing node to obtain an analysis result indicating the service quality from the aspect of the non-functional service quality;
[0009] If the analysis result indicates that the service is abnormal, an early warning process is performed.
[0010] Optionally, determining, from a node cluster, a processing node corresponding to the text to be processed according to the data volume of the text to be processed in the text set to be processed comprises:
[0011] Determining, according to the data amount of the text to be processed in the text set to be processed, whether the processing node currently in working state in the node cluster can process all the text to be processed;
[0012] If the processing node currently in working state can process all the texts to be processed, the processing node currently in working state is determined as the processing node corresponding to the texts to be processed;
[0013] If the processing node currently in working state cannot process all the texts to be processed, at least one processing node in working state is added to the node cluster according to the relationship between the data volume of the texts to be processed in the text set to be processed and the processing capacity of the processing node currently in working state, so as to allocate a processing node that matches the data volume of the texts to be processed in the text set to be processed.
[0014] Optionally, adding at least one processing node in a working state in the node cluster includes:
[0015] If there is a processing node in a non-working state in the node cluster, and the processing node in a non-working state and the processing node in a working state can process all the texts to be processed, then control the processing node in a non-working state in the node cluster to switch to a working state;
[0016] If there are processing nodes in a non-working state in the node cluster, but the processing nodes in a non-working state and the processing nodes in a working state cannot process all the texts to be processed, then the processing nodes in a non-working state in the node cluster are controlled to switch to a working state, and processing nodes for processing the texts to be processed are deployed in the node cluster and controlled to enter a working state;
[0017] If there is no processing node in a non-working state in the node cluster, a processing node for processing the to-be-processed text is deployed in the node cluster and controlled to enter a working state.
[0018] Optionally, the performing sensitive word analysis on each of the to-be-processed texts by the determined processing node to obtain an analysis result indicating the service quality from the aspect of the non-functional service quality includes:
[0019] For any of the texts to be processed, perform word segmentation on the text to be processed to obtain each word in the text to be processed;
[0020] Calculating the word similarity between each word and each sensitive word in the preset sensitive word library, and calculating the word similarity between each word and the derivative of each sensitive word in the preset sensitive word library;
[0021] Based on the word similarities of all the texts to be processed, an analysis result indicating the service quality from the aspect of the non-functional service quality is obtained.
[0022] Optionally, obtaining the analysis result indicating the service quality from the aspect of the non-functional service quality based on the word similarity of all the texts to be processed includes:
[0023] For any of the texts to be processed, based on the word similarity of the text to be processed and the sensitivity level of the sensitive words corresponding to the word similarity, a score of the text to be processed for the service quality is obtained;
[0024] Based on the business type of the text to be processed, the text to be processed in the set of text to be processed is divided to obtain a text subset to which the text to be processed corresponding to the same business type belongs;
[0025] Based on the scores of all the to-be-processed texts in the text subset for the service quality, the score of the text subset for the service quality is obtained, and the score of the text subset for the service quality is used as the analysis result of the text subset.
[0026] Optionally, if the analysis result indicates that the service is abnormal, performing early warning processing includes:
[0027] If the analysis result indicates a service anomaly, and a target warning rule matching the to-be-processed text set is found from the warning rules of the preset sensitive word library, warning processing is performed.
[0028] In another aspect, the present application provides a data processing device, the device comprising:
[0029] An acquisition unit, used for acquiring a set of to-be-processed texts at a current moment, wherein the set of to-be-processed texts includes at least one to-be-processed text, and the to-be-processed text is a text used for monitoring service quality from the aspect of non-functional service quality;
[0030] A determination unit, configured to determine, from a node cluster, a processing node corresponding to the text to be processed according to the data volume of the text to be processed in the text set to be processed;
[0031] An analysis unit, configured to perform sensitive word analysis on each of the to-be-processed texts through a determined processing node, so as to obtain an analysis result indicating the service quality from the aspect of the non-functional service quality;
[0032] The early warning unit is used to perform early warning processing if the analysis result indicates that the service is abnormal.
[0033] Optionally, the determining unit includes:
[0034] A first determining subunit is used to determine whether a processing node currently in working state in the node cluster can process all the texts to be processed according to the data amount of the texts to be processed in the text set to be processed;
[0035] A selection subunit is used to determine the processing node currently in working state as the processing node corresponding to the text to be processed if the processing node currently in working state can process all the text to be processed;
[0036] A sub-unit is added, which is used to add at least one working processing node in the node cluster if the processing node currently in working state cannot process all the texts to be processed, according to the relationship between the data amount of the texts to be processed in the text set to be processed and the processing capacity of the processing node currently in working state, so as to allocate a processing node that matches the data amount of the texts to be processed in the text set to be processed.
[0037] Optionally, the additional subunit is used to control the processing nodes in the node cluster that are in a non-working state to switch to a working state if there are processing nodes in the node cluster that are in a non-working state and the processing nodes in a working state can process all the texts to be processed;
[0038] If there are processing nodes in a non-working state in the node cluster, but the processing nodes in a non-working state and the processing nodes in a working state cannot process all the texts to be processed, then the processing nodes in a non-working state in the node cluster are controlled to switch to a working state, and processing nodes for processing the texts to be processed are deployed in the node cluster and controlled to enter a working state;
[0039] If there is no processing node in a non-working state in the node cluster, a processing node for processing the to-be-processed text is deployed in the node cluster and controlled to enter a working state.
[0040] Optionally, the analysis unit is used to perform word segmentation processing on any of the texts to be processed to obtain each word in the text to be processed; calculate the word similarity between each word and each sensitive word in a preset sensitive word library, and calculate the word similarity between each word and the derivatives of each sensitive word in the preset sensitive word library; based on the word similarities of all texts to be processed, obtain an analysis result indicating the service quality from the aspect of the non-functional service quality.
[0041] The above data processing method and device obtain the current time set of to-be-processed texts, the to-be-processed text set includes at least one to-be-processed text, and the to-be-processed text is a text used to monitor the service quality from the non-functional service quality aspect; according to the data volume of the to-be-processed texts in the to-be-processed text set, determine the processing node corresponding to the to-be-processed text from the node cluster; perform sensitive word analysis on each to-be-processed text through the determined processing node to obtain the analysis result indicating the service quality from the non-functional service quality aspect; if the analysis result indicates that the service is abnormal, perform early warning processing to realize monitoring the service quality from the non-functional service quality aspect, so as to increase the monitoring of the non-functional service quality aspect on the basis of the functional service quality aspect, and improve the monitoring accuracy. And in the process of monitoring from the non-functional service quality aspect, according to the data volume of the to-be-processed texts in the to-be-processed text set, determine the processing node corresponding to the to-be-processed text from the node cluster, so that the processing node used to process the to-be-processed text matches the data volume of the to-be-processed texts in the to-be-processed text set, realize the automatic scaling capacity of the processing nodes in the node cluster, and effectively control the monitoring cost while ensuring the stability of the service. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0043] Figure 1 is a flow chart of a data processing method provided in an embodiment of the present application;
[0044] Figure 2 is a system architecture diagram of a processing system corresponding to the data processing method provided in an embodiment of the present application;
[0045] Figure 3 is a schematic diagram of a data processing method provided in an embodiment of the present application;
[0046] Figure 4 It is a structural diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0048] See also Figure 1 , which shows a flow chart of a data processing method provided by an embodiment of the present application, which may include the following steps:
[0049] 101: Obtain the current set of texts to be processed, the set of texts to be processed includes at least one text to be processed, and the text to be processed is a text used to monitor the service quality from the non-functional service quality aspect. The functional service quality aspect is mainly aimed at each device used to provide services, and the service quality is monitored from the hardware data and software data of each device used to provide services, such as the above-mentioned CPU usage, memory usage, IO usage status and log data, that is, collecting data reflecting the changes of each device itself in the process of providing services, because these data can reflect the changes of the device itself, and the changes of the device itself are mainly changes in the function of the device, such as an increase in CPU usage, etc., so the data reflecting the changes of the device itself can monitor the service quality of the device from the functional service quality aspect.
[0050] Non-functional service quality corresponds to functional service quality. Unlike monitoring functional service quality from data that reflects changes in the device itself, non-functional service quality can be understood as user experience. In the process of services provided by various devices, users will comment on the services, etc. In this process, text for monitoring service quality from the perspective of non-functional service quality (or user experience) will be generated, such as generating UGC (User Generated Content) for the service. These UGCs are users' feelings about the services, etc. Therefore, UGC that indicates users' feelings about the services can be used as text to be processed.
[0051] The text to be processed can be obtained through, but not limited to, the comment area and the bullet screen area. These comment areas and bullet screen areas can be located in the service interface that provides the service, so that users can directly evaluate the service experience in the comment area or bullet screen area of the service interface, eliminating the need for users to find the complaint entry from the service interface, and realizing online complaints about the service experience. Of course, the text to be processed can also be obtained from the complaint interface corresponding to the complaint entry. Compared with the comment area or the bullet screen area, the user needs to find the complaint entry from the service interface, and then trigger the complaint entry to display the complaint interface, so that the user can enter the service experience in the complaint interface. The above-mentioned comment area or bullet screen area can directly enter the service experience in the service interface, which is convenient for users to input.
[0052] In this embodiment, the text to be processed can be obtained through the data access module. For example, the data access module is associated with the service interface. During the process of inputting the text to be processed regarding the usage experience of the service in the service interface, the data access module can monitor the user's input operation, thereby obtaining the text to be processed through the data access module.
[0053] After obtaining the text to be processed, the text to be processed is stored in a message queue to achieve effective buffering of high-concurrency data through the message queue and the data access module to control the amount of text to be processed at the same time. The message queue can be in the form of but not limited to a Kafka queue, and any text to be processed collected by the data access module can be stored in the Kafka queue.
[0054] The above-mentioned current time set of to-be-processed texts is a set of to-be-processed texts for sensitive word analysis at the current time. The data volume of the to-be-processed texts in the to-be-processed text set is not limited in this embodiment. The to-be-processed text set can be obtained from, but not limited to, a kafka queue, such as extracting one to-be-processed text piece by piece from the kafka queue in a first-in-first-out manner. The data volume of the to-be-processed texts in the to-be-processed text sets at different times can be the same or different, which is not specifically limited in this embodiment.
[0055] 102: According to the data volume of the text to be processed in the text set to be processed, determine the processing node corresponding to the text to be processed from the node cluster, and with reference to the data volume of the text to be processed in the current text set to be processed, allocate the processing node matching the current data volume, so that the current processing capacity of the node cluster matches the current data volume to be processed, so that the node cluster can automatically scale its processing capacity with reference to the current data volume to be processed.
[0056] One way to determine the processing node corresponding to the text to be processed from the node cluster is as follows:
[0057] 1) According to the data volume of the to-be-processed texts in the to-be-processed text set, determine whether the processing node currently in working state in the node cluster can process all the to-be-processed texts. Whether the processing node currently in working state can process all the to-be-processed texts depends on the processing capacity of the processing node currently in working state. If the processable data volume indicated by the processing capacity is greater than the data volume of the to-be-processed texts in the to-be-processed text set, determine that the processing node currently in working state can process all the to-be-processed texts, otherwise determine that the processing node currently in working state cannot process all the to-be-processed texts.
[0058] The processing capacity of the processing node currently in working state is determined by the currently available resources of the processing node, such as the current CPU availability (ie, remaining usage), memory status, etc., which will not be described in detail in this embodiment.
[0059] 2) If the processing node currently in working state can process all the texts to be processed, the processing node currently in working state is determined as the processing node corresponding to the texts to be processed.
[0060] 3) If the processing node currently in working state cannot process all the texts to be processed, at least one processing node in working state is added to the node cluster according to the relationship between the data amount of the texts to be processed in the text set to be processed and the processing capacity of the processing node currently in working state, so as to allocate a processing node that matches the data amount of the texts to be processed in the text set to be processed.
[0061] The relationship between the data volume of the text to be processed in the text to be processed set and the processing capacity of the processing node currently in working state is used to characterize the additional processing capacity required to process all the text to be processed in the text to be processed set. For example, the relationship between the data volume and the processing capacity is used to indicate the number of processing nodes added in the node cluster and the number of text to be processed that the added processing nodes need to process, so that additional processing nodes in working state are added in the node cluster based on this information so as to be able to process all the text to be processed in the current text to be processed set.
[0062] In this embodiment, the method of adding at least one processing node in working state includes but is not limited to the following methods:
[0063] If there are processing nodes in a non-working state in the node cluster, and the processing nodes in a non-working state and the processing nodes in a working state can process all the texts to be processed, the processing nodes in a non-working state in the control node cluster are switched to a working state. In other words, if the existing processing nodes in the node cluster can process all the texts to be processed, the processing nodes in a non-working state in the control node cluster are started and switched to a working state to process the texts to be processed.
[0064] If there are processing nodes in a non-working state in the node cluster, but the processing nodes in a non-working state and the processing nodes in a working state cannot process all the texts to be processed, then the processing nodes in a non-working state in the node cluster are controlled to switch to a working state, and processing nodes for processing the texts to be processed are deployed in the node cluster and controlled to enter a working state.
[0065] The processing nodes in the non-working state and the processing nodes in the working state cannot process all the texts to be processed, which means that the existing processing nodes in the node cluster cannot process all the texts to be processed, and there are processing nodes that have not been started. In this case, on the one hand, the processing nodes in the non-working state in the node cluster are controlled to start up and switch to the working state to process the texts to be processed. On the other hand, new processing nodes need to be deployed in the node cluster and the controller needs to enter the working state so that the processing nodes in the current node cluster can process all the texts to be processed in the set of texts to be processed.
[0066] If there is no processing node in a non-working state in the node cluster, a processing node for processing the to-be-processed text is deployed in the node cluster and controlled to enter a working state.
[0067] One point that needs to be explained here is that the number of processing nodes in working state in the node cluster and the total number of processing nodes in the node cluster are related to the data volume of the text to be processed in the text collection to be processed. They can change according to the change of the data volume of the text to be processed, thereby realizing the automatic scaling capacity of the node cluster.
[0068] In this embodiment, all processing nodes in the node cluster include a cluster of master processing nodes (also called Master nodes) and slave processing nodes (also called nodes). The master processing node receives a set of texts to be processed, and controls the slave processing nodes in the node cluster that are in working state according to the set of texts to be processed and the configuration data in the slave processing nodes. The configuration data is used to indicate the processing capability of the slave processing nodes. For example, the configuration data includes but is not limited to data indicating the available status of each hardware in the slave processing node, such as data indicating the available status of the CPU and memory. The master processing node controls the status of the slave processing node according to the configuration data to determine an adapted slave processing node from all the slave processing nodes.
[0069] The node cluster can be built through docker technology and k8s technology. K8s technology is used to manage slave processing nodes. K8s technology can determine the status of slave processing nodes and whether to increase slave processing nodes based on configuration data to achieve automatic scaling. In the case of automatic scaling, especially expansion, the sensitive word analysis service for the text to be processed can be deployed to the newly added slave processing nodes through docker technology to simplify service deployment.
[0070] After determining the processing nodes, this embodiment can allocate the number of texts to be processed corresponding to each processing node according to the configuration data of the processing node. For example, if the configuration data of the processing node indicates that the processing node has a higher processing capability, a greater number of texts to be processed are allocated. Similarly, if the configuration data of the processing node indicates that the processing node has a lower processing capability, a smaller number of texts to be processed are allocated, thereby achieving load balancing among the processing nodes. This embodiment does not limit the number of texts to be processed that are allocated.
[0071] 103: Perform sensitive word analysis on each to-be-processed text through the determined processing node to obtain an analysis result indicating the service quality from the aspect of non-functional service quality.
[0072] It can be understood that: sensitive words are pre-set words that indicate service abnormalities from the perspective of non-functional service quality, and the sensitive words are stored in a preset sensitive word library. The acquisition method of the sensitive words in the preset sensitive word library includes but is not limited to: extracting words that can indicate service abnormalities from the text corresponding to the historical warning processing, and using the extracted words as sensitive words; displaying a word input interface, and acquiring the sensitive words entered by the user in the word input interface.
[0073] The preset sensitive word library storing sensitive words is obtained in the above manner, so as to manage and maintain sensitive words related to service quality through the preset sensitive word library, such as: stuck, slow, black screen, etc. However, a sensitive word can be expressed by other words, such as the sensitive word "idiot" can also be expressed by stupid, idiot, etc. Therefore, after obtaining the sensitive word, the corresponding derivative word of the sensitive word can be added to the preset sensitive word library to increase the coverage of the preset sensitive word library and meet the needs of multiple descriptions. For example, the target text for extracting sensitive words is segmented to obtain multiple words in the target text; sensitive words are determined from multiple words, and synonyms of sensitive words are determined by analyzing the meaning of sensitive words, and synonyms of sensitive words are regarded as derivatives of sensitive words. For example, LUCENCE and NLP (Natural Language Processing) can be combined to obtain sensitive words and derivatives of sensitive words, which will not be described in detail in this embodiment.
[0074] In this embodiment, one way to perform sensitive word analysis on each text to be processed to obtain the analysis result is as follows:
[0075] For any text to be processed, the text to be processed is segmented to obtain each word in the text to be processed; the word similarity between each word and each sensitive word in the preset sensitive word library is calculated, and the word similarity between each word and the derivative words of each sensitive word in the preset sensitive word library is calculated; based on the word similarity of all texts to be processed, an analysis result indicating the service quality from the aspect of non-functional service quality is obtained.
[0076] The word segmentation processing of the text to be processed includes but is not limited to using at least one of LUCENCE and NLP to perform word segmentation processing to split the text to be processed into independent words, so that it can be compared with the sensitive words and derivative words in the preset sensitive word library. The method for calculating the word similarity can refer to but is not limited to the current similarity calculation algorithm, which will not be elaborated in this embodiment.
[0077] The higher the word similarity, the more similar the words in the text to be processed are to the words in the preset sensitive word library, and vice versa, the more different the words in the text to be processed are from the words in the preset sensitive word library. Based on this, this embodiment pre-sets a similarity threshold. If the word similarity is greater than the similarity threshold, it is determined that the words in the text to be processed are hit in the preset sensitive word library, that is, they appear in the preset sensitive word library; if the word similarity is less than or equal to the similarity threshold, it is determined that the words in the text to be processed are not hit in the preset sensitive word library, that is, they do not appear in the preset sensitive word library, and the analysis result is obtained by counting the number of hits of the words in all the texts to be processed in the set of texts to be processed. For example, if the number of hits is greater than the threshold, an analysis result indicating that the service is abnormal is obtained; if the number of hits is less than or equal to the threshold, an analysis result indicating that the service is normal is obtained.
[0078] In addition to using the method of number of hits, this embodiment can also use other methods to obtain analysis results. For example, for any text to be processed, based on the word similarity of the text to be processed and the sensitivity level of the sensitive words corresponding to the word similarity, the score of the service quality of the text to be processed is obtained, and based on the scores of the service quality of all texts to be processed, the analysis result is obtained, such as averaging the scores of all texts to be processed for service quality, and taking the average score as the analysis result.
[0079] Among them, the sensitivity level of the sensitive word is used to characterize the degree of service abnormality indicated by the sensitive word. The higher the sensitivity level, the higher the degree of service abnormality, and the higher the weight corresponding to the corresponding sensitivity level, so that the proportion occupied by the sensitive word in the evaluation process is also greater. Based on this, this embodiment obtains the score of the service quality of the text to be processed based on the word similarity and the weight of the sensitive word corresponding to the word similarity, such as weighted summing the word similarity by weight to obtain the score of the service quality of the text to be processed.
[0080] In this embodiment, the analysis results can be differentiated by business type to obtain the analysis results corresponding to each business type. The process is as follows:
[0081] For any text to be processed, based on the word similarity of the text to be processed and the sensitivity level of the sensitive words corresponding to the word similarity, the score of the text to be processed for service quality is obtained; based on the business type of the text to be processed, the text to be processed in the text set to be processed is divided to obtain the text subsets to which the text to be processed corresponding to the same business type belongs; based on the scores of all the texts to be processed in the text subset for service quality, the score of the text subset for service quality is obtained, and the score of the text subset for service quality is used as the analysis result of the text subset.
[0082] The difference from the above analysis result based on the service quality score of all the to-be-processed texts is that: the analysis result of the text subset corresponding to each business type is calculated based on the text subset corresponding to a business type, so that the service quality of the business type can be monitored. For the process of obtaining the analysis result of the text subset corresponding to each business type, please refer to the analysis result obtained based on the service quality score of all the to-be-processed texts, which is not elaborated in this embodiment.
[0083] 104: If the analysis result indicates that the service is abnormal, an early warning process is performed, such as outputting an alarm message to a monitoring terminal, and an operator at the monitoring terminal processes the alarm message based on the alarm message, such as outputting the alarm message by at least one of, but not limited to, email and SMS notification.
[0084] In this embodiment, if the analysis result indicates a service abnormality, it is also possible to determine whether to perform warning processing in combination with the warning rules in the preset sensitive word library. A feasible method is: if the analysis result indicates a service abnormality and a target warning rule matching the text set to be processed is found from the warning rules in the preset sensitive word library, then warning processing is performed.
[0085] The warning rules in the preset sensitive word library are configured with warning trigger conditions. For example, one warning trigger condition is the number of times a sensitive word appears within a preset time. For instance, if the sensitive word "lag" appears 1000 times within 1 minute, a warning will be triggered for processing. In this embodiment, a comparison is made between the warning rules and the texts to be processed in the text set to be processed to determine whether there is a target warning rule that matches the text set to be processed. The target warning rule can also correspond to the recipient of the warning information, so that during the warning processing, the warning information can be output to the recipient set by the target warning rule.
[0086] For the above data processing method, a text set to be processed at the current moment is obtained. The text set to be processed includes at least one text to be processed, and the text to be processed is a text for monitoring service quality from the aspect of non-functional service quality; according to the data volume of the texts to be processed in the text set to be processed, a processing node corresponding to the text to be processed is determined from the node cluster; sensitive word analysis is performed on each text to be processed through the determined processing node to obtain an analysis result indicating service quality from the aspect of non-functional service quality; if the analysis result indicates service anomaly, warning processing is performed, so as to monitor service quality from the aspect of non-functional service quality, thereby adding monitoring from the aspect of non-functional service quality on the basis of functional service quality and improving the monitoring accuracy. And during the monitoring from the aspect of non-functional service quality, according to the data volume of the texts to be processed in the text set to be processed, a processing node corresponding to the text to be processed can be determined from the node cluster, so that the processing node for processing the text to be processed matches the data volume of the texts to be processed in the text set to be processed, realizing the automatic scaling of the processing nodes in the node cluster and effectively controlling the monitoring cost while ensuring service stability.
[0087] Corresponding data processing method Figure 2 The system architecture diagram of the processing system implementing the data processing method is shown, including a data access module, a metadata module, a processing module, and an alarm notification module.
[0088] The data access module is used to collect texts to be processed, and the data access module effectively buffers high-concurrency data (i.e., a large number of texts to be processed) by using big data technology. The metadata module manages and stores a preset sensitive word library, such as managing and storing sensitive words and warning rules, which is convenient for dynamic maintenance and continuous iterative optimization. The processing module is used to execute the above data processing method to obtain an analysis result, and when the analysis result indicates service anomaly, it triggers the alarm notification model to output warning information.
[0089] The following example uses the UGC of user experience obtained from comments and bullet comments to explain that the UGC representing the user experience is stored in the Kafka queue to provide a unified data acquisition channel in the sensitive word analysis process. The Kafka queue has the characteristics of high throughput and can also provide effective buffering for high concurrency problems.
[0090] After the UGC representing the user's usage experience is stored in the Kafka queue, it can be processed by a device that executes the above data processing method ( Figure 3 The consumer in the text collection to be processed is then used to determine the processing node (such as Figure 3 The processing node then performs sensitive word analysis to obtain the analysis result. If the analysis result indicates that the service is abnormal and the target warning rule is found from the preset sensitive word library, warning processing is performed.
[0091] Corresponding to the above method embodiment, the present application embodiment also provides a data processing device, whose structure is as follows: Figure 4 As shown, it may include: an acquisition unit 10, a determination unit 20, an analysis unit 30 and an early warning unit 40.
[0092] The acquisition unit 10 is used to acquire a set of texts to be processed at the current moment, wherein the set of texts to be processed includes at least one text to be processed, and the text to be processed is a text used for monitoring service quality from the aspect of non-functional service quality.
[0093] Non-functional service quality aspects correspond to functional service quality aspects. Different from monitoring functional service quality aspects from data reflecting changes in the device itself, non-functional service quality aspects can be understood as user experience aspects. In the process of services provided by various devices, users will comment on the services, etc. In this process, texts for monitoring service quality from non-functional service quality aspects (or user experience aspects) will be generated, such as generating UGC for services. These UGCs are users' usage experience of the services, etc. Therefore, UGC that indicates users' usage experience of the services can be used as text to be processed.
[0094] The text to be processed can be obtained from, but not limited to, the comment area and the bullet screen area. These comment areas and the bullet screen area can be located in the service interface that provides the service, so that users can directly evaluate the service experience in the comment area or the bullet screen area of the service interface, eliminating the need for users to find a complaint entry from the service interface, and realizing online complaints about the service experience. The acquisition unit 20 can also obtain the text to be processed in other ways, which will not be described one by one in this embodiment.
[0095] The determination unit 20 is used to determine the processing nodes corresponding to the text to be processed from the node cluster according to the data volume of the text to be processed in the text set to be processed, and to allocate the processing nodes matching the current data volume with reference to the data volume of the text to be processed in the current text set to be processed, so that the current processing capacity of the node cluster matches the current data volume to be processed, so that the node cluster can automatically scale its processing capacity with reference to the current data volume to be processed.
[0096] Among them, an optional structure of the determination unit 20 is that the determination unit includes: a first determination subunit, a selection subunit and an addition subunit.
[0097] The first determination subunit is used to determine whether the processing node currently in working state in the node cluster can process all the texts to be processed according to the data volume of the texts to be processed in the text set to be processed. Whether the processing node currently in working state can process all the texts to be processed depends on the processing capacity of the processing node currently in working state. If the processable data volume indicated by the processing capacity is greater than the data volume of the texts to be processed in the text set to be processed, it is determined that the processing node currently in working state can process all the texts to be processed, otherwise it is determined that the processing node currently in working state cannot process all the texts to be processed.
[0098] The processing capacity of the processing node currently in working state is determined by the currently available resources of the processing node, such as the current CPU availability (ie, remaining usage), memory status, etc., which will not be described in detail in this embodiment.
[0099] The selection subunit is used to determine the processing node currently in working state as the processing node corresponding to the text to be processed if the processing node currently in working state can process all the text to be processed.
[0100] A sub-unit is added, which is used to add at least one working processing node in the node cluster if the processing node currently in working state cannot process all the texts to be processed, based on the relationship between the data volume of the texts to be processed in the text set to be processed and the processing capacity of the processing node currently in working state, so as to allocate a processing node that matches the data volume of the texts to be processed in the text set to be processed.
[0101] The relationship between the data volume of the text to be processed in the text to be processed set and the processing capacity of the processing node currently in working state is used to characterize the additional processing capacity required to process all the text to be processed in the text to be processed set. For example, the relationship between the data volume and the processing capacity is used to indicate the number of processing nodes added in the node cluster and the number of text to be processed that the added processing nodes need to process, so that additional processing nodes in working state are added in the node cluster based on this information so as to be able to process all the text to be processed in the current text to be processed set.
[0102] In this embodiment, the method of adding at least one processing node in working state includes but is not limited to the following methods:
[0103] If there are processing nodes in a non-working state in the node cluster, and the processing nodes in a non-working state and the processing nodes in a working state can process all the texts to be processed, the processing nodes in a non-working state in the control node cluster are switched to a working state. In other words, if the existing processing nodes in the node cluster can process all the texts to be processed, the processing nodes in a non-working state in the control node cluster are started and switched to a working state to process the texts to be processed.
[0104] If there are processing nodes in a non-working state in the node cluster, but the processing nodes in a non-working state and the processing nodes in a working state cannot process all the texts to be processed, then the processing nodes in a non-working state in the node cluster are controlled to switch to a working state, and processing nodes for processing the texts to be processed are deployed in the node cluster and controlled to enter a working state.
[0105] The processing nodes in the non-working state and the processing nodes in the working state cannot process all the texts to be processed, which means that the existing processing nodes in the node cluster cannot process all the texts to be processed, and there are processing nodes that have not been started. In this case, on the one hand, the processing nodes in the non-working state in the node cluster are controlled to start up and switch to the working state to process the texts to be processed. On the other hand, new processing nodes need to be deployed in the node cluster and the controller needs to enter the working state so that the processing nodes in the current node cluster can process all the texts to be processed in the set of texts to be processed.
[0106] If there is no processing node in a non-working state in the node cluster, a processing node for processing the to-be-processed text is deployed in the node cluster and controlled to enter a working state.
[0107] The analysis unit 30 is used to perform sensitive word analysis on each text to be processed through a determined processing node to obtain an analysis result indicating the service quality from the perspective of non-functional service quality. For example, the analysis unit 30 performs word segmentation processing on any text to be processed to obtain each word in the text to be processed; calculates the word similarity between each word and each sensitive word in a preset sensitive word library, and calculates the word similarity between each word and the derivatives of each sensitive word in the preset sensitive word library; based on the word similarity of all texts to be processed, obtains an analysis result indicating the service quality from the perspective of non-functional service quality. For the specific process and the process of obtaining the preset sensitive word library, please refer to the above embodiment, which will not be elaborated in this embodiment.
[0108] The early warning unit 40 is used to perform early warning processing if the analysis result indicates that the service is abnormal, such as outputting an alarm message to the monitoring terminal, and the operator at the monitoring terminal processes the alarm message based on the alarm message, for example, the alarm message can be outputted by at least one of but not limited to EMAIL and SMS notification.
[0109] In this embodiment, if the analysis result indicates that the service is abnormal, it can also be combined with the warning rules in the preset sensitive word library to determine whether to perform warning processing. The feasible way is: if the analysis result indicates that the service is abnormal and the target warning rule matching the text set to be processed is found from the warning rules in the preset sensitive word library, then the warning processing is performed. Please refer to the above method embodiment for the description of the warning rules, which will not be elaborated here.
[0110] The above data processing device obtains the current set of to-be-processed texts, the set of to-be-processed texts includes at least one to-be-processed text, and the to-be-processed text is a text used to monitor the service quality from the non-functional service quality aspect; according to the data volume of the to-be-processed texts in the to-be-processed text set, the processing node corresponding to the to-be-processed text is determined from the node cluster; sensitive word analysis is performed on each to-be-processed text through the determined processing node to obtain an analysis result indicating the service quality from the non-functional service quality aspect; if the analysis result indicates that the service is abnormal, an early warning process is performed to monitor the service quality from the non-functional service quality aspect, so as to increase the monitoring of the non-functional service quality aspect on the basis of the functional service quality aspect, and improve the monitoring accuracy. And in the process of monitoring from the non-functional service quality aspect, according to the data volume of the to-be-processed texts in the to-be-processed text set, the processing node corresponding to the to-be-processed text can be determined from the node cluster, so that the processing node used to process the to-be-processed text matches the data volume of the to-be-processed texts in the to-be-processed text set, and the automatic scaling capacity of the processing nodes in the node cluster is realized, and the monitoring cost is effectively controlled under the condition of ensuring the stability of the service.
[0111] This embodiment also provides a server, which includes a processor and a memory. The memory stores computer program codes, and the processor runs the computer program codes to implement the above data processing method.
[0112] This embodiment also provides a storage medium, in which program codes are stored, and the above-mentioned data processing method is implemented when the program codes are executed.
[0113] It should be noted that the various embodiments in this specification can be described in a progressive manner, and the features recorded in the various embodiments in this specification can be replaced or combined with each other. Each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0114] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0115] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
[0116] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A data processing method, characterized in that: The method comprises: Obtaining a set of texts to be processed at the current moment, wherein the set of texts to be processed includes at least one text to be processed, and the text to be processed is a text used to monitor the service quality from the perspective of non-functional service quality; wherein the text to be processed is the user's experience of using the service inputted in the comment area or the bullet screen area of the service interface providing the service; According to the data volume of the text to be processed in the text set to be processed, a processing node corresponding to the text to be processed is determined from a node cluster; all processing nodes in the node cluster include a cluster of a master processing node and a slave processing node, the master processing node receives the text set to be processed, and controls the slave processing nodes in the node cluster that are in a working state according to the text set to be processed and configuration data in the slave processing nodes, the configuration data is used to indicate the processing capacity of the slave processing nodes, and the master processing node controls the state of the slave processing nodes according to the configuration data to determine an adapted slave processing node from all the slave processing nodes; Performing sensitive word analysis on each of the to-be-processed texts through a determined processing node to obtain an analysis result indicating the service quality from the aspect of the non-functional service quality; If the analysis result indicates that the service is abnormal, early warning processing is performed to monitor the service quality from the non-functional service quality aspect, so as to increase the monitoring of non-functional service quality on the basis of functional service quality; wherein, the functional service quality aspect is to monitor the service quality through data reflecting the changes of each device itself during the process of providing services.
2. The method according to claim 1, characterized in that The step of determining, from a node cluster, a processing node corresponding to the text to be processed according to the data amount of the text to be processed in the text set to be processed comprises: Determining, according to the data amount of the text to be processed in the text set to be processed, whether the processing node currently in working state in the node cluster can process all the text to be processed; If the processing node currently in working state can process all the texts to be processed, the processing node currently in working state is determined as the processing node corresponding to the texts to be processed; If the processing node currently in working state cannot process all the texts to be processed, at least one processing node in working state is added to the node cluster according to the relationship between the data volume of the texts to be processed in the text set to be processed and the processing capacity of the processing node currently in working state, so as to allocate a processing node that matches the data volume of the texts to be processed in the text set to be processed.
3. The method according to claim 2, characterized in that The adding at least one processing node in working state in the node cluster includes: If there is a processing node in a non-working state in the node cluster, and the processing node in a non-working state and the processing node in a working state can process all the texts to be processed, then control the processing node in a non-working state in the node cluster to switch to a working state; If there are processing nodes in a non-working state in the node cluster, but the processing nodes in a non-working state and the processing nodes in a working state cannot process all the texts to be processed, then the processing nodes in a non-working state in the node cluster are controlled to switch to a working state, and processing nodes for processing the texts to be processed are deployed in the node cluster and controlled to enter a working state; If there is no processing node in a non-working state in the node cluster, a processing node for processing the to-be-processed text is deployed in the node cluster and controlled to enter a working state.
4. The method according to claim 1, characterized in that: The performing of sensitive word analysis on each of the to-be-processed texts by the determined processing node to obtain an analysis result indicating the service quality from the aspect of the non-functional service quality includes: For any of the texts to be processed, perform word segmentation on the text to be processed to obtain each word in the text to be processed; Calculating the word similarity between each word and each sensitive word in the preset sensitive word library, and calculating the word similarity between each word and the derivative of each sensitive word in the preset sensitive word library; Based on the word similarities of all the texts to be processed, an analysis result indicating the service quality from the aspect of the non-functional service quality is obtained.
5. The method according to claim 4, characterized in that The analysis result indicating the service quality from the aspect of the non-functional service quality based on the word similarity of all the texts to be processed comprises: For any of the texts to be processed, based on the word similarity of the text to be processed and the sensitivity level of the sensitive words corresponding to the word similarity, a score of the text to be processed for the service quality is obtained; Based on the business type of the text to be processed, the text to be processed in the set of text to be processed is divided to obtain a text subset to which the text to be processed corresponding to the same business type belongs; Based on the scores of all the to-be-processed texts in the text subset for the service quality, the score of the text subset for the service quality is obtained, and the score of the text subset for the service quality is used as the analysis result of the text subset.
6. The method according to claim 4, characterized in that If the analysis result indicates that the service is abnormal, performing early warning processing includes: If the analysis result indicates a service anomaly, and a target warning rule matching the to-be-processed text set is found from the warning rules of the preset sensitive word library, warning processing is performed.
7. A data processing device, characterized in that: The device comprises: An acquisition unit is used to acquire a set of texts to be processed at the current moment, wherein the set of texts to be processed includes at least one text to be processed, and the text to be processed is a text used to monitor the service quality from the perspective of non-functional service quality; wherein the text to be processed is the user experience of the service inputted in the comment area or the bullet screen area of the service interface providing the service; A determination unit is used to determine, from a node cluster, a processing node corresponding to the text to be processed according to the data volume of the text to be processed in the text set to be processed; all processing nodes in the node cluster include a cluster of a master processing node and slave processing nodes, the master processing node receives the text set to be processed, and controls the slave processing nodes in the node cluster that are in a working state according to the text set to be processed and configuration data in the slave processing nodes, the configuration data being used to indicate the processing capacity of the slave processing nodes, and the master processing node controls the state of the slave processing nodes according to the configuration data, so as to determine an adapted slave processing node from all the slave processing nodes; An analysis unit, configured to perform sensitive word analysis on each of the to-be-processed texts through a determined processing node, so as to obtain an analysis result indicating the service quality from the aspect of the non-functional service quality; An early warning unit is used to perform early warning processing if the analysis result indicates that the service is abnormal, so as to monitor the service quality from the non-functional service quality aspect, so as to increase the monitoring of the non-functional service quality aspect on the basis of the functional service quality aspect; wherein, the functional service quality aspect is to monitor the service quality through the data reflecting the changes of the equipment itself during the service provision process of each device.
8. The device according to claim 7, characterized in that The determining unit comprises: A first determining subunit is used to determine whether the processing node currently in working state in the node cluster can process all the texts to be processed according to the data amount of the texts to be processed in the text set to be processed; A selection subunit is used to determine the processing node currently in working state as the processing node corresponding to the text to be processed if the processing node currently in working state can process all the text to be processed; A sub-unit is added, which is used to add at least one working processing node in the node cluster if the processing node currently in working state cannot process all the texts to be processed, according to the relationship between the data amount of the texts to be processed in the text set to be processed and the processing capacity of the processing node currently in working state, so as to allocate a processing node that matches the data amount of the texts to be processed in the text set to be processed.
9. The device according to claim 8, characterized in that The additional subunit is used to control the processing nodes in the node cluster that are in a non-working state to switch to a working state if there are processing nodes in the node cluster that are in a non-working state and the processing nodes in a working state can process all the texts to be processed; If there are processing nodes in a non-working state in the node cluster, but the processing nodes in a non-working state and the processing nodes in a working state cannot process all the texts to be processed, then the processing nodes in a non-working state in the node cluster are controlled to switch to a working state, and processing nodes for processing the texts to be processed are deployed in the node cluster and controlled to enter a working state; If there is no processing node in a non-working state in the node cluster, a processing node for processing the to-be-processed text is deployed in the node cluster and controlled to enter a working state.
10. The device according to claim 7, characterized in that The analysis unit is used to perform word segmentation processing on any of the texts to be processed to obtain each word in the text to be processed; calculate the word similarity between each word and each sensitive word in a preset sensitive word library, and calculate the word similarity between each word and a derivative of each sensitive word in the preset sensitive word library; Based on the word similarities of all the texts to be processed, an analysis result indicating the service quality from the aspect of the non-functional service quality is obtained.
