Telemetry data

Abstract

Describes an apparatus and method, including: classifying service ticket data related to a service request into service topics, where the service ticket data is obtained from a service request related to a device; for the service request, determining the degree of match between the service topic and a telemetry data category, where the telemetry data category is related to activities at the device; and providing an output based on the determination.

Description

Background Art

[0001] An organization can use a service ticket to track problems with a device. For example, a ticket can be created when a problem is reported. Actions can be taken to resolve the problem. There is still a need for further improvement in this area. Brief Description of the Drawings

[0002] Examples will now be described by way of example only with reference to the following schematic drawings, in which:

[0003] Figures 1 to 3 is a block diagram of a system according to an example;

[0004] Figure 4 and 5 is a flowchart according to an example;

[0005] Figure 6 is a block diagram of an output format according to an example;

[0006] Figures 7 to 10 is a flowchart according to an example; and

[0007] Figure 11 is a block diagram of a system according to an example;

[0008] Figure 12 is a block diagram of a neural network according to an example. Detailed Description of the Embodiments

[0009] In the specification and drawings, like reference numerals refer to like elements throughout.

[0010] Figure 1 is a block diagram of a system according to an example, which system is generally indicated by reference numeral 10. System 10 includes a first module 12, which first module 12 can be configured to perform operations as described in further detail below. The first module 12 can receive telemetry data 11 and service ticket data 13. The telemetry data 11 can include information related to an activity (or activities) performed at a device. The telemetry data 11 can be received from the device (e.g., a printer), and can indicate the activities that occurred at the device during a given time period. The service ticket data 13 can be related to a service request involving the device. The first module 12 can use the telemetry data 11 and the service ticket data 13 to provide an output. In one example, the telemetry data 11 can be related to activities performed at multiple devices, and can be received from the respective multiple devices. The service ticket data can also be related to multiple service requests.

[0011] By way of example, activities performed at the device (e.g., a printer, scanner, and / or similar device) can be performed in response to problems reported via a service request. However, if the activities performed are not in response to any problem, or the activities are not an appropriate response to any problem reported via a service request, there may be an indication of suspicious activity (e.g., activities performed by malware, or someone performing activities that need not be performed or are not recommended).

[0012] Thus, the output provided by the first module 12 can be an indication of whether the activity is suspicious. For example, the output can include information about the likelihood that the activity was performed in response to a service request. The output can include, for example, an alert level.

[0013] Figure 2 is a block diagram of a system, which is generally indicated by reference numeral 20. System 20 is an example implementation of the first module 12. As Figure 2 shown, system 20 includes a first classification module 21 and an output module 22. System 20 can receive service ticket data (e.g., the service ticket data 13 discussed above) as an input, which can be received at the first classification module 21. The classification of the service ticket data (e.g., service topic) can be provided as an input to the output module 22. System 20 can further receive telemetry data categories (e.g., based on the telemetry data 11 discussed above) as inputs, which can be obtained at the output module 22.

[0014] Figure 3 is a block diagram of a system according to an example, which is generally indicated by reference numeral 30. System 30, as an example implementation of the first module 12, includes the first classification module 21 and the output module 22 as Figure 2 described. System 30 further includes a telemetry data classification module 31 such that telemetry data categories can be generated at the telemetry data classification module 31 based on input telemetry data (as Figure 5 discussed in detail). Telemetry data (such as telemetry data 11) can be provided as an input to system 30 such that the telemetry data classification module 31 can generate telemetry data categories, and the generated telemetry data categories can be provided as an input to the output module 22. The output module 22 can provide an output based on the service topic and the telemetry data categories.

[0015] Modules such as the first module 12, the first module 20, the first classification module 21, the output module 22, and / or the telemetry data classification module 31 can be implemented in software, hardware, application logic, or a combination of software, hardware, and application logic.

[0016] In one example, telemetry data can be generated at the device where an activity or event occurs (such as an instrument of a system or subsystem), such that the telemetry data can relate to various device components and actions. For example, the telemetry data can relate to security notifications and alerts, device configurations, changes in device parameters and / or settings, software installations, software reinstallations, software uninstallations, firmware upgrades, firmware downgrades, physical component defects or replacements, and / or interactions with the device (e.g., human interactions) (e.g., print jobs, scan jobs, user authentication, logins, logouts, etc.). The telemetry data can also include metadata such as timestamps, device identifiers (e.g., serial numbers, IP addresses, or hostnames), and / or usernames of the users performing the activity.

[0017] Figure 4 is a flowchart according to an example, which is generally indicated by reference numeral 40. To better understand these examples, it can be viewed in conjunction with Figures 1 to 3 to watch Figure 4 .

[0018] At operation 41, a telemetry data category can be obtained such that the telemetry data category is provided as an input to an output module 22. Alternatively or additionally, the telemetry data category can be generated by a telemetry data classification module 31 of the system 30. In one example, multiple telemetry data categories can be obtained or generated.

[0019] At operation 42, service ticket data related to a service request can be received such that the service ticket data is provided as an input to a first classification module 21. For example, when a problem with the device is reported, a service request can be created for each problem or for multiple problems (e.g., related problems that can be grouped into one service request). The service ticket data can include information about the service request.

[0020] At operation 43, service ticket data related to a service request can be classified into service topics at a first classification module 21. For example, service topics can include information about the problem identified in the service request, contextual information (e.g., the context in which the problem occurred), and / or metadata (e.g., timestamp, geographical information, environmental information, organizational enrichment information, etc.). In one example, natural language processing can be used to perform the classification. As described above, an organization can use a ticketing system to track issues (such as information technology (IT) issues), and the ticketing system can allow for reporting, prioritization, scheduling, assignment, and tracking of issues. For example, the ticketing system can allow a user to enter a service request such that an administrator can be informed of the problem. The content of the service request can include, for example, text written by the user who entered the service request. It is possible that different users may describe the same problem in different ways using words. Thus, at operation 43, the service topic for the service ticket data can be determined by processing the text in the service request. For example, natural language processing can be used to determine the service topic to which the service request may belong. For example, a first service request can include the text " The printer cannot be connected to the network ", and a second service request can include the text " Printer Cannot be used wirelessly ". Even though the first service request and the second service request differ in wording from each other, the first service request and the second service request may be related to the same problem. Therefore, the first classification module 21 can use natural language processing to classify the first service request and the second service request into the same service topic.

[0021] At operation 44, the output module 22 can determine, for each service request, the degree to which the service topic matches the telemetry data category. For example, for each service request, an individual can attempt to fix the problem identified in the service request by performing an activity at the device. The activities performed can be logged as telemetry data. As described above, the telemetry data can be classified into telemetry data categories. For example, if the activity indicated by the telemetry data category matches the expected activity performed in response to the service request, then the service topic can match the telemetry data category to a high degree. For example, if the service topic is related to printer ink not being applied to the substrate in the correct color or concentration, then the expected activities can include replacing the ink cartridge, changing settings related to the ink cartridge, checking if the ink cartridge is properly installed, etc. If the telemetry data category indicates an activity related to settings not related to the ink cartridge (e.g., changing network settings, security settings, passwords, etc.), then the degree to which the service topic matches the telemetry data category may be low. The activity indicated by the telemetry data category can include activities performed within a threshold time period after the service request is entered. Alternatively, the telemetry data category can indicate activities performed at a time period where there are no outstanding service requests, such that the degree to which any one of a plurality of service topics matches any one of the telemetry data categories can be determined to be low.

[0022] At operation 45, the output module 22 provides an output based on the determination at operation 44. For example, the output can provide an indication of the degree to which the service topic matches the telemetry data category. For example, if the activity includes changing the security settings or password of the device and there is no service topic that would likely require such an activity to be performed, then the output can provide an alert to the user that a suspicious or malicious activity may have been performed at the device. (It should be noted that such identified activity may not necessarily be malicious. For example, such a change may be accidental or made in error).

[0023] In one example, the service ticket data considered at operation 42 can relate to service requests logged within a first threshold time period before the activity of the telemetry data is performed. For example, if an activity related to a firmware upgrade is performed within the threshold time period (e.g., one week) of the service request logged for the firmware upgrade, then the degree to which the service topic matches the telemetry data category may be high and the firmware upgrade activity is likely not malicious. However, if the firmware upgrade is performed outside of the first threshold time period (e.g., before the service request is entered, or one year after the service request is entered), then the degree to which the service topic matches the telemetry data category may be low and the firmware upgrade activity is likely to be suspicious or malicious (or, as noted above, may be accidental or made in error).

[0024] In one example, the service ticket data received at operation 42 includes metadata. For example, the metadata can include information such as the device identifier of the device for which the service request was entered, the identity of the user who entered the service request, the timestamp of the service request being entered or resolved, and so on. Telemetry data can also include metadata such that the metadata can indicate a timestamp of when the activity was performed, the device identifier of the device on which the activity was performed, the user identifier of the user or administrator who performed the activity, and so on. In one example, the determination at operation 44 of the degree to which the service topic matches the telemetry data categories can be at least partially based on the metadata. For example, the metadata can provide timestamps corresponding to the service request and the telemetry data such that it can be determined whether an activity performed at a first time period (as indicated by the metadata of the telemetry data) was expected to be performed in response to a service request during or before that first time period. In another example, there can be authorized individuals who can be authorized to perform certain activities, such as a firmware upgrade for the device. If it is determined from the metadata that an activity related to a firmware upgrade was performed on a device used by an unauthorized individual, the degree of match between the service request and the telemetry data categories can be low.

[0025] In one example, at operation 41, one or more telemetry data categories can be generated or obtained for activities at one or more devices. At operation 42, the received service ticket data can relate to one or more service requests related to at least some of the one or more devices. At operation 43, the service ticket data related to each service request can be classified into service topics. At operation 44, the determined degree can include the degree to which the service topic matches any of the one or more telemetry data categories.

[0026] Figure 5 is a flowchart according to an example, which is generally indicated by reference numeral 50. As described above, the first module can generate or obtain telemetry data categories related to activities at the device. For example, at operation 51, the output module 22 of the telemetry data classification module 31 can receive telemetry data related to an activity at the device or another device. At operation 52, the telemetry data can be classified (e.g., by the classification module 31) into one or more telemetry data categories. For example, when a problem is resolved by performing an activity in response to a service request, the activity can be logged as telemetry data and a description of the activity can also be logged. For example, the description can include text such as " The error was cleared by applying a firmware upgrade Error ". Then, the telemetry data related to the service request can be classified into a telemetry data category such as "firmware upgrade".

[0027] Figure 6is a block diagram according to the output format of the example, which is typically indicated by reference numeral 90. The output 91 (e.g., reference Figure 4 is generated at operation 45, or reference Figure 2 or 3 is generated at the output module 22) may include an alert message 92 and / or an alert level 93. In cases where the telemetry data category (e.g., activity) does not match the service topic, or where the telemetry data category matches the service topic to a low degree, the output may include an alert message 92 to notify the administrator that a suspicious or malicious activity may have been performed at the device. Alternatively or additionally, the output may include an alert level 93. For example, if the degree of match between the telemetry data category and the service topic is high, the alert level may be low to indicate that there is no or a low likelihood that the activity is malicious. Alternatively, if the degree of match between the telemetry data category and the service topic is low, the alert level may be high to indicate a high likelihood that the activity is suspicious or malicious.

[0028] In one example, a first score may be determined such that the first score indicates the degree of match between the telemetry data category and the service topic. The first score may be based on the match between the telemetry data category and the service topic, at least in part based on natural language processing techniques and / or machine learning techniques. Thus, the alert level 93 may be based on the first score. In one example, if the first score and / or the alert level 93 is higher than a threshold, the output includes the alert message 92 such that the alert message 92 is sent to the administrator as a notification that a malicious activity may have been performed at the device.

[0029] In one example, if the degree of match between the service topic and the telemetry data category is lower than a threshold, it may be determined (e.g., indicated by a score such as a binary score) that the service topic does not match the telemetry data category and that the activity being performed may be suspicious or malicious. Similarly, if the degree of match between the service topic and the telemetry data category is higher than a threshold, it may be determined (e.g., indicated by a score such as a binary score) that the service topic matches the telemetry data category and that the activity being performed is not suspicious or malicious.

[0030] The first score may be generated in a variety of ways. Example arrangements include providing a high score in cases where the action does not match the service topic, but alternative arrangements are possible.

[0031] For example, assume that the telemetry category and the service category are equal. Natural language processing can provide a "likelihood" within the range [0, 1], which represents the degree of certainty that a service ticket belongs to a given category. Given n categories and a service ticket S, a classifier can give numbers s_i, where 0 ≤ i ≤ n and s_i is within [0, 1]. For a given S, we can choose N = argmax(s_i) as the most likely category and choose s_N as the corresponding score. Each category i can also have a priority score (e.g., c_i within [0, 1]). As an example, if i indicates a password change, then c_i may be high (close to 1), but if i indicates a printer cartridge change, then c_i may be low (close to 0), such that a password change may have a higher priority than a cartridge change. The priority may depend on the customer and / or based on a risk posture, etc. A sequence of telemetry data can be classified in a similar manner. For example, T can represent a sequence of telemetry items. T can also be assigned to one of the n categories, and a score t_i (0 ≤ i ≤ n) can be assigned as the likelihood that the sequence T belongs to category i. Here, the score can be a binary score (e.g., predetermined by an expert). This means that T can be determined with absolute certainty to belong to a certain category. There can be multiple Ts within a window. For the pair (S, T), the score (i.e., the "first score" mentioned above) can be simply created as (1 - s_N) * c_N * t_N. Then, it can be compared with a threshold. If the condition that the telemetry category and the service category are equal is relaxed, a similar process and result can be achieved by mapping the telemetry category to a predefined service category with a different score. As an example, this can become a multiplier in the subsequent score.

[0032] In one example, when classifying telemetry data into categories of action types, there can be scores associated with each of these categories. For example, messages classified as associated with clearing a paper jam or changing toner can be considered low risk (e.g., the low priority described above), while user management or security setting changes can be considered high risk (e.g., the high priority described above), and thus the scores will be used to represent risk knowledge about the category. The scores provided may be customer specific: for example, if a customer is concerned about toner theft, this may be designated as high risk. Alternatively or additionally, a matrix of service topics versus telemetry data categories (i.e., what the action indicates is happening) can be provided. For example, if there are some known operations that are frequently performed when fixing a particular service topic, score modifiers can be encoded even if these operations may not be relevant (possibly due to an engineer needing to look for and locate a fault or try different options). In such a matrix, there can be a score such that when a particular service topic is defined from a trouble ticket, the expected (or acceptable) telemetry group has a low score. For a given service topic, the sum of the scores can be calculated for any telemetry category from the device over a given time period. Here, a higher total score indicates a more important alert. Thus, for example, when reporting a network issue, security setting changes may be downplayed (as they are expected to be), but when reporting an issue in terms of user management or printer functionality (e.g., a request for more toner), security setting changes are not downplayed. In this way, the scores can be modified within the context of the service topic.

[0033] As a further example, there can be cases where a trouble ticket has a reasonable classification level across multiple service topics. A matrix can be defined for a given service topic (i) and telemetry category (j) such that the matrix element Mi,j contains the probability that telemetry category j will be observed when service topic i occurs. For each telemetry topic j, we can look across all service topics (0,..,n) and take the minimum of tj*(1-(si*mij)). Then, the scores can be summed for each telemetry category. Effectively, the scores are calculated such that if there are several highly classified service topics, telemetry data matching any high score is classified into that service topic at least in part based on how well that service topic is classified.

[0034] Many alternative arrangements for generating the first score are possible. For example, additional weighting can be added to certain telemetry categories representing high risk operations.

[0035] Figure 7is a flowchart according to an example, which is generally indicated by reference numeral 100. At operation 101, an output module (e.g., output module 22) can determine whether a second threshold period before and / or after the execution of an activity is related to a maintenance period. At operation 102, if the second threshold period is related to the maintenance period, it can be determined that the degree to which the service topic matches the telemetry data category is likely to be high, and the activity is likely not malicious. For example, during the maintenance period, various activities can be performed even if no specific problem is specified in any service request. Thus, these activities may be related to the maintenance of the device and may not be executed maliciously.

[0036] Figure 8 is a flowchart according to an example, which is generally indicated by reference numeral 110. At operation 111, it is determined whether the activity indicated by the telemetry data matches the expected activity corresponding to the service topic. At operation 112, if the activity matches the expected activity, it can be determined that the degree to which the service topic matches the telemetry data category is likely to be high. For example, the expected activity corresponding to the service topic related to the ink cartridge can be an activity related to the ink cartridge. Thus, if the ink cartridge replacement activity is performed in response to the service topic related to the ink cartridge, it can be determined that the degree to which the service topic matches the telemetry data category is likely to be high.

[0037] Figure 9 is a flowchart according to an example, which is generally indicated by reference numeral 115. At operation 116, a natural language processor can be trained, for example, based on the received service ticket data as described in reference Figures 2 to 3 In one example, the service ticket data for training the natural language processor can include training service ticket data. At operation 117, the trained natural language processor can be used to classify the service ticket data into service topics. For example, the service ticket data can be used to train the natural language processor. However, the trained natural language processor can then be used to classify the service ticket data at operation 117. Thus, when the service ticket data is received, the operation 116 of training the natural language processor with the service ticket data can be performed before the classification at operation 117, or may not be performed before the classification at operation 117.

[0038] Figure 10is a flowchart according to an example, which is indicated by reference numeral 120. At operation 121, training service ticket data can be received, where the training service ticket data can be related to an example service request involving a device (or devices). At operation 122, the natural language processor can be trained using the training service ticket data and an associated service topic. At operation 123, service ticket data related to an operational service request can be received. The operational service request can relate to the device corresponding to the training service ticket data. At operation 124, the natural language processor can be used to classify the service ticket data related to the operational service request (e.g., new service ticket data) into service topics. Thus, the natural language processor can be pre-trained (e.g., statically trained) using the training service ticket data related to the example service request. The natural language processor can further be trained (e.g., dynamic update of training data, and dynamic training of the natural language processor) using the service ticket data related to the operational service request. The following reference Figure 11 is used to describe the training of the natural language processor in further detail. Operations 121 and 122 can be repeated multiple times before operations 123 and 124 are performed. Additionally, multiple instances of operations 121 and 122 can be implemented after multiple instances of operations 123 and 124 are performed (i.e., the training of the relevant natural language processor can be updated).

[0039] Figure 11 is a block diagram of a system according to an example, which is generally indicated by reference numeral 160. System 160 includes a telemetry data module 174, a service ticket module 175, and an output module 176. System 160 is an example implementation of the first module 12 and systems 20 and 30 described above with reference to Figures 1 to 3 For example, referring to Figure 3 , the first classification module 21 can be similar to the classification module 171 included in the service ticket module 175, the telemetry data classification module 31 can be similar to the classification module 173 included within the telemetry data module 174, and the output module 22 can be similar to the output module 176.

[0040] The elements of system 160 are described in detail below. It will be apparent to those skilled in the art that multiple variations of system 160 are possible. For example, multiple modules can be omitted, combined, or altered.

[0041] The service ticket module 175 may include a service ticket system 161, an interface 162, a service ticket database 163, a training set 164, a new ticket record 165, metadata 166, a tokenizing module 167, a training module 168, a natural language processing model 169, a service ticket category 170, and a classification module 171. In one example, the service ticket system 161 is used to receive a service request related to a device from a user (e.g., refer to Figure 4 to implement operation 42), such that the service ticket data can be analyzed. The service ticket system 161 may include a third-party ticketing system, and the service ticket data may be received at the database 163 using an application programming interface such as the interface 162. The service ticket data may be recorded in the new ticket record 165 and may also be used as part of the training set 164 for training a natural language processing model, such as a machine learning model (e.g., refer to Figure 9 and / or Figure 10 to implement operations 116 and / or 122).

[0042] For example, in the training route (indicated by the dashed arrow), the service ticket data may be used as training data. The tokenizing module 167 may parse the service ticket data and may help tag (one or more) parts of the service request. For example, the tagging may include voice tagging (e.g., for finding nouns, etc.) and name entity recognition that may identify certain types of words (such as locations). The training module 168 may use the service ticket data and / or the parsed service ticket data to train a machine learning model (e.g., the NLP model 169). For example, the training module 168 may extract the theme of the service request based on natural language processing techniques. The theme may be related to: the type of service request, the location of the device related to the service request, the type of activity required to resolve the problem specified in the service request, etc. The service ticket data may then be classified (e.g., refer to Figure 4 , 9 and / or 10 to implement operations 43, 117 and / or 124) into the category 170, where the category 170 may include service themes.

[0043] The classification indicated by category 170 can be used at the NLP model 169, which is used to classify service ticket data related to the new ticket record 165. For example, in the new data route (indicated by the dotted arrow), when a new service request is entered, the new service ticket data related to the new service request can be recorded at the new ticket record 165. The tokenization module 167 can parse the new service ticket data and can help label (one or more) parts of the new service request (e.g., voice tagging and / or name entity recognition). The new ticket record can also record the metadata 166 from the new service request. The NLP model 169 can be used to determine, for example, the topic related to the new service ticket data based on natural language processing techniques. The classification module 171 can be used to classify service requests into service topics (e.g., to implement operations 43, 117, or 124 with reference to Figure 4 , 9 and 10 respectively).

[0044] The telemetry data module 174 can include an event stream 172 and a classification module 173. The event stream 172 can be generated as a continuous time series, which can include information about the activities performed at the device. For example, telemetry data (similar to the telemetry data 11 described above with reference to Figure 1 ) can be logged via protocols such as system logs and other event types. Such telemetry data related to activities or events can be designed to be provided to an event management engine for further processing and analysis. The classification module 173 can aggregate, analyze the telemetry data received in the event stream and classify the telemetry data (e.g., to implement operation 52 described above with reference to Figure 5 ). For example, a single activity can correspond to multiple data points in the telemetry data, and the multiple data points can be aggregated to identify the single activity corresponding to the multiple data points. For example, a firmware upgrade can include various operations such as logging in, uploading a firmware file, entering an authentication code, entering preferences, agreeing to terms and conditions, confirming the upgrade, etc. These operations can be aggregated and classified into a telemetry data category named "firmware upgrade".

[0045] At output module 176, the telemetry data categories received from the telemetry data module 174 can be compared with the service topics received from the service ticket module 175. This comparison can be based on the nature of the activities indicated by the telemetry data categories, the nature of the service topics, and / or metadata associated with the telemetry data and / or service tickets. For example, activities from an event stream can be compared with a service topic to determine if the activity (e.g., an administrative action) is consistent with a service request (e.g., a trouble ticket). Metadata associated with the telemetry data and service requests (such as a timestamp) can be used to determine if the activity occurred within a threshold time period of the logged service request. Alternatively or additionally, metadata associated with the telemetry data and service requests (such as device identification) can be used to determine if the activity performed and the service request involve the same or different devices, or if the activity performed at a device is associated with a service request related to a device in the same or different office. Metadata including login data can also be used to identify the administrator of the device at which the activity was performed and can be compared with the metadata in the service request to indicate if the same administrator is working to resolve the problem specified in the service request. Thus, the telemetry data and service ticket data can be compared to identify correlations or inconsistencies in the activities performed at the device. Output module 176 can provide an output (e.g., with reference to Figure 4 to implement operation 45) based on the degree to which the service topic matches the telemetry data category. For example, if an activity cannot be explained by the presence of a related service request, the output can include an alert level or alert message indicating a high risk associated with the activity. The alert level or message can be provided to an administrator so that the activity can be investigated. For example, if the activity involves a configuration or security setting that was changed without any reason or when no configuration change was required, the output (e.g., an alert message or alert level) can indicate that such activity may be suspicious or malicious and that these activities should be investigated.

[0046] In one example, the telemetry data categories and / or service topics determined by classification modules 173 and 171, respectively, can be manually updated by an administrator. For example, when an administrator investigates an event, the administrator can investigate whether the classification of telemetry data into telemetry data categories, or the classification of service ticket data into service topics, is inaccurate. Such inaccurate classification may generate false alarms and can be corrected by the administrator. Such inaccuracies can also be logged as examples and can be used to further train the NLP model to provide better classification. The administrator can also create new telemetry data categories or service topics and classify the event or service ticket data accordingly. This provides an extended database for refining and retraining the NLP or telemetry data or service topic classification. Alternatively or additionally, classification module 171 can be updated by further training and / or updating the NLP model 169, and classification module 173 can be updated by training and / or updating a model (e.g., a machine learning model) for classifying telemetry data into telemetry data categories.

[0047] In one example, the natural language processor can include a machine learning model. Figure 12 A neural network, generally indicated by reference numeral 200, is shown that is used in some examples. For example, the natural language processor can include a machine learning model such as neural network 200. Neural network 200 can be trained using an input that includes service ticket data (e.g., related to an example service request and / or an operational service request). Neural network 200 includes an input layer 201, a hidden layer 202 (or multiple hidden layers 202), and an output layer 203. At the input layer 201, service ticket data related to multiple service requests, along with the associated service topic to which the service ticket data can be classified, can be received as input. The hidden layer 202 can include multiple hidden nodes where processing can be performed based on the received data. At the output layer 203, the classification of the service ticket data into the service topic can be determined.

[0048] As described above, for a given printer, for example, within a time window of a service ticket, a service topic can be generated and matched with an appropriately corresponding telemetry data category. Thus, a set of related events can be collected together (e.g., service tickets, telemetry events, telemetry categories, service topics, match scores) and then stored as an audit record (which can be referred to as an "evidential unit"). Storing such audit records can allow for retrospective analysis of the data to correlate the events linked to a trouble ticket and how they were interpreted.

[0049] The described examples can be implemented in software, hardware, application logic, or a combination of software, hardware, and application logic. The software, application logic, and / or hardware can reside in a memory or any computer medium. In one example, the application logic, software, or instruction set is maintained on any of a variety of computer-readable media. In the context of this document, "memory" or "computer-readable medium" can be any non-transitory medium or means capable of containing, storing, transmitting, propagating, or transporting instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer.

[0050] If desired, the different functions discussed herein can be performed in a different order and / or simultaneously with each other. Additionally, if desired, at least some of the functions described above can be optional or can be combined.

[0051] It will be appreciated that the examples described above are purely illustrative and do not limit the scope of the invention. After reading this specification, other variations and modifications will be apparent to those skilled in the art.

[0052] Furthermore, the disclosure of this application should be understood to include any novel feature or any novel combination of features or any generalization thereof that is explicitly or implicitly disclosed herein, and during the prosecution of this application or any application derived therefrom, new claims can be formulated to cover any such feature and / or combination of such features.

[0053] It should also be noted herein that although various examples have been described above, these descriptions should not be taken in a limiting sense. Rather, several variations and modifications can be made without departing from the scope of the invention as defined in the appended claims.

Claims

1. A method, comprising: Classifying service ticket data related to a service request into service topics, wherein the service ticket data is obtained from the service request related to a device; Determining, for the service request, the degree to which the service topic matches a telemetry data category, wherein the telemetry data category indicates a classification of activities generated at the device; And Providing an output based on the determination.

2. The method according to claim 1, further comprising: Receiving telemetry data related to activities generated at the device; And Classifying the telemetry data into the telemetry data category.

3. The method according to claim 1, wherein natural language processing is used to perform the classification of the service ticket data.

4. The method according to claim 3, further comprising: Training a natural language processor to perform the natural language processing.

5. The method according to claim 1, further comprising: Determining a first score, wherein the first score indicates the degree to which the service topic matches the telemetry data category.

6. The method according to claim 1, wherein the service ticket data includes metadata.

7. The method according to claim 6, wherein determining whether the service topic matches the telemetry data category is at least partially based on the metadata.

8. The method according to claim 1, wherein in a case where the degree to which the service topic matches the telemetry data category is lower than a threshold, the output includes an alert message.

9. The method according to claim 1, wherein the output includes an alert level.

10. The method according to claim 1, wherein the service ticket data is related to the service request such that the service request is recorded within a first threshold time period before the activity of the telemetry data is performed.

11. The method according to claim 1, further comprising: Determining whether a second threshold time period before and / or after the performance of the activity is related to a maintenance time period; And If the second threshold time period is related to the maintenance time period, determining that the activity matches the service topic.

12. The method according to claim 1, wherein determining whether the service topic matches the telemetry data category includes: Determining whether the activity matches an expected activity corresponding to the service topic.

13. An apparatus, comprising: A first classification module for classifying service ticket data related to a service request into service topics, wherein the service ticket data is obtained from the service request related to a device; And An output module for determining, for each service request, whether the service topic matches a telemetry data category and providing an output accordingly, wherein the telemetry data category indicates a classification of activities generated at the device.

14. A non - transitory machine - readable storage medium encoded with instructions executable by a processor, the machine - readable storage medium comprising instructions for: Classifying service ticket data related to a service request into service topics, wherein the service ticket data is obtained from the service request related to a device; Determining, for the service request, the degree to which the service topic matches a telemetry data category, wherein the telemetry data category indicates a classification of activities generated at the device; and Providing an output based on the determination.

Images