Computer-implemented method for arranging error reports in at least two error report groups of similar error situation

Abstract

The invention relates to a computer-implemented method (100) for arranging error reports (10) in at least two error report groups (30) of different error situation (31, 32, 33, 34), and to an associated computer program product (8), group-forming system (5) and system (200), comprising the group-forming system (5) and at least one test system (1).

Description

[0001] Title: Computer-implemented method for grouping error reports into at least two error report groups of similar error situation

[0002] Description

[0003] The invention relates to a computer-implemented method for grouping error reports into at least two error report groups of similar error situations, as well as to an associated computer program product, grouping system and system comprising the grouping system and at least one test system.

[0004] A test system is a system that issues a bug report. The test system can be represented by a computer program that is executed by a corresponding processor, in particular on a production system, a

[0005] development system, a security system, or a quality assurance system. The use of computer programs or software is essential for numerous technical products, as it has become an integral part of these products. This is the case, for example, with machines used in industry, particularly machine tools, which require one or more computer programs to perform their functions. For example, these may be machine tools that use laser technology for metalworking tasks and chip lithography, which often use particularly complex software applications.

[0006] Technical products, including their computer programs, may be faulty and issue corresponding error messages. Error messages may also be false alarms, i.e., without a product defect or without a real error in the use of the product, e.g., the machine tool, or the computer program. To ensure that the products are as error-free as possible, they are tested in advance.

[0007] Testing is the process of interacting with a product to verify that it meets requirements and is free of defects that could impair its functionality. The goal is to determine whether or not the product can be released. For small products, testing can be successfully performed manually. However, as the product grows in size and complexity, particularly with the aforementioned machine tools, the size of the system state space grows exponentially, making this task impossible to perform manually. In this case, it is advisable to perform automated product testing using a computer-based testing system.

[0008] For example, automated tests themselves are error-prone, and each failed test requires manual review to verify its result—i.e., to classify its cause as either a product defect or a false alarm. False alarms are due to testing and infrastructure issues, not to defects in the product itself. Developers must examine the error reports, which consist of error messages, stack traces, and log files, to perform this verification.

[0009] Automating test procedures is known from the prior art. US 2021 / 0287109 A1 describes a computer-implemented method for analyzing test errors using an AI model. The method first performs clustering, in which test errors are clustered into individual error clusters. The clustering result is then used to train the AI ​​model in order to automatically identify the cause of new, unclassified errors.

[0010] Although the methods known from the state of the art can be used for troubleshooting and identification, they are not as suitable for preventing product defects as would be desirable. For example, it has been found that many error messages, whether false alarms or product defects, are not analyzed in the known test procedures. This can lead to defects remaining in the product that only become apparent later at the customer's site and lead to complications there. Such defects often result in them having to be resolved through a new test phase, which is very time-consuming and expensive. If, for example, a machine tool with various product defects is delivered to a customer after a supposedly successful test phase, downtimes caused by product defects can cause major production losses at the customer.

[0011] The invention is therefore based on the object of increasing the reliability of the products, in particular of proposing a particularly robust computer-implemented method for grouping error reports in order to increase the reliability of the products.

[0012] The problem is solved by a computer-implemented method for grouping error reports into at least two error report groups of different error situations. The method comprises at least the following steps:

[0013] - Reading in error reports with at least two error report attachments from different domains,

[0014] - Determining a similarity operator for each of the different domains to determine a similarity between bug report attachments of the same domain,

[0015] - Determining similarity values ​​of the error report attachments of the same domain by comparing individual error report attachments of the same domain of different error reports using the previously determined similarity operator, and grouping the error reports into the at least two

[0016] Bug report groups of similar bug situations based on the previously determined similarity values.

[0017] The method according to the invention is therefore based on the error reports being processed separately in their individual components, i.e. with their error report attachments. To make this possible, different similarity operators are determined and used for the different domains in which the error report attachments are present. The method according to the invention can thus access the maximum amount of error-relevant information in the error reports in order to enable the error reports to be assigned to different error situations by comparing the similarity and grouping. Finally, it was recognized that a comparison of the error reports as a whole does not provide the desired high level of robustness because the data cannot be compared with one another with sufficient accuracy, as is the case according to the invention with the different similarity operators.Furthermore, when comparing individual bug report attachments of the same format, important bug information may be overlooked, making robust grouping impossible. By using the similarity values ​​of bug report attachments for grouping, as much bug information as possible can be used for robust bug report grouping.

[0018] The different error report attachments can be differentiated from one another based on their domain. The domain defines, in particular, the context and / or the format of the information they contain. For example, the same formats, in particular file formats, can exist and yet different domains can exist. For example, with an image format in context view, different images can be present as error report attachments, for example, a screenshot of a test sequence on the one hand and a photo of a machined workpiece on the other. Because of the different context, different domains exist here, even if the file format can be the same.On the other hand, with the same context, for example a screen capture of a test run at the time an error was output on the one hand and an error text at the time the error was output on the other, different formats, in particular file formats, can exist, so that different domains can also exist. In order to determine the similarity of error report attachments of different formats in the same context, it is possible to convert the error report attachments into a common format. To do this, one error report attachment can be converted into the format of the other error report attachment, or both error report attachments can be converted into a third format. For example, different image formats or video formats can be adapted.

[0019] The error reports can each be issued by a test system, although it is also possible for several test systems to be used. Error reports can indicate an error, which can in particular be expressed by an error message. The products tested are in particular machines, for example machine tools, or computer programs or software applications that are executed on the machines. A corresponding test sequence or a system test of the test system can in this respect be carried out by a computer, in particular a processor, of the respective machine. The test system can be represented by a computer program that can be executed by a corresponding processor, in particular on the product. The test system thus enables, for example, a test of the functionality of the product in terms of its functionality or.with regard to possible errors at least in relation to one or more computer programs that can be executed on the product. The test system also enables, for example, a service or a service check of the product. The test can be designed in particular as a UI (User Interface) test, in particular a GUI (Graphical User Interface) test, which can run on a corresponding system screen. The result of such an exemplary test sequence is the error report with its error report attachments of different domains, which can be understood as individual components of the error report. In particular, the error report attachments can be separate files of different domains, in particular different file formats. The method according to the invention can of course also comprise the step of testing ortest procedure itself, particularly in the form described above.

[0020] The similarity value is a value relating to a similarity between two bug report attachments, which may in particular correspond to the human intuitive concept of similarity. In particular, it is a real value which may, for example, be restricted to an interval such as the unit interval, i.e. [0, 1], or a Boolean value. The similarity value may, in particular, be symmetrical in order to ensure that the similarity between two bug report attachments is always the same, regardless of the order in which they are determined using the similarity operator. Given a data set of bug report attachments of the same format, the similarities between all bug report attachments can be calculated using the similarity operator specific to the format, which operator may, in particular, comprise or consist of one or more similarity functions, and can be represented, for example, in a similarity matrix.In particular, different similarity operators are determined and used for different formats to provide the most accurate similarity results possible and thus enable robust grouping of the bug reports. Examples of possible similarity operators will be explained in more detail later with reference to examples of possible domains.

[0021] Once the similarity values ​​of the error report attachments have been determined, it is possible to state for each individual error report attachment which error report attachments of error reports with which they have been compared they most closely resemble or are similar, in order to enable error report attachments with the same or similar error situation to be assigned or summarized. An error situation is understood to be at least one error or a summary of several errors that are the same or similar to one another. This relates in particular to the cause of the error, for example faulty programming in a specific part of the computer program executed by the test system, a specific category of hardware errors, etc. In order to group or, in other words, cluster the error reports, the similarity values ​​determined in each case are used, in particular by comparing the similarity values ​​with one another.Various grouping operations are possible here.

[0022] For example, it is possible to compare the similarity values ​​with a grouping logic in order to group the error reports into at least two error report groups. For example, to compare two error reports, the similarity values ​​of the error report attachments of the same domain can be compared with individual grouping thresholds of the grouping logic. This way, for each similarity value, the respectively defined grouping threshold can be used to determine whether there is sufficient similarity to assign the error report attachments to the same error situation. Alternatively or additionally, it is also possible to combine the individual similarity values ​​to form an overall similarity value of the grouping logic, followed by a comparison against an overall grouping threshold. The two error reports are only grouped into the same error report group if the overall grouping threshold is met by the overall similarity value.The grouping logic can also be designed as a maximum value or a maximum requirement, for example, such that the maximum of individual similarity values ​​or overall similarity values ​​of a bug report with its compared bug reports is used for grouping. This can ensure, if necessary, that each bug report is only grouped into one bug report group and that the sensitivity of a selected grouping threshold is not chosen to be too small or too large. In particular, an artificial intelligence (AI) model, particularly one based on machine learning, can be used here to continuously improve the grouping. The AI ​​model can use machine learning to select ever more optimized grouping logic, for example grouping thresholds, in particular different grouping thresholds for different error situations.

[0023] It can be provided that, when determining the similarity values, error report attachments from already grouped error reports are compared with imported, not yet grouped error report attachments using the previously determined similarity operator. In this way, an existing database of already grouped error reports can be used in order not to compare the error reports currently being read in with each other, or not only to compare them, but also to use and expand already identified error report groups with specific error situations. In this way, new error situations can also be quickly identified if an error report cannot be grouped into any previously created error report group, particularly taking into account the grouping limit value, so that a new error report group can be created.

[0024] It may further be provided that the error reports are multimodal, and that the different domains are different formats, in particular different media formats. Accordingly, in particular, not only text files are to be compared with each other to determine their similarity. This requires special similarity operators for the different media formats.

[0025] Metadata of the bug report attachment can also be considered as a possible domain , in particular the format of a bug report attachment .

[0026] In particular, the different media formats can be from the group comprising text file, in particular error message, log file and / or stack trace, image file, video file and / or audio file.

[0027] Specifically, the error message, as a subformat of the text file, is a very short, immutable, and unambiguous sentence that accompanies an error. When an error is thrown, this portion of text is also returned as a hint to the developer about the source of the error. In some cases, these messages are very informative about the cause of the error, e.g., "Access is denied" or "The parameter is incorrect," while in other cases, they are a general statement that can be caused by numerous events, e.g., "The timeout has expired and the operation did not complete." In this case, nothing is known about the operation, only that it did not complete.

[0028] The text file can also be assigned the Stack Trace subformat, which can contain very relevant information about error messages. This is because a program used for testing usually consists of functions that interact with each other through calls and requests. If an error occurs in one of these functions, the runtime environment throws an exception. In some cases, the program is prepared to catch these and continue running. However, exceptions that are not caught cause the program to crash. A single error and the associated error message are often not enough to determine the cause of the problem. In this case, the stack trace proves useful because it provides more details about the source of the uncaught error. It collects the stack of functions called up to the point at which the error was raised.This makes it possible to identify the problem in the code. Stack traces can be very revealing about a bug.

[0029] Another important subformat of the text file is the so-called log file. During a test run, a large number of events and operations take place. Although most of them occur routinely in the system, some can be used to track down the cause of a specific problem. Log files are machine-generated files that record all of these operations. They can also be described as containers with extensive and varied information that can be valuable for troubleshooting. Although there are no standards for their format and size, they are typically structured in entries, with each entry comprising one or more lines and reporting a specific event along with a timestamp indicating the exact time of the event and, if necessary, even identification numbers and paths.The length of the log files in the dataset varies greatly. The log file length depends directly on the type of test. Some tests are very short and simply start and end the test sequence, while others are longer and perform several consecutive subtests to test the endurance of the product. Among the multitude of existing log files, frontend and backend log files are preferred for analysis, as they have proven to be particularly informative for machines, especially machine tools. These files are similar in format but differ in the information they contain. The frontend log is responsible for recording the events on the user interface of the machine under test or its processor, while the backend log collects information about the process in the backend of the same processor.It is understood that a large number of processes can be executed on the machine and, accordingly, a large number of log files can be output by the test system.

[0030] The image file can be a screen recording of a test sequence for the error report at the time the error was output. The video file can be a screen video recording of the test sequence for the error report. Particularly when carrying out GUI tests, the system screen can be recorded to create a video that can be saved in an appropriate file format, for example as an MP4, AVI file or similar. Such a video can be crucial for identifying the problematic event that led to the test failure. In order to conserve the limited data storage space available, it may be sufficient if only one or more screen recordings are created as error report attachments at and / or shortly before and / or after the time the error was output and are taken into account or read in in the process.Screen recording and screen video recording can also be considered together as bug report attachments in order to be able to evaluate the largest possible data set for grouping.

[0031] Finally, it can also be provided that the formats include different text formats of text files. The text formats can refer to the previously mentioned different subformats, which are specified by the type of subformat, such as error message, log file, and / or stack trace. However, they can alternatively or additionally also refer to different formats within formats or subformats of the same type, for example, different formatting of the frontend log file and the backend log file.

[0032] Different similarity operators can be determined for different formats, particularly media formats. For example, a Levenshtein edit distance can be determined for stack traces. This approach is based on the fact that, although there is a precise mathematical definition of similarity, it is in a sense the inverse of a distance measure. This makes it possible to derive similarity from a distance measure. Since distance measures have some well-defined properties, a distance can be calculated instead of a similarity measure. For log files, for example, the calculation of Jaccard similarity can be used as a similarity operator. This metric quantifies the similarity between two sample sets. It is equal to the ratio between the number of elements in the intersection and the union of these sets. That is, the more elements they have in common, the greater the similarity.For video files and other media formats, for example, a cosine similarity can be determined. Given two objects A and B, each represented as a vector, the cosine of the deviation between these two vectors is the cosine similarity, or, in other words, Ochiai or Carbo similarity.

[0033] However, it is also possible that several similarity operators are provided or can be determined for the same domains, whereby one of these several similarity operators can be selected for the respective domain or a comparison can be carried out with different similarity operators for the same domain.

[0034] The result of the error report groups can be used in a variety of ways, and this use can be a further component of the process. For example, the result can be transmitted to the test system(s) executing the test sequence to improve the testing, particularly in the form of a machine learning algorithm; to the product(s) being tested, for example, to output or display the specific error situation on a corresponding screen or a control element of the product; to a service technician for troubleshooting error situations, etc., and used by them.

[0035] It can also be provided that for each error report group at least one group-based metric is determined, in particular a ratio of the number of log files for one of the error situations to the total number of log files. This metric can be used to very easily determine the most frequent error situation. This metric is preferably applied to the log files of a predetermined period of time, e.g. the log files from the last week. An error situation can describe both a product error and a false alarm. The metric can therefore be used to determine the most frequently occurring product errors and false alarms. This allows the work of the developers to be prioritized.

[0036] The method may also further comprise the step of combining the error reports of at least one of the error report groups into a group error report. This results in only one group error report, particularly one that is aggregated with regard to its error information. The aggregated group error report can facilitate faster error resolution, since it contains only the most important or necessary error information and no data unnecessary for error investigation.

[0037] In addition, the method can further comprise the step of determining an error criticality of the error report groups and in particular prioritizing the error report groups according to the determined error criticality. In other words, an error severity of the error report groups is determined. The error criticality can be determined, for example, on the basis of one or more of the previously mentioned metrics. This can lead to prioritization in error correction in order to make the product, for example the machine tool, functional again or error-free as quickly as possible. Finally, it can also be provided that machine learning is used when determining a similarity operator, when determining the similarity values ​​and / or when grouping the error reports.As already mentioned, a corresponding Kl model can increase the robustness of the method, for example by better determining similarity operators and grouping thresholds.

[0038] The object mentioned at the outset is further achieved by a computer program product according to claim 12. The computer program product comprises instructions which, when the computer executes the computer program product, cause the computer to carry out the method according to the invention.

[0039] The computer program product may be a computer program as such or a computer-readable storage medium on which the computer program or its instructions are stored. The storage medium may be a tangible device that can store and save instructions for use by an instruction execution device, i.e. a computer, in particular a processor. The computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the aforementioned devices.A non-exhaustive list of more specific examples of a computer-readable storage medium includes the following: a portable computer diskette, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punched cards or raised structures in a groove with instructions recorded thereon, and any suitable combination of the foregoing. A computer-readable storage medium, as used herein, should not be understood to include transitory signals per se, such as, for example, flash memory.Radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses traveling through a fiber optic cable), or electrical signals transmitted through a wire.

[0040] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to the respective computing / processing devices or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.The computer-readable program instructions for performing operations within the scope of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state data, integrated circuit configuration data, or either source code or object code written in any combination of one or more programming languages, including an object-oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages ​​such as the "0" programming language or similar programming languages.The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the latter case, the remote computer may be connected to the user's computer over any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., over the Internet using an Internet service provider).In some embodiments, electronic circuits, including, for example, programmable logic circuits, field programmable gate arrays (FPGA), or programmable logic arrays (PLA), may execute the computer-readable program instructions by using state information of the computer-readable program instructions to personalize the electronic circuits to perform aspects of the present invention.

[0041] The object mentioned at the outset is also achieved by a grouping system according to claim 13. The grouping system according to the invention comprises a memory and a processor connected to the memory, wherein the computer program product according to the invention is stored on the memory and the processor is configured to execute the instructions of the computer program product.

[0042] The grouping system may have at least one interface to at least one test system, wherein the at least one test system is configured to generate error reports each having at least two error report attachments of different domains.

[0043] Finally, the object mentioned at the outset is achieved by a system according to claim 15, comprising the grouping system according to the invention and the at least one test system.

[0044] In this case, features which are described herein with reference to the method according to the invention apply equally with reference to the computer program product according to the invention, the grouping system and the system and vice versa.

[0045] Further details and advantageous embodiments of the invention can be found in the following description, on the basis of which embodiments of the invention are described and explained in more detail.

[0046] Shown are: Figure 1 a schematic view of a method according to an embodiment of the invention;

[0047] Figure 2 is a schematic view of part of the

[0048] Method from Fig. 1 according to a first embodiment of the invention;

[0049] Figure 3 is a schematic view of part of the

[0050] Method from Fig. 1 according to a second embodiment of the invention;

[0051] Figure 4 is a schematic view of part of the

[0052] Method of Fig. 1 according to a third embodiment of the invention; and

[0053] Figure 5 is a schematic view of a system according to an embodiment of the invention, comprising a grouping system according to an embodiment of the invention and a plurality of test systems.

[0054] In the following description and in the figures, the same reference symbols are used for identical or corresponding features.

[0055] Figure 1 schematically shows a computer-implemented method 100 for grouping error reports 10 into at least two error report groups 30 of different error situations 31, 32, 33, 34, i.e. the error report groups 30 each have different error situations 31, 32, 33, 34 with respect to the error reports 10 grouped therein.

[0056] A step 102 that can be attributed to the method 100 or is separate therefrom is the testing itself or the test sequence or system test. For this purpose, products are tested using test systems 1. Purely as an example, two test systems 1 are tested here, although of course there can be significantly more than two test systems 1 or just a single test system 1. Each of the test systems 1 can be used on one or more products (not shown), e.g. a machine tool with the computer program(s) located thereon.

[0057] Software applications or, alternatively, the computer program as such. The testing in step 102 can otherwise be carried out according to known principles, in particular as automated testing of the respective test system 1. For example, GUI tests can be used. The result of the system test is in each case an error report 10. The error report 10 can contain errors.

[0058] In the subsequent step 104, the error reports 10 generated in the test sequence of step 102 are read in. In particular, only those error reports that actually contain errors can be read in, since only these contain an error message and are relevant for error analysis in order to increase the robustness of the products.

[0059] The error reports 10 each comprise two or more error report attachments 11, as shown schematically in Fig. 1. These error report attachments 11 can be of different domains 12, 13, 14, in particular of different formats, as illustrated in Fig. 1 by way of example for three different formats 12, 13, 14, although of course more different formats 12, 13, 14 are present and can be distinguished.

[0060] By way of example, the following description refers to different formats 12, 13, 14 as a special embodiment of the different domains 12, 13, 14. In principle, however, different domains can not only have different formats, they can also be of the same format but refer to a different context or contain different context information. An example of an identical file format with different context information and thus a different domain is, for example, a video file as a recording of a screen during the execution of a test and a video file as a recording of a laser cutting process of a workpiece.

[0061] The different formats 12, 13, 14 can, for example, be different text formats of error report attachments 11 in the form of text files, for example differently formatted log files, stack traces, error messages, etc. The formats 12, 13, 14 can, in particular, be different media formats, such as text files, image files, video files, and / or audio files. For example, the image files can be screen recordings or screenshots of the test sequence, in particular in the case of a GUI test, at the time of the error. For example, the video files can be video recordings of the system screen of the test system 1 carrying out the test or its computer, in particular in the case of a GUI test, the video comprising at least a period of time at and / or shortly before and / or shortly after the time of the error. In this respect, the error reports 10 ortheir bug report attachments 11 are described as multimodal.

[0062] In step 106 of the method 100, an associated similarity operator 20, 21, 22 is determined for each of the different formats 12, 13, 14. Example similarity operators 20, 21, 22 can be, for example, specialized edit distances for stack traces, Jaccard similarities of text content or text files, and cosine similarities for other content, in particular image and video files. The similarity operator 20, 21, 22 is determined such that it serves to identify bug report attachments 11 with similar error situations. In other words, the similarity operator 20, 21, 22 can be used to determine how similar two bug report attachments 11 are to each other (by calculating a similarity value 15). This will be explained later by way of example and clearly with reference to Figures 2 and 3.

[0063] In step 108 of the method, similarity values ​​15 between the error report attachments 11 are determined by comparing error report attachments 11 of the same format 12, 13, 14 and by means of the similarity operator 20, 21, 22 associated with the respective format 12, 13, 14. The respective similarity values ​​15 of an error report attachment 11 indicate the similarity for the specific error report attachment 11 of a format 12, 13, 14 compared to various other error report attachments 11 of the same format 12, 13, 14. In particular, similarity matrices (not shown) can be created in this way. As Figs. 2, 3 and 4 show, the read-in error report attachments 11 are compared in particular with previously read-in or saved error report attachments 11 of the same format 12, 13, 14.These stored error report attachments 11 can in particular originate from previously grouped error report groups 30, so that each of these stored error report attachments 11 has already been assigned one of the various four error situations 31, 32, 33, 34 shown purely by way of example in Fig. 1 based on the grouping into the respective error report group 30.

[0064] After step 108, the information is now available as to which error report attachment 11 of an already grouped error report 10 of a specific error situation 31, 32, 33, 34 a respective error report attachment 11 of a specific format 12, 13, 14 of the read-in error report 10 can be assigned.

[0065] However, the error report attachments 11 are not considered separately for grouping purposes. Instead, in step 110 of the method 100, the error reports 10 are grouped as a whole based on the similarity values ​​15 of the error report attachments 11 of each error report 10. Thus, in step 110, the entire error report 10 can be assigned to a specific error situation 31, 32, 33, 34.

[0066] At the same time, a particularly precise comparison of the individual error report attachments 11 of the same format 12, 13, 14 could advantageously be carried out. By also taking into account error report attachments 11 of different formats 12, 13, 14 and comparing them with each other by the respective similarity operators 20, 21, 22, it is ensured that no information regarding error messages from the

[0067] Error reports 10 are disregarded. This is because it may happen that an error report attachment 11 of a specific format 12, 13, 14 does not contain an error message or related information, while an error report attachment 11 of a different format 12, 13, 14 of the same error report 10 contains related information. In this way, the robustness of the method 100 and, as a consequence, of the respective product can be significantly increased.

[0068] In step 110, the error reports 10 are now grouped into the individual error report groups 30 with the different error situations 31, 32, 33, 34, wherein error reports 10 that are similar with regard to the error situation 31, 32, 33, 34 are grouped into the respective error report group 30. This grouping is carried out based on the previously determined similarity values ​​15 of the read-in error report attachments 11 of each error report 10.

[0069] The result of the error report groups 30 can now be used in a variety of ways, with the optional steps 112, 114 and 116 being shown as an example in the further course of the method 100 in order to increase the possible uses.

[0070] For example, the result can be transmitted to the test system(s) 1 to improve the testing, in particular in the form of a machine learning algorithm, to the product(s) being tested, for example to output or display the specific error situation on a corresponding screen or a control element of the product, to a service technician for rectifying product errors, etc., and used by this or that system.

[0071] For example, for step 112, it is provided that for the error report groups 30 of different error situations 31, 32, 33, 34, at least one group-based metric is determined, in particular a ratio of a number of log files 10 of one of the error situations 31, 32, 33, 34 to a total number of log files 10.

[0072] For step 114, it is intended that the error reports 10 in the error report groups 30 are summarized into a group error report with aggregated information regarding the respective error situation 31, 32, 33, 34, for example, error cause, frequency, image, etc. Thus, only one aggregated group error report is available. This allows, for example, faster error resolution.

[0073] For step 116, it is provided that a fault criticality, or in other words, fault severity, of the fault report groups is determined. In this case, a prioritization of the fault report groups 30 according to the determined fault criticality can also be provided. The fault criticality can be determined, for example, based on one or more of the previously mentioned metrics. This allows for prioritization during fault rectification in order to restore the product, for example, the machine tool, to maximum functionality or fault-free status as quickly as possible.

[0074] In addition, the database created from the error report groups 30 can be easily searched for specific error situations 31 , 32 , 33 , 34 in order to quickly classify new, possibly previously unknown error situations.

[0075] Figure 2 now shows a schematic detailed view of steps 108, 110 of the method 100 from Fig. 1 with respect to an error report 10, which here, by way of example, has two error report attachments 11. The error report attachments 11 can be read in directly or previously processed in order to be used in the method 100. For example, in the case of image files as the file format of the error report attachments 11, the error report attachments 11 can be vectorized beforehand.

[0076] By means of the similarity operator 20, 21 determined for each of the formats 12, 14 of the error report attachments 11 of the read-in error report 10, the determination of similarity values ​​15 for each compared pair of error report attachments 11 is then carried out by comparison with error report attachments 11, in particular error reports 10 (here purely as an example for the sake of clarity only two are shown) which are already grouped into the error report groups 30 (here purely as an example of the error situations 31, 32).

[0077] In this example, four similarity values ​​15 are now present, as illustrated in the table with example values ​​shown in Fig. 2. In the left column, the error report attachments 11 of formats 12 and 14 of the imported error report 10 are printed in bold, while in the top column, the error report attachments 11 of formats 12, 14 of the already grouped error reports 10 are printed (identified and distinguished by the assignment of the respective error situation in parentheses). Each similarity value 15 now represents how similar the respective error report attachments 11 are relative to one another.As can be seen, in this example, the error report attachments 11 of the read-in error report 10 are more similar to the error report attachments 11 of the error report 10 grouped in the error report group 30 of the error situation 32 than to the error report attachments 11 of the error report 10 grouped in the error report group 30 of the error situation 31, because they each have a higher similarity value 15.

[0078] In this example of step 110, the highest similarity values ​​15 of all error report attachments 11 of the same format 12, 14 that are compared with one another are used. These are, for example, 0.89 for the error report attachment 11 of the read-in error report 10 of format 12 and 0.96 for the error report attachment 11 of the read-in error report 10 of format 14. The respective highest similarity values ​​15 are then compared with a respective grouping logic 35, in this case in the form of a comparison with a grouping limit value that is selected differently for each of the formats 12, 14, for example 0.8 and 0.9.

[0079] In the example of Fig. 2, the highest similarity values ​​15 are each greater than the respective grouping limit values, so that an overall similarity value 16 of 100% (2 / 2 similarity values ​​15 above the respective grouping limit values ​​35) is determined for the error report 10. A further grouping limit value of the grouping logic 35, which is not shown here, can now also be compared with this overall similarity value 16, which must be 100%, for example, so that each of the grouping limit values ​​for the similarity values ​​15 must be reached by the highest similarity values ​​15. Accordingly, the error report 10 can now be grouped into the associated error report group 30 based on its error pattern, which here corresponds to the error situation 32 as an example.

[0080] Furthermore, it is possible to combine or merge error report groups 30 of different error situations 31, 32 during the process 100 if it emerges from the individual error report attachments 11 that these relate to similar error situations 31, 32, for example, if error report attachments 11 can be assigned to the same error report groups 30. It is also of course possible to create a new error report group 30 of a different error situation 33, 34 (see Fig. 1) if one or more error report attachments 11 cannot be assigned to any of the previous error report groups 30 according to the grouping logic 35.

[0081] Figure 3 shows an alternative example for steps 108, 110 of method 100, wherein the same exemplary error reports 10 as in Fig. 2 with the same similarity values ​​15 are used in the example. The overall similarity value 16 is determined from the two highest similarity values ​​15, here as an example as the mean, and then compared with a grouping limit value of the grouping logic 35 for the overall similarity value 16, which here is an example of 0.85. Then, depending on the comparison, here positive because the grouping limit value 35 is exceeded by the overall similarity value 16, the error report 10 is grouped into the error report group 30 of the error situation 32.

[0082] Figure 4 again shows an alternative example for the steps 108, 110 of the method 100, wherein the same exemplary error reports 10 as in Fig. 2 with the same similarity values ​​15 are used in the example. Here, the grouping logic 35 is designed as a maximum requirement or condition that the error report 10 with the greatest overall similarity mean 16, here 0.925 for error report 10 of error situation 32 and 0.855 for error report 10 of error situation 31 (cf. Fig. 2), is assigned to the respectively associated error report group 30, thus here again to error situation 32.

[0083] The steps 108, 110 shown in Figs. 2 and 3 with the operations contained therein with regard to the similarity values ​​15, the overall similarity value 16 and the grouping logic 35 are preferred, but only by way of example.

[0084] Of course, other operations can also be provided by which overall similarity values ​​16 are determined from similarity values ​​15, and other grouping logics 35 can also be provided, so that the examples given here are merely examples and in no way restrictive. What is crucial is that similarity values ​​15 are determined for the error report attachments 11, these are then brought to an error report level, for example with the overall similarity value 16, and finally the grouping is carried out, wherein, for example, one or more grouping threshold values ​​can be used within the framework of the grouping logic 35. Figure 5 finally schematically shows a system 200 which shows a grouping system 5 according to an embodiment of the invention.The grouping system 5 in the present case comprises a processor 6 and a memory 7 on which a computer program product 8 is stored, which comprises instructions which, when the computer program product 8 is executed by the processor 6, cause the processor 6 to carry out the method 100 according to Fig. 1.

[0085] In order to carry out the method 100, the computer program product 8 requires the error reports 10 from the test system(s) 1. The test system(s) 1 are part of the shown overarching system 200, which also includes the grouping system 5. The grouping system 5 has an interface 9, through which it can receive the error reports 10 from the test systems 1. Of course, the test systems 1 can have a corresponding interface (not shown) for data communication. In the case of physical interfaces, the transmission can be wired or wireless, for example. The test systems 1 each have a test processor 2 and a test memory 3, on which a test computer program 4 for carrying out step 102 of the method 100 is stored. The test system(s) 1 can in this case each be the product(s) itself, with the test computer program 4 being executed thereon in order to test the products orto test their computer programs .

[0086] As an alternative to the illustration in Fig. 5, the test system 1 and the grouping system 5 can also be executed on one of the two systems 1, 5, for which purpose only one memory 7 and one processor 6 can be present in each case, wherein both computer programs 4, 8 can be stored in the memory 7. One of these two systems 1, 5 can also be formed by the respective product, in particular the machine, so that the method 100 according to the invention can also be carried out partially or completely directly on the product. The interface 9 can also be implemented by computer program code.

Claims

Patent claims 1. Computer-implemented method (100) for grouping error reports (10) into at least two error report groups (30) of different error situations (31, 32, 33, 34), wherein the method (100) comprises at least the following steps (104, 106, 108, 110): - Reading in error reports (10) with at least two error report attachments (11) of different domains (12, 13, 14), - determining a similarity operator (20, 21, 22) for at least two of the different domains (12, 13, 14) in order to determine a similarity between error report attachments (11) of the same domain (12, 13, 14), - determining similarity values ​​(15) of the error report attachments (11) of the same domain (12, 13, 14) by comparing individual error report attachments (11) of the same domain (12, 13, 14) of different error reports (10) by means of the previously determined similarity operator (20, 21, 22), and - Grouping the error reports (10) into at least two error report groups (30) of different error situations (31, 32, 33, 34) based on the previously determined similarity values ​​(15).

2. Method (100) according to claim 1, wherein the Similarity values ​​(15) are compared with a grouping logic (35) in order to group the error reports (10) into the at least two error report groups (30).

3. Method (100) according to claim 1 or 2, wherein, when determining the similarity values ​​(15), error report attachments (11) of already grouped error reports (10) are compared with read-in error report attachments (11) of not yet grouped error reports (10) by means of the respectively previously determined similarity operator (20, 21, 22).

4. Method (100) according to one of the preceding claims, wherein the error reports (10) are multimodal and the different domains (12, 13, 14) have different formats (12, 13, 14), in particular different media formats.

5. The method (100) according to claim 4, wherein the different media formats are selected from the group comprising image file, video file, audio file and / or text file, in particular error message and / or log file and / or stack trace.

6. The method (100) according to claim 5, wherein the image file is a screen capture of a test sequence for the error report (10) at an error output time and / or the Video file is a screen video recording of the test procedure for the bug report (10).

7. The method (100) according to any one of the preceding claims, wherein the error report attachments (11) of different domains (12, 13, 14) comprise text files of different text formats.

8. Method (100) according to one of the preceding claims, wherein at least one group-based metric is determined for each of the error report groups (30), in particular a ratio of a number of log files (10) of one of the error situations (31, 32, 33, 34) to a total number of log files (10).

9. The method (100) according to any one of the preceding claims, wherein the method (100) further comprises the step (114) of summarizing the error reports (10) of at least one of the error report groups (30) into a group error report.

10. The method (100) according to any one of the preceding claims, wherein the method (100) further comprises the step (116) of determining an error criticality of the error report groups (30) and in particular prioritizing the error report groups (30) according to the determined error criticality.

11. Method (100) according to one of the preceding claims, wherein machine learning is used in determining a respective similarity operator (20, 21, 22), in determining the similarity values ​​(15) and / or in grouping the error reports (10).

12. Computer program product (8) comprising instructions which, when the computer program product is executed by a computer, cause the computer to carry out the method (100) according to one of the preceding claims.

13. Grouping system (5) comprising a memory (7) and a processor (6) connected to the memory (7), wherein the computer program product (8) according to claim 12 is stored on the memory (7) and the processor (6) is configured to execute the instructions of the computer program product (8).

14. Grouping system (5) according to claim 13, further comprising at least one interface (9) to at least one test system (1), wherein the at least one test system (1) is configured to generate error reports (10) each having at least two error report attachments (11) of different domains (12, 13, 14).

15. System (200) comprising the grouping system (5) according to Claim 14 and the at least one test system (1) .

Images