Abnormity collection method and device, electronic equipment and storage medium

Abstract

The invention provides an exception collection method and device, electronic equipment and a storage medium, and the method comprises the steps: collecting basic data, stack tracking data including abnormal data and virtual machine internal error data through a preset mode, and obtaining an exception collection result based on each type of abnormal data in the basic data and the occurrence frequency of the error data; sampling the abnormal data and the error data corresponding to each type according to a preset sampling frequency aiming at the occurrence frequency to obtain to-be-uploaded data; wherein the occurrence frequency is obtained through a Bloom filter; and adding the to-be-uploaded data into the uploading array so as to upload the to-be-uploaded data in the array in batches. According to the method, stack tracking data and virtual machine internal errors are collected through different collection modes, the integrity of basic data is improved, abnormity tracing is facilitated, the abnormity analysis accuracy is improved, the basic data is sampled, abnormity and error information is reserved, and meanwhile the data size of abnormal data and error data needing to be uploaded is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an abnormality collection method, device, electronic equipment and storage medium. Background Art

[0002] Various network services are generally implemented through underlying codes. When an exception occurs in a network service, it is necessary to analyze the underlying code of the network service to obtain the cause of the exception, so as to improve the underlying code of the network service according to the cause of the exception.

[0003] The underlying code of network services can be implemented in Java language. The operating environment of Java language is usually implemented by Java Virtual Machine (JVM). Java Virtual Machine (JVM) is an abstract computer that simulates various computer functions on an actual computer. It is a virtual computer that can run Java code. As long as the interpreter is transplanted to a specific computer according to the JVM specification description, it can be guaranteed that any compiled Java code can run on the system.

[0004] In the related art, when obtaining exceptions and errors, usually only an error code can be obtained, and the error code can only identify the general type of exception, but cannot locate the location where the exception occurs. Summary of the invention

[0005] In view of this, embodiments of the present invention provide an exception collection method, device, electronic device and storage medium to improve the completeness of exception collection and reduce the amount of data processing.

[0006] According to one aspect of the present invention, a method for collecting abnormalities is provided, the method comprising:

[0007] Collecting basic data in a preset manner, wherein the basic data includes stack trace data of the abnormal data and internal error data of the virtual machine, and the stack trace data is used to identify a call link of the abnormal data;

[0008] Based on the number of occurrences of each type of abnormal data and error data in the basic data, sampling the abnormal data and error data corresponding to each type according to a sampling frequency preset for the number of occurrences to obtain data to be uploaded; wherein the number of occurrences is obtained through a Bloom filter;

[0009] The data to be uploaded is added to the upload array, so as to batch upload the data to be uploaded in the upload array.

[0010] In a possible embodiment, the method further includes:

[0011] When the upload array is full, based on the service identifier of the data to be uploaded, encapsulate the data to be uploaded into a data block corresponding to the service identifier, and add the data block to the message queue;

[0012] When a preset callback condition is met, calling a preset callback function to process the data block in the message queue, wherein the logic of the preset callback function at least includes sending a data block or storing a data block, and the preset callback condition is that the number of data blocks in the message queue reaches a preset number or the waiting time of the data blocks in the message queue exceeds a preset duration;

[0013] In a possible embodiment, collecting basic data in a preset manner includes:

[0014] By embedding the fillInStackTrace() method, the exception data captured by throwable is obtained, and the stack trace data corresponding to the exception data is obtained through the getStackTrace() method of throwable; and,

[0015] An uncaught error data operation function is added to each thread of the JVM virtual machine. The uncaught error data operation function is used to capture error events when the thread runs abnormally, and obtain the stack trace data of the error event through the getStackTrace() method of throwable.

[0016] In a possible embodiment, the method further includes:

[0017] For each basic data, the basic data is mapped using a preset hash function to obtain each mapping position of the basic data in the Bloom filter, and the counting result of the counter of the mapping position is increased by 1;

[0018] A statistical value based on a counting result of a counter corresponding to the basic data in the Bloom filter is used as the number of occurrences of the basic data, wherein the statistical value is a minimum value or an average value.

[0019] In a possible embodiment, based on the number of occurrences of each type of abnormal data and error data in the basic data, sampling the abnormal data and error data corresponding to each type according to a sampling frequency preset for the number of occurrences to obtain the data to be uploaded includes:

[0020] For each type of abnormal data and error data in the basic data, the abnormal data and error data of the type are sampled based on a sampling order corresponding to a number interval in which the number of occurrences of the type lies, wherein the sampling order is generated based on the number interval and the sampling frequency, and is used to identify the order of the data to be uploaded that needs to be sampled in the abnormal data and error data of the type.

[0021] According to another aspect of the present invention, there is provided an abnormality collection device, the device comprising:

[0022] A collection module, used to collect basic data in a preset manner, wherein the basic data includes stack trace data of abnormal data and internal error data of the virtual machine, and the stack trace data is used to identify a call link of the abnormal data;

[0023] A sampling module, for sampling the abnormal data and the error data corresponding to each type according to a sampling frequency preset for the number of occurrences of each type of abnormal data and the error data in the basic data, to obtain data to be uploaded; wherein the number of occurrences is obtained through a Bloom filter;

[0024] The uploading module is used to add the data to be uploaded to the upload array so as to batch upload the data to be uploaded in the upload array.

[0025] In a possible embodiment, the device further includes:

[0026] A waiting module, configured to encapsulate the data to be uploaded into a data block corresponding to the service identifier based on the service identifier of the data to be uploaded, and add the data block to the message queue when the upload array is full; and to call a preset callback function to process the data block in the message queue when a preset callback condition is met, wherein the logic of the preset callback function at least includes sending a data block or storing a data block, and the preset callback condition is that the number of data blocks in the message queue reaches a preset number or the waiting time of the data blocks in the message queue exceeds a preset duration;

[0027] In a possible embodiment, collecting basic data in a preset manner includes:

[0028] By embedding the fillInStackTrace() method, the exception data captured by throwable is obtained, and the stack trace data corresponding to the exception data is obtained through the getStackTrace() method of throwable; and,

[0029] Add an uncaught error data operation function in each thread of the JVM virtual machine, wherein the uncaught error data operation function is used to capture an error event when the thread runs abnormally, and obtain the stack trace data of the error event through the getStackTrace() method of throwable;

[0030] The sampling module is used to map each basic data by using a preset hash function to obtain each mapping position of the basic data in the Bloom filter, and add 1 to the counting result of the counter of the mapping position;

[0031] A statistical value of a counting result of a counter corresponding to the basic data in the Bloom filter is used as the number of occurrences of the basic data, wherein the statistical value is a minimum value or an average value;

[0032] The method of sampling the abnormal data and the error data corresponding to each type according to a sampling frequency preset for the number of occurrences based on the number of occurrences of each type of abnormal data and the error data in the basic data to obtain the data to be uploaded includes:

[0033] For each type of abnormal data and error data in the basic data, the abnormal data and error data of the type are sampled based on a sampling order corresponding to a number interval in which the number of occurrences of the type lies, wherein the sampling order is generated based on the number interval and the sampling frequency, and is used to identify the order of the data to be uploaded that needs to be sampled in the abnormal data and error data of the type.

[0034] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0035] Processor; and

[0036] Memory for storing programs,

[0037] The program includes instructions, which, when executed by the processor, enable the processor to execute any of the above-mentioned abnormality collection methods.

[0038] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the above-mentioned abnormality collection methods.

[0039] One or more technical solutions provided in the embodiments of the present invention can collect stack data of exception data and internal error data of the virtual machine by setting different collection methods for different exceptions and error types, thereby improving the integrity of the collected basic data and facilitating the tracing of the exceptions, thereby improving the accuracy of subsequent exception analysis. Furthermore, by sampling the corresponding basic data based on the number of occurrences of the exception data and the error data, while retaining the exception and error information, the amount of exception data and error data that need to be uploaded is greatly reduced, thereby reducing the amount of data processing for subsequent exception analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0041] Figure 1 A schematic diagram of a flow chart of an abnormality collection method provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a process for collecting general abnormal data in an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of a process for collecting internal error data of a virtual machine in an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of counting basic data in an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of a flow chart for obtaining the number of occurrences of abnormal and erroneous data in an embodiment of the present invention;

[0046] Figure 6 A schematic diagram of a process for collecting abnormal and error data in an embodiment of the present invention

[0047] Figure 7 A schematic diagram of a process of uploading data to be uploaded in an embodiment of the present invention;

[0048] Figure 8 A schematic diagram of the structure of an abnormality collection device provided by an embodiment of the present invention;

[0049] Fig. 9 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0050] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0051] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0052] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0053] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0054] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0055] In order to improve the integrity of abnormal data collection and reduce the amount of abnormal analysis data, the embodiments of the present invention provide an abnormal collection method, device, electronic device and storage medium. The abnormal collection method provided by the embodiments of the present invention can be applied to any electronic device with an abnormal collection function, which can be a server, a computer or a mobile terminal, etc. The scheme of the present invention is described below with reference to the accompanying drawings:

[0056] Figure 1 A flowchart of an abnormality collection method provided by an embodiment of the present invention may include the following steps:

[0057] S101. Collect basic data in a preset manner, wherein the basic data includes stack trace data of abnormal data and internal error data of a virtual machine, and the stack trace data is used to identify a call link of the abnormal data;

[0058] S102, based on the number of occurrences of each type of abnormal data and error data in the basic data, sampling the abnormal data and error data corresponding to each type according to a sampling frequency preset for the number of occurrences to obtain data to be uploaded; wherein the number of occurrences is obtained through a Bloom filter;

[0059] S103: Add the data to be uploaded to an upload array, so as to batch upload the data to be uploaded in the upload array.

[0060] By applying the embodiments of the present invention, by setting different collection methods for different exceptions and error types, stack data of exception data and internal error data of the virtual machine can be collected, thereby improving the integrity of the collected basic data, facilitating the tracing of the exceptions, and thus improving the accuracy of subsequent exception analysis. Furthermore, by sampling the corresponding basic data based on the number of occurrences of the exception data and the error data, while retaining the exception and error information, the amount of exception data and error data that need to be uploaded is greatly reduced, thereby reducing the amount of data processing for subsequent exception analysis.

[0061] The above S101-S103 are exemplarily described below:

[0062] The exception collection method provided by the present invention can be applied to scenarios such as operating systems and application APPs to collect exception data and error data in these scenarios. In a possible embodiment, the above-mentioned preset method may include a collection method for general exceptions and error data settings and a collection method for error data inside the JVM virtual machine. Among them, an exception (Exception) is a mechanism for indicating errors or unexpected situations that occur during program operation. Exceptions provide a way to separate error handling code from conventional code execution paths, allowing programs to better manage and handle error conditions. The exception mechanism in Java is implemented through try-catch blocks and exception classes (such as Exception and Error). Errors refer to serious problems that occur during program operation, which are usually beyond the control of the program itself and cannot be recovered by conventional exception handling mechanisms. Errors usually indicate system-level problems, such as virtual machine errors, insufficient memory, etc. This type of error is represented by the Error class and its subclasses.

[0063] As a possible implementation method, the above basic data can be collected through the following steps:

[0064] By embedding the fillInStackTrace() method, the exception data captured by throwable is obtained, and the stack trace data corresponding to the exception data is obtained through the getStackTrace() method of throwable; and,

[0065] An uncaught error data operation function is added to each thread of the JVM virtual machine. The uncaught error data operation function is used to capture error events when the thread runs abnormally, and obtain the stack trace data of the error event through the getStackTrace() method of throwable.

[0066] In a possible embodiment, for common exception data and error data, the public synchronized Throwable fillInStackTrace() method in java.lang.Throwable can be embedded. Specifically, it can be set to collect Throwable data through the return value when calling fillInStackTrace(), and the corresponding stack trace data can be obtained through the getStackTrace() method of Throwable. Among them, Throwable is the superclass of all errors and exceptions in Java. It is a throwable object that represents an error or abnormal situation that occurs during the running of the program. The Throwable class has two main subclasses: Error and Exception. Common exception data and error data are exceptions that can be collected by throwable. The stack trace data includes the location where the exception occurs and the dependent parameter path of the exception. The location where the above exception occurs is the specific line of the specific code where the exception is located, and the dependent parameter path of the exception is the parameter that the exception depends on and the code information that generates the parameter.

[0067] The fillInStackTrace() method is a synchronized method of the java.lang.Throwable class in Java, which is used to fill the stack trace information of the exception. This method is usually called automatically when an exception is caught in order to obtain the stack trace when the exception occurs.

[0068] like Figure 2 , Figure 2The present invention is a flowchart for collecting exception and error data in an embodiment of the present invention. Specifically, when an exception or error occurs, the constructor of throwable is called to call the fillInStackTrace() method to collect the throwable object, that is, the data where the exception or error occurs, and the getStackTrace() method is called to obtain the stack trace data of the object. The stack trace data includes the specific location in the code where the exception or error data occurs and the call chain of the data, that is, the code location of the parameters on which the data depends.

[0069] In the related art, generally, only the exceptions and errors thrown by the buried point are collected by pre-burying the point. If the exception and error are not thrown, they cannot be collected. In the embodiment of the present invention, the fillInStackTrace() method is used to actively obtain the throwable object to obtain all exception and error objects. In the related art, only the exception code is generally returned for the exception, resulting in the inability to locate the root cause of the problem even if the exception is seen. In the embodiment of the present invention, the exception information and the stack trace data of the exception are returned to trace the data that caused the exception, which is convenient for locating the exception.

[0070] In the related art, it is generally impossible to collect error data inside the JVM virtual machine, such as the error that causes the thread to exit or crash. In the present invention, the constructor of java.lang.Thread can be embedded, and a custom UncaughtExceptionHandler can be set to the thread to collect uncaught exceptions in the custom UncaughtExceptionHandler. The above-mentioned uncaught exceptions usually refer to exceptions that are not caught by throwable. Specifically, Figure 3 As shown, Figure 3 A flow chart of collecting internal error data of a virtual machine provided in an embodiment of the present invention. Specifically, after the program is running and the thread is started, the thread construction method is called to set a custom UncaughtExceptionHandler. If an uncaught exception occurs during the thread running, the UncaughtExceptionHandler can be used to capture the uncaught exception and collect records.

[0071] Each of the above-collected basic data may include its exception information and stack trace data. The above-mentioned exception information may include the type of exception, the time of occurrence, the location of occurrence, and the like.

[0072] Through the above steps, a full amount of exception and error data can be collected, but the same exception or error data may exist in the exception and error data. If the same error is collected multiple times, it will greatly affect the efficiency of data processing. In addition, if a key-value map is used, if there are too many exception types in a time period, the occupied memory space will be uncontrollable, which is a waste of memory space. Therefore, the basic data collected above can be filtered.

[0073] In a possible embodiment, repeated basic data can be filtered by a Bloom filter, which is a combination of a very long binary vector and a series of random mapping functions, and can be used to retrieve whether an element is in a set. And the Bloom filter can record the number of occurrences of data by counting, without storing the specific content of the data, which greatly saves memory space.

[0074] As a possible implementation, for each basic data, a preset hash function may be used to map the basic data to obtain each mapping position of the basic data in the Bloom filter, and the count result of the counter of the mapping position is increased by 1;

[0075] A statistical value based on a counting result of a counter corresponding to the basic data in the Bloom filter is used as the number of occurrences of the basic data, wherein the statistical value is a minimum value or an average value.

[0076] In a possible embodiment, different types of basic data can be mapped separately, and the types of the basic data can be distinguished by error code identifiers or code identifiers where errors or exceptions occur. The above error code identifiers can indicate specific exception types, such as 404 for page not found, 401 for unauthorized access, etc. The above code identifiers can be the code name where the exception occurs and the identifier of the row where the exception code is located. Basic data with the same error code identifier or code identifier are data of the same type.

[0077] There may be multiple preset hash functions, and the specific number of hash functions and the specific function contents can be flexibly set according to the actual application scenario. By mapping the basic data using the multiple preset hash functions, a hash value corresponding to the basic data can be obtained, and the hash value can identify the position of the basic data in the Bloom filter. The same type of basic data can obtain the same hash value after being mapped by the hash function.

[0078] In a possible embodiment, in order to prevent the hash value from overflowing the storage range of the Bloom filter, the length of the Bloom filter may be set to a larger value, which may be specifically set according to an actual business scenario.

[0079] Each position in the Bloom filter is a counter. After obtaining the hash value of the basic data, the counter at the position corresponding to the hash value can be increased by 1. Figure 4 As shown, for the basic data, the basic data is mapped by hash function 1, hash function 2 and hash function 3 to obtain corresponding counter positions 1, 2, and 3. Therefore, the counts at counter positions 1, 2, and 3 can all be increased by 1.

[0080] The number of occurrences of the same type of basic data can be subsequently queried based on the Bloom filter. For example, the statistical value of the counting results of each counter position corresponding to the basic data of the type can be used as the number of occurrences of the basic data of the type. The statistical value can be an average, a median, a minimum, or the like. As a possible implementation, since each basic data may be mapped to the same position after being mapped by the multiple preset hash functions, the minimum value of the count value in the counter corresponding to the basic data may be mapped to the least position by other basic data, that is, a value closer to the number of occurrences of the basic data. Therefore, the minimum count value in each counter can be used as the number of occurrences of the basic data, thereby improving the accuracy of judging the number of occurrences.

[0081] In a possible embodiment, the Bloom filter can store each position counter and its counting result through a counter array. Figure 5 As shown, Figure 5 The present invention is a flowchart of using the minimum value in the counting results of each counter as the number of occurrences of basic data. Specifically, the basic data to be queried is determined, and the basic data to be queried is mapped based on a preset hash function, the count value in the counter position recording the basic data is determined, and the minimum value among the count values ​​is determined as the approximate value of the number of occurrences of the basic data. Exemplarily, when recording the number of occurrences of the basic data, the basic data is mapped by hash function 1, hash function 2, and hash function 3, and 1 count is added in the corresponding counter position 1, counter position 2, and counter position 3. When querying an element, the basic data to be queried can be mapped by hash function 1, hash function 2, and hash function 3, and the count values ​​of counter position 1, counter position 2, and counter position 3 can be obtained from the counter array, and the counter array includes counter position 1, counter position 2, counter position 3, counter position 4, and counter position 5. Based on the count values ​​of counter position 1, counter position 2, and counter position 3, the minimum value calculation is performed to obtain the number of elements of the basic data to be queried, that is, the number of occurrences.

[0082] In a possible embodiment, a new counting Bloom filter can be initialized per unit time. During the unit time, each exception processing needs to first obtain the number of current type records through the counting Bloom filter. If the number of records is 0, then insert it directly, and directly collect the exception this time. If the number of records is not 0, then the basic data can be sampled and collected. The above unit time can be set according to the actual application scenario, such as 1 hour, 6 hours, etc.

[0083] In a possible embodiment, for each type of abnormal data and error data in the basic data, the abnormal data and error data of the type can be sampled based on a sampling order corresponding to a number interval in which the number of occurrences of the type lies, wherein the sampling order is generated based on the number interval and the sampling frequency, and is used to identify the order of the data to be uploaded that needs to be sampled in the abnormal data and error data of the type.

[0084] The above-mentioned number intervals and sampling frequencies can be set according to the actual application scenario, and it is only necessary to follow the rule that the more times the abnormality occurs, the lower the sampling frequency. Exemplarily, the sampling rate can be set to 10% for the 1-10 stage, 2% for the 10-100 stage, and 0.5% for the 100-1000 stage. In a possible embodiment, a sampling order can be generated based on the number interval and the sampling rate. Exemplarily, the sampling rate of 10-100 is 2%, so two random numbers in the range of 10-100 can be generated as the sampling order.

[0085] When the number of occurrences of the basic data reaches the number interval, the basic data that conforms to the sampling order corresponding to the number interval can be collected. For example, based on the above example, the sampling order generated for the 10-100 interval is 45 and 76, so the basic data that appears for the 45th and 76th time in this type of basic data can be collected.

[0086] like Figure 6 As shown, Figure 6 The present invention is a flowchart of collecting basic data in an embodiment of the present invention. Specifically, a new counting Bloom filter is initialized per unit time, and a sampling rule is preset. Specifically, the sampling rate is 10% in the 1-10 stage, 2% in the 10-100 stage, and 0.5% in the 100-1000 stage. After obtaining the abnormal data, the value of the type record of the abnormal data is obtained through the counting Bloom filter. If the number of records is 0, the abnormal data is directly collected. If the number of records is not 0, the sampling rule is used to calculate whether to collect. Specifically, it is determined whether the order of occurrence of the abnormal data is the preset sampling order. If so, the abnormal data can be collected. If not, the abnormal data is not collected.

[0087] For ease of description, the collected basic data is referred to as data to be uploaded in the present invention. The data to be uploaded refers to abnormal or error data that needs to be uploaded to the server for further abnormality analysis. The data to be uploaded may include abnormal information and corresponding stack trace data.

[0088] Since the collected stack trace data is a relatively large amount of data, if too much data is sent at one time, the data packet will be too large, resulting in a long network transmission time and container timeout failure. If the packet sent is small but the frequency is too high, it is easy to waste resources, resulting in frequent network and high CPU consumption. Based on this, the upload array transferDataArray can be preset, and the maximum amount of data that can be uploaded at a time can be specified in the upload array. As a possible implementation method, the data to be uploaded can be added to the transferDataArray by calling the add method.

[0089] In a possible embodiment, the method may further include encapsulating the data to be uploaded into a data block corresponding to the service identifier based on the service identifier of the data to be uploaded when the upload array is full, and adding the data block to the message queue;

[0090] When a preset callback condition is met, a preset callback function is called to process the data blocks in the message queue, wherein the logic of the preset callback function at least includes sending data blocks or storing data blocks, and the preset callback condition is that the number of data blocks in the message queue reaches a preset number or the waiting time of the data blocks in the message queue exceeds a preset duration.

[0091] When the upload array is full or the preset upload time is reached, the data in the upload array can be uploaded. In a possible embodiment, if the upload array is full or batchadd is false, that is, the data cannot be uploaded, the data to be uploaded can be encapsulated as a data block (in the actual code, it can be encapsulated as a TransferDataBlock object), and the data block is added to the message queue TransferDataBlockQueue, which is a blocking queue. As a possible implementation method, the data to be uploaded belonging to the same business can be encapsulated into the same data block, so that abnormal data of the cloud storage business is convenient.

[0092] In a possible embodiment, a consumption thread may be set, which may take out data blocks from the TransferDataBlockQueue queue and process the data by calling the process method of the TransferDataCallBack interface to send the data in the data block to the server.

[0093] In a possible embodiment, a callback function can also be registered in the message queue, and the callback condition of the callback function can be set. The callback function and the callback condition can be set according to the actual application scenario. Exemplarily, the callback function can be set to send the data in the data block to the server by calling the process method of the TransferDataCallBack interface, or to call the batchToQueue method to add the temporarily stored data in the message queue to the upload array. The above callback condition can be that the number of data blocks in the message queue reaches a preset number or the waiting time of the data blocks in the message queue exceeds a preset duration.

[0094] In a possible embodiment, the callback condition can be set by the poll method of the TransferDataBlockQueue class, which is usually used to retrieve and remove the head element from the queue. If the queue is empty and the time from the last callback exceeds pollCallbackTimeOut (callback timeout), the callback will be triggered and the poll method of all registered NotificationCallback will be called, where NotificationCallback is an interface that defines a set of callback methods. In the poll method of the TransferDataBlockProducer class, an attempt will be made to call the batchToQueue method to add the temporarily stored data to the queue.

[0095] like Figure 7 As shown, Figure 7 The figure is a flow chart of uploading data in an embodiment of the present invention. Specifically, after the data to be uploaded is generated in the TransferDataBlockProducer class, the data is added to the transferDataArray array when the add method is called. If the array is full or batchAdd is false, the data is immediately encapsulated into a TransferDataBlock object and added to the queue. When there is no data in the queue, the consumer thread will be blocked until a new data block is added to the queue.

[0096] The callback functions are pre-registered in the TransferDataBlockQueue. When the data in the queue reaches the preset callback conditions, these callback functions will be triggered. TransferDataTask is the data consumption thread, which is responsible for taking data blocks from the TransferDataBlockQueue queue and processing these data. The processing method can be to send the data in the data block to the server by calling the process method of the TransferDataCallBack interface.

[0097] By applying the embodiment of the present invention, a control function is added to the JVM virtual machine to capture the internal error data of the virtual machine, thereby improving the acquisition integrity of the basic abnormal data. In addition, the basic data is counted by the Bloom filter to reduce the consumption of memory. The abnormal data that needs to be sent is greatly reduced by sampling the repeated abnormal data. In addition, the collected data is uploaded by setting an array and a queue method to achieve a balance between system performance and transmission efficiency, thereby realizing efficient, rapid and reasonable collection of abnormal data.

[0098] Based on the same inventive concept, the embodiment of the present invention also provides an abnormality collection device, such as Figure 8 As shown, the apparatus 800 may include:

[0099] The collection module 801 is used to collect basic data in a preset manner, wherein the basic data includes stack trace data of abnormal data and internal error data of the virtual machine, and the stack trace data is used to identify the call link of the abnormal data;

[0100] The sampling module 802 is used to sample the abnormal data and the error data corresponding to each type according to the sampling frequency preset for the number of occurrences of each type of abnormal data and error data in the basic data, so as to obtain data to be uploaded; wherein the number of occurrences is obtained through a Bloom filter;

[0101] The uploading module 803 is used to add the data to be uploaded to the upload array, so as to batch upload the data to be uploaded in the upload array.

[0102] In a possible embodiment, the device further includes:

[0103] A waiting module, configured to encapsulate the data to be uploaded into a data block corresponding to the service identifier based on the service identifier of the data to be uploaded, and add the data block to the message queue when the upload array is full; and to call a preset callback function to process the data block in the message queue when a preset callback condition is met, wherein the logic of the preset callback function at least includes sending a data block or storing a data block, and the preset callback condition is that the number of data blocks in the message queue reaches a preset number or the waiting time of the data blocks in the message queue exceeds a preset duration;

[0104] In a possible embodiment, collecting basic data in a preset manner includes:

[0105] By embedding the fillInStackTrace() method, the exception data captured by throwable is obtained, and the stack trace data corresponding to the exception data is obtained through the getStackTrace() method of throwable; and,

[0106] Add an uncaught error data operation function in each thread of the JVM virtual machine, wherein the uncaught error data operation function is used to capture an error event when the thread runs abnormally, and obtain the stack trace data of the error event through the getStackTrace() method of throwable;

[0107] The sampling module is used to map each basic data by using a preset hash function to obtain each mapping position of the basic data in the Bloom filter, and add 1 to the counting result of the counter of the mapping position;

[0108] A statistical value of a counting result of a counter corresponding to the basic data in the Bloom filter is used as the number of occurrences of the basic data, wherein the statistical value is a minimum value or an average value;

[0109] The method of sampling the abnormal data and the error data corresponding to each type according to a sampling frequency preset for the number of occurrences based on the number of occurrences of each type of abnormal data and the error data in the basic data to obtain the data to be uploaded includes:

[0110] For each type of abnormal data and error data in the basic data, the abnormal data and error data of the type are sampled based on a sampling order corresponding to a number interval in which the number of occurrences of the type lies, wherein the sampling order is generated based on the number interval and the sampling frequency, and is used to identify the order of the data to be uploaded that needs to be sampled in the abnormal data and error data of the type.

[0111] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0112] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.

[0113] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0114] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0115] refer to Fig. 9 , a block diagram of an electronic device 900 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0116] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0117] A plurality of components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 may be any type of device capable of inputting information to the electronic device 900, and the input unit 906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 908 may include, but is not limited to, a disk, an optical disk. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0118] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned abnormal acquisition method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. In some embodiments, the computing unit 901 may be configured to perform the above-mentioned abnormal acquisition method by any other appropriate means (e.g., by means of firmware).

[0119] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0120] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0124] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. An abnormality collection method, characterized in that: The method comprises: Collecting basic data in a preset manner, wherein the basic data includes stack trace data of the abnormal data and internal error data of the virtual machine, and the stack trace data is used to identify a call link of the abnormal data; Based on the number of occurrences of each type of abnormal data and error data in the basic data, sampling the abnormal data and error data corresponding to each type according to a sampling frequency preset for the number of occurrences to obtain data to be uploaded; wherein the number of occurrences is obtained through a Bloom filter; The data to be uploaded is added to the upload array, so as to batch upload the data to be uploaded in the upload array.

2. The method according to claim 1, characterized in that The method further comprises: When the upload array is full, based on the service identifier of the data to be uploaded, encapsulate the data to be uploaded into a data block corresponding to the service identifier, and add the data block to the message queue; When a preset callback condition is met, a preset callback function is called to process the data blocks in the message queue, wherein the logic of the preset callback function at least includes sending data blocks or storing data blocks, and the preset callback condition is that the number of data blocks in the message queue reaches a preset number or the waiting time of the data blocks in the message queue exceeds a preset duration.

3. The method according to claim 1, characterized in that The basic data is collected in a preset manner, including: By embedding the fillInStackTrace() method, the exception data captured by throwable is obtained, and the stack trace data corresponding to the exception data is obtained through the getStackTrace() method of throwable; and, An uncaught error data operation function is added to each thread of the JVM virtual machine. The uncaught error data operation function is used to capture error events when the thread runs abnormally, and obtain the stack trace data of the error event through the getStackTrace() method of throwable.

4. The method according to claim 1, characterized in that The method further comprises: For each basic data, the basic data is mapped using a preset hash function to obtain each mapping position of the basic data in the Bloom filter, and the counting result of the counter of the mapping position is increased by 1; A statistical value based on a counting result of a counter corresponding to the basic data in the Bloom filter is used as the number of occurrences of the basic data, wherein the statistical value is a minimum value or an average value.

5. The method according to claim 1, characterized in that The method of sampling the abnormal data and the error data corresponding to each type according to a sampling frequency preset for the number of occurrences based on the number of occurrences of each type of abnormal data and the error data in the basic data to obtain the data to be uploaded includes: For each type of abnormal data and error data in the basic data, the abnormal data and error data of the type are sampled based on a sampling order corresponding to a number interval in which the number of occurrences of the type lies, wherein the sampling order is generated based on the number interval and the sampling frequency, and is used to identify the order of the data to be uploaded that needs to be sampled in the abnormal data and error data of the type.

6. An abnormality collection device, characterized in that: The device comprises: A collection module, used to collect basic data in a preset manner, wherein the basic data includes stack trace data of abnormal data and internal error data of the virtual machine, and the stack trace data is used to identify a call link of the abnormal data; A sampling module, for sampling the abnormal data and the error data corresponding to each type according to a sampling frequency preset for the number of occurrences of each type of abnormal data and the error data in the basic data, to obtain data to be uploaded; wherein the number of occurrences is obtained through a Bloom filter; The uploading module is used to add the data to be uploaded to the upload array so as to batch upload the data to be uploaded in the upload array.

7. The device according to claim 6, characterized in that The device also includes: A waiting module is used to encapsulate the data to be uploaded into a data block corresponding to the business identifier of the data to be uploaded based on the business identifier of the data to be uploaded, and add the data block to the message queue when the upload array is full; when a preset callback condition is met, call a preset callback function to process the data block in the message queue, wherein the logic of the preset callback function at least includes sending data blocks or storing data blocks, and the preset callback condition is that the number of data blocks in the message queue reaches a preset number or the waiting time of the data blocks in the message queue exceeds a preset duration.

8. The device according to claim 6, characterized in that The basic data is collected in a preset manner, including: By embedding the fillInStackTrace() method, the exception data captured by throwable is obtained, and the stack trace data corresponding to the exception data is obtained through the getStackTrace() method of throwable; and, Add an uncaught error data operation function in each thread of the JVM virtual machine, wherein the uncaught error data operation function is used to capture an error event when the thread runs abnormally, and obtain the stack trace data of the error event through the getStackTrace() method of throwable; The sampling module is used to map each basic data by using a preset hash function to obtain each mapping position of the basic data in the Bloom filter, and add 1 to the counting result of the counter of the mapping position; A statistical value of a counting result of a counter corresponding to the basic data in the Bloom filter is used as the number of occurrences of the basic data, wherein the statistical value is a minimum value or an average value; The method of sampling the abnormal data and the error data corresponding to each type according to a sampling frequency preset for the number of occurrences based on the number of occurrences of each type of abnormal data and the error data in the basic data to obtain the data to be uploaded includes: For each type of abnormal data and error data in the basic data, the abnormal data and error data of the type are sampled based on a sampling order corresponding to a number interval in which the number of occurrences of the type lies, wherein the sampling order is generated based on the number interval and the sampling frequency, and is used to identify the order of the data to be uploaded that needs to be sampled in the abnormal data and error data of the type.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 5.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-5.

Images