Log processing method, device, electronic device and storage medium

Abstract

The present invention discloses a log processing method, device, electronic device and storage medium. The method includes: establishing a log collection channel between a log collection application and at least one business application according to a thread pool of a log collection application, wherein each thread in the thread pool corresponds to a business application; obtaining log data sent by the business application in the log collection application according to the log collection channel; and synchronously storing the log data to a file system according to a preset memory mapping strategy. The embodiment of the present invention sets up a separate log collection application, uses the thread pool of the log collection application to establish a log collection channel with each business application, enables the log collection application to obtain the log data sent by each business application through the log collection channel, and synchronously stores the log data to a file system according to a preset memory mapping strategy, thereby realizing persistent storage of log data without calling an additional log storage service, and having strong system compatibility.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a log processing method, device, electronic device and storage medium. Background Art

[0002] In the process of automotive software development, in order to facilitate the analysis and location of software and hardware problems, the current data, time, status, etc. are usually printed and stored in the form of files on the disk through the software interface provided by the log system through the log system. As a distributed real-time operating system commonly used in the automotive field, the QNX operating system provides a slogger2 log service that can store all recorded log data in shared memory, and then store the log data in the file system through another slog2info process service. The above log storage method needs to rely on the native service of the QNX operating system to realize the storage of log data, and cannot be customized and optimized according to actual needs and usage scenarios; at the same time, for different versions of the QNX operating system, the supported log service functions are also different, so the above log processing method has the problem of poor compatibility. Summary of the invention

[0003] The present invention provides a log processing method, device, electronic device and storage medium to solve the problem of poor compatibility of the existing log processing method.

[0004] According to one aspect of the present invention, a log processing method is provided, the method comprising:

[0005] Establishing a log collection channel between the log collection application and at least one business application according to the thread pool of the log collection application, wherein each thread in the thread pool corresponds to one business application;

[0006] Obtain log data sent by business applications in the log collection application according to the log collection channel;

[0007] Log data is synchronously stored in the file system according to the preset memory mapping strategy.

[0008] According to another aspect of the present invention, a log processing device is provided, the device comprising:

[0009] A channel establishment module, used to establish a log collection channel between the log collection application and at least one business application according to the thread pool of the log collection application, wherein each thread in the thread pool corresponds to a business application;

[0010] A data acquisition module is used to acquire log data sent by a business application in a log collection application according to a log collection channel;

[0011] The data storage module is used to synchronously store log data to the file system according to a preset memory mapping strategy.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log processing method described in any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log processing method described in any embodiment of the present invention when executed.

[0017] The technical solution of the embodiment of the present invention is to establish a log collection channel between the log collection application and at least one business application according to the thread pool of the log collection application, wherein each thread in the thread pool corresponds to a business application; obtain the log data sent by the business application in the log collection application according to the log collection channel; and synchronously store the log data to the file system according to a preset memory mapping strategy. The embodiment of the present invention sets up a separate log collection application, uses the thread pool of the log collection application to establish a log collection channel with each business application, enables the log collection application to obtain the log data sent by each business application through the log collection channel, and synchronously stores the log data to the file system according to a preset memory mapping strategy, thereby realizing persistent storage of log data without calling an additional log storage service, and having strong system compatibility.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1is a flow chart of a log processing method provided according to Embodiment 1 of the present invention;

[0021] Figure 2 is a flow chart of a log processing method provided according to Embodiment 2 of the present invention;

[0022] Figure 3 This is an example diagram of a log processing system provided according to Embodiment 3 of the present invention;

[0023] Figure 4 This is an example diagram of a message transmission mechanism provided according to Embodiment 3 of the present invention;

[0024] Figure 5 is a flow chart of a log processing method provided according to Embodiment 3 of the present invention;

[0025] Figure 6 This is an example diagram of a function call provided according to Embodiment 3 of the present invention;

[0026] Figure 7 This is an example diagram of a memory mapping mechanism provided according to Embodiment 3 of the present invention;

[0027] Figure 8 is a structural schematic diagram of a log processing device provided according to Embodiment 4 of the present invention;

[0028] Fig. 9 It is a structural schematic diagram of an electronic device for implementing the log processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 This is a flow chart of a log processing method provided in the first embodiment of the present invention. This embodiment can be applied to the case where a log collection application is controlled to obtain log data and write the log data into a file system. The method can be executed by a log processing device, which can be implemented in the form of hardware and / or software. The log processing device can be configured in an electronic device with computing capabilities. Figure 1 As shown, a log processing method provided in this embodiment includes the following steps:

[0033] S110 . Establish a log collection channel between the log collection application and at least one business application according to the thread pool of the log collection application, wherein each thread in the thread pool corresponds to one business application.

[0034] Wherein, business application can refer to the application program (Application, APP) for generating log data and sending log data to log collection application, the number of business applications can be one or more, the specific number can be determined according to the actual business, and the embodiment of the present invention does not limit this. Log collection application can refer to the application program for receiving the log data sent by each business application, and writing the log data to the file system. Thread pool can be understood as a multithreaded processing form, and a plurality of threads can be included in the thread pool, the number of threads can be at least the same as the number of business applications, and each thread corresponds to a business application respectively, for realizing the communication connection between log collection application and each business application, completing the sending (Send), receiving (Receive) and reply (Reply) for log data. Log collection channel can be understood as the channel that log collection application connects with each business application respectively, and the number of log collection channels can be the same as the number of business applications, that is, log collection application is connected with each business application respectively through a separate log collection channel.

[0035] In an embodiment of the present invention, in order to enable a log collection application to collect log data sent by each business application, the log collection application can be controlled to create and start a thread pool that includes at least the same number of threads as the business applications, and corresponding threads are allocated to each business application according to the thread pool, that is, each thread in the thread pool corresponds to a business application; then the log collection application can be controlled to create a log collection channel corresponding to the business application in the thread corresponding to each business application by calling an application programming interface (API) function, etc., wherein the log collection application is connected to each business application through a separate log collection channel, and subsequently message transmission can be implemented according to the corresponding log collection channel between the log collection application and each business application.

[0036] S120 . Obtain log data sent by the business application in the log collection application according to the log collection channel.

[0037] In an embodiment of the present invention, the business application acts as a client and, after each business application generates log data according to actual business needs, based on the log collection channel between the created log collection application and each business application, controls the business application to send the log data to the log collection application through the corresponding log collection channel by calling an API function or the like in the corresponding thread; the log collection application acts as a server and can obtain the corresponding log data through the log collection channel and feed back a response message to the corresponding business application according to the actual data processing situation.

[0038] S130. Synchronously store the log data in the file system according to a preset memory mapping strategy.

[0039] The preset memory mapping strategy can be understood as a pre-configured strategy for realizing a one-to-one mapping relationship between the memory space of the log collection application and the log file of the file system. Through the preset memory mapping strategy, writing to the memory can be realized instead of writing to the file, thereby improving the efficiency of writing to the log data. The file system can refer to a system for persistently storing log data.

[0040] In an embodiment of the present invention, the log collection application can be controlled to create a log file under the file system, and a preset memory mapping function is called to map the log file to the memory space of the log collection application, and then the acquired log data is written to the memory space. Since the memory space of the log collection application has established a mapping relationship (binding) with the log file, reading and writing operations on the mapped memory space are the same as reading and writing operations on the log file, skipping the I / O process of the log file; at the same time, some active synchronization trigger conditions can be set. For example, when the amount of log data in the memory space reaches a preset capacity threshold, or the log file of the file system is full, all log data in the memory space is actively synchronized and stored in the file system by calling an API function, etc., thereby realizing persistent storage of log data and avoiding data loss in some special scenarios (such as power outages).

[0041] The technical solution of the embodiment of the present invention is to establish a log collection channel between the log collection application and at least one business application according to the thread pool of the log collection application, wherein each thread in the thread pool corresponds to a business application; obtain the log data sent by the business application in the log collection application according to the log collection channel; and synchronously store the log data to the file system according to a preset memory mapping strategy. The embodiment of the present invention sets up a separate log collection application, uses the thread pool of the log collection application to establish a log collection channel with each business application, enables the log collection application to obtain the log data sent by each business application through the log collection channel, and synchronously stores the log data to the file system according to a preset memory mapping strategy, thereby realizing persistent storage of log data without calling an additional log storage service, and having strong system compatibility.

[0042] Embodiment 2

[0043] Figure 2 The flowchart of a log processing method provided in the second embodiment of the present invention is further optimized and expanded based on the above implementation, and can be combined with various optional technical solutions in the above implementation. Figure 2 As shown, a log processing method provided in the second embodiment is applied to the QNX operating system and specifically includes the following steps:

[0044] S210. Control the log collection application to create and start a thread pool, wherein the thread pool includes at least the same number of threads as the business application.

[0045] In an embodiment of the present invention, based on the maximum number of supported threads, priority, scheduling frequency, and other thread pool configuration parameters, the log collection application can be controlled to call a thread pool creation function such as a thread_pool_create function to create a thread pool that includes at least the same number of threads as the business application, and call a thread pool start function such as a thread_pool_start function to start the thread pool.

[0046] S220 , controlling the log collection application to call a preset channel creation function according to each thread to create the same number of log collection channels as the business applications.

[0047] The preset channel creation function may refer to a pre-configured function for creating a log collection channel, and the preset channel creation function may be a ChannelCreate function of a QNX operating system.

[0048] In an embodiment of the present invention, a corresponding thread can be allocated to each business application according to the thread pool started by the log collection application, and a thread identifier corresponding to each thread can be obtained. Then, the log collection application is controlled to call a preset channel creation function such as a ChannelCreate function in the thread corresponding to each business application to create a log collection channel corresponding to the business application, and obtain a channel identifier corresponding to each log collection channel, wherein each business application corresponds to a separate thread, and the number of log collection channels is the same as the number of business applications, so that each business application can be controlled to connect to the log collection channel corresponding to the log collection application, thereby realizing the communication connection between the log collection application and each business application.

[0049] S230 , according to the channel identifier of each log collection channel and the thread identifier of each thread corresponding to the business application, control each business application to call a preset connection creation function so that each business application is connected to the log collection channel corresponding to the log collection application.

[0050] The channel identifier may refer to an identifier corresponding to the created log collection channel, and each log collection channel corresponds to a unique channel identifier. The preset connection creation function may refer to a pre-configured function for establishing a connection with the log collection channel, and the preset channel creation function may be a ConnectAttach function of the QNX operating system.

[0051] In an embodiment of the present invention, each business application can be controlled to call a preset connection creation function such as the ConnectAttach function to connect itself to the log collection channel corresponding to the log collection application according to the channel identifier of the corresponding log collection channel and the thread identifier of the corresponding thread, and at the same time obtain the connection identifier corresponding to the log collection application, so as to establish a communication connection between the log collection application and each business application through the log collection channel, and then message transmission can be performed between the log collection application and each business application, wherein the input parameters of the preset connection creation function, namely the channel identifier of the log collection channel and the thread identifier of the thread, can be used to uniquely identify the log collection application (server) to which the business application (client) is to connect.

[0052] S240 , controlling each business application to call a preset message sending function to send the log data to the log collection application through a corresponding log collection channel.

[0053] Among them, the preset message sending function may refer to a function of a pre-configured business application used to implement sending messages, i.e., log data. The preset message sending function may be the MsgSend function of the QNX operating system. The input parameters of the preset message sending function may include at least: a connection identifier of the log collection application, a pointer to the log data to be sent, the size of the log data to be sent, a pointer to the reply data, and the size of the reply data.

[0054] In an embodiment of the present invention, after each business application generates log data according to actual business needs, the corresponding business application can be controlled to call a preset message sending function, such as a MsgSend function, to send the log data to the log collection application through the corresponding log collection channel, wherein the input parameters of the preset message sending function may include at least: a connection identifier of the log collection application, a pointer to the log data to be sent, the size of the log data to be sent, a pointer to the reply data, and the size of the reply data.

[0055] S250 , controlling the log collection application to call a preset message receiving function to receive log data sent by each business application through a corresponding log collection channel.

[0056] Among them, the preset message receiving function may refer to a function of a pre-configured log collection application used to realize the reception of log data sent by each business application. The preset message receiving function may be the MsgReceive function of the QNX operating system. The input parameters of the preset message receiving function may include at least: the channel identifier of the log collection channel, a pointer to the reply data, and the size of the reply data.

[0057] In an embodiment of the present invention, the log collection application can be controlled to call a preset message receiving function, such as the MsgReceive function, to receive log data sent by each business application through the corresponding log collection channel, wherein the input parameters of the preset message receiving function can include at least: a channel identifier of the log collection channel, a pointer to the reply data, and the size of the reply data.

[0058] S260 , controlling the log collection application to call a preset message response function to feed back a response message corresponding to the log data to the business application through a corresponding log collection channel.

[0059] Among them, the preset message response function may refer to a pre-configured function used to implement the log collection application to feed back a response message, i.e., reply data, to the corresponding business application. The preset message response function may be the MsgReply function of the QNX operating system. The input parameters of the preset message response function may include at least: the thread identifier of the thread corresponding to the business application, a pointer to the reply data, and the size of the reply data.

[0060] In an embodiment of the present invention, after the log collection application processes the log data sent by each business application, the log collection application can be controlled to call a preset message response function, such as the MsgReply function, and the response message (including reply data) corresponding to the log data is fed back to the business application through the corresponding log collection channel, wherein the input parameters of the preset message response function can include at least: the thread identifier of the thread corresponding to the business application, a pointer to the reply data, and the size of the reply data.

[0061] Further, based on the above-mentioned embodiments of the invention, a log processing method provided in Embodiment 2 of the present invention further includes:

[0062] After the business application calls the preset message sending function, the thread corresponding to the business application is set to a blocked state until the corresponding business application receives a response message fed back by the log collection application, and then the blocked state of the corresponding thread is released.

[0063] In an embodiment of the present invention, after a business application calls a preset message sending function to send a piece of log data, the thread corresponding to the business application can be set to a blocked state until the thread executes a preset message receiving function and a preset message reply function, that is, after the corresponding business application receives the reply message fed back by the log collection application, the blocking state of the corresponding thread is released, and then the business application is controlled to call the preset message sending function to send the next piece of log data.

[0064] S270: Control the log collection application to create a log file in the file system.

[0065] The log file may refer to a file used to store log data, and the log file may be located in a disk of a file system.

[0066] In the embodiment of the present invention, the log collection application can be controlled to create one or more log files in the file system by calling a pre-configured log file creation function such as an Open function, and the file size can be set for the log file.

[0067] S280: Call a preset memory mapping function to map the log file to the memory space of the log collection application.

[0068] The preset memory mapping function may refer to a pre-configured function for implementing mapping of a log file of a file system to a memory space of a log collection application. The preset memory mapping function may not be limited to an MMAP function or the like.

[0069] In an embodiment of the present invention, a preset memory mapping function such as but not limited to the MMAP function can be called to map the created and opened log file to the memory space of the log collection application, and obtain a mapped memory address. Subsequently, the log file can be read and written according to the memory address, thereby reducing the number of memory copies and improving the efficiency of file read and write operations.

[0070] S290. Control the log collection application to write the received log data into the memory space.

[0071] In an embodiment of the present invention, since the memory space of the log collection application has established a mapping relationship (binding) with the log file, reading and writing operations on the mapped memory space are the same as reading and writing operations on the log file. Therefore, the log collection application can be controlled to write the received log data directly to the memory space, skipping the I / O process of the log file, thereby improving the efficiency of writing log data.

[0072] S2100: When the amount of log data in the memory space reaches a preset capacity threshold, all log data in the memory space are synchronously stored in the file system.

[0073] In an embodiment of the present invention, since the kernel of the QNX operating system will not actively synchronize the log data mapped to the memory space, in order to avoid the possibility of data loss in some special scenarios (such as power failure), an active synchronization trigger condition can be set, for example, when the amount of log data in the memory space reaches a certain preset capacity threshold, the msync function can be called to actively synchronize and store all log data in the memory space to the file system. The log data is written to the log file of the file system by memory mapping, and the I / O process of the log file is skipped, which reduces all system calls that need to be executed in the process of writing logs to storing logs, reduces the number of memory copies, and improves the efficiency of reading and writing file operations.

[0074] The technical solution of the embodiment of the present invention is to control a log collection application to create and start a thread pool, wherein the thread pool includes at least the same number of threads as the business application; control the log collection application to call a preset channel creation function according to each thread to create a log collection channel with the same number as the business application; control each business application to call a preset connection creation function according to the channel identifier of each log collection channel and the thread identifier of the thread corresponding to each business application so that each business application is connected to the log collection channel corresponding to the log collection application; control each business application to call a preset message sending function to send log data to the log collection application through the corresponding log collection channel; control the log collection application to call a preset message receiving function to receive the log data sent by each business application through the corresponding log collection channel; control the log collection application to call a preset message reply function to feed back the reply message corresponding to the log data to the business application through the corresponding log collection channel; control the log collection application to create a log file under the file system; call a preset memory mapping function to map the log file to the memory space of the log collection application; control the log collection application to write the received log data to the memory space; when the amount of log data in the memory space reaches a preset capacity threshold, all log data in the memory space are synchronously stored in the file system. The embodiment of the present invention is based on the QNX operating system, and by setting a separate log collection application, a thread pool started by the log collection application is used to allocate corresponding threads to each business application, and in the thread corresponding to each business application, a log collection channel is created as well as the sending, receiving and responding of log data, and then the log data is synchronously written into a log file of a file system by means of memory mapping, thereby realizing persistent storage of log data without calling an additional log storage service, and having strong system compatibility; at the same time, the memory mapping method is used to realize writing to the memory instead of writing to the file, thereby reducing all system calls that need to be executed in the process from writing the log to storing the log, reducing the number of memory copies, and improving the efficiency of file reading and writing operations.

[0075] Embodiment 3

[0076] Figure 3 This is an example diagram of a log processing system provided by Embodiment 3 of the present invention. Figure 3 As shown, the log processing system includes:

[0077] At least one business application, used to act as a client and send generated log data to a log collection application;

[0078] The log collection application is used as a server to receive log data sent by business applications and synchronize the log data to the file system through memory mapping;

[0079] The file system is used to persistently store the log data obtained by the log collection application.

[0080] Furthermore, based on the above-mentioned embodiments of the invention, the log collection application can start a thread pool, allocate corresponding threads to each business application, and realize the connection between the log collection application and each business application through the corresponding threads to receive the log data sent by each business application.

[0081] Further, based on the above-mentioned embodiments of the invention, Figure 4 As shown in the figure, the message passing mechanism provided by the QNX operating system microkernel mainly uses the blocked Send() function, Receive() function and Reply() function, which are maintained by the kernel and do not require the application to implement inter-process synchronization. In addition, when the business application sends a Send() to the log collection application, the business application will be blocked until the log collection application calls the Receive() function, processes the received message, and sends a Reply(), and then the corresponding business application will exit the blocking.

[0082] Based on the above log processing system, Figure 5 This is a flow chart of a log processing method provided in Example 3 of the present invention. Based on the above examples, this example provides an implementation of a log processing method, which can enable a log collection application to uniformly obtain log data of each business application and store the log data persistently in the file system through memory mapping. Figure 5 As shown, a log processing method provided by Embodiment 3 of the present invention is applied to a QNX operating system and specifically includes the following steps:

[0083] S310, controlling the log collection application to create and start a thread pool, and calling a preset channel creation function for each thread to create a log collection channel.

[0084] In the embodiment of the present invention, Figure 6 As shown, the log collection application acts as a server, which can control the log collection application to create and start the thread pool, and assign corresponding threads to each business application, obtain the thread identifier pid corresponding to each thread, and then call the preset channel creation function ChannelCreate function in the thread corresponding to each business application to create the log collection channel corresponding to the business application, and obtain the channel identifier chid corresponding to each log collection channel.

[0085] S320 , controlling each business application to call a preset connection creation function so that each business application is connected to a log collection channel corresponding to the log collection application.

[0086] In the embodiment of the present invention, the business application, as a client, can control the business application preset connection creation function ConnectAttach function to connect itself to the log collection channel corresponding to the log collection application, and obtain the connection identifier coid corresponding to the log collection application, so as to realize the log collection application to establish a communication connection with each business application through the log collection channel, wherein the ConnectAttach function is defined as follows:

[0087] coid=ConnectAttach(pid,chid)

[0088] S330 , controlling each business application to call a preset message sending function to send log data to a log collection application.

[0089] In the embodiment of the present invention, the business application can be controlled to call the preset message sending function MsgSend function to send the log data to the log collection application through the corresponding log collection channel, wherein the MsgSend function is defined as follows:

[0090] MsgSend(coid,smsg,sbytes,rmsg,rbytes)

[0091] Among them, coid represents the connection identifier of the log collection application; smsg represents the pointer to the log data to be sent; sbytes represents the size of the log data to be sent; rmsg represents the pointer to the reply data; and rbytes represents the size of the reply data.

[0092] S340 , controlling the log collection application to call a preset message receiving function to receive log data sent by each business application.

[0093] In the embodiment of the present invention, the log collection application can be controlled to call a preset message receiving function MsgReceive function to receive log data sent by each business application through the corresponding log collection channel, wherein the MsgReceive function is defined as follows:

[0094] MsgReceive(chid,rmsg,rbytes)

[0095] S350, controlling the log collection application to call a preset message response function to feed back a response message corresponding to the log data to the business application through a corresponding log collection channel.

[0096] In an embodiment of the present invention, after the log collection application processes the log data sent by each business application, the log collection application can be controlled to call a preset message reply function MsgReply function, and the reply message (including reply data) corresponding to the log data is fed back to the business application through the corresponding log collection channel, wherein the MsgReply function is defined as follows:

[0097] MsgReply(pid,rmsg,rbytes)

[0098] Further, based on the above-mentioned embodiment of the invention, after the business application calls the preset message sending function to send a piece of log data, the thread corresponding to the business application can be set to a blocked state until the thread executes the preset message receiving function and the preset message reply function, that is, after the corresponding business application receives the reply message fed back by the log collection application, the blocking state of the corresponding thread is released, and then the business application is controlled to call the preset message sending function to send the next piece of log data.

[0099] S360. Log data acquired by the log collection application is persistently stored in the file system by means of memory mapping.

[0100] In the embodiment of the present invention, Figure 7 As shown, S360 may specifically include the following steps:

[0101] ① Control the log collection application to create log files in the file system.

[0102] Specifically, the log collection application can call the Open function to create a log file in the file system, and can set the file size file_size for the log file, where the Open function is defined as follows:

[0103] fd=open(log,O_CREAT|O_RDWR,0777)

[0104] Among them, fd represents the file descriptor corresponding to the log file, which is used to specify the log file to be mapped; log represents the path of the log file; O_CREAT means to create a new file if the file does not exist, and use the third parameter (i.e. parameter 0777) to set permissions for it; O_RDWR means to open the file in read-write mode; 0777 means full read, write and execute permissions.

[0105] ② Call the preset memory mapping function to map the log file to the memory space of the log collection application, and control the log collection application to write the received log data into the memory space.

[0106] Specifically, a preset memory mapping function MMAP function may be called to map the log file to the memory space of the log collection application, and the log collection application may be controlled to write the received log data to the memory space, wherein the MMAP function is defined as follows:

[0107] addr=mmap(file_size,PROT_READ|PROT_WRITE,MAP_SHARED,fd)

[0108] Among them, addr represents the memory address after mapping; PROT_READ represents the access right of the memory space as readable; PROT_WRITE represents the access right of the memory space as writable; MAP_SHARED represents the mapping mode of the memory space as shared, that is, the mapping space can be shared with all other processes mapped to this file.

[0109] ③ When the log file is full, all log data in the memory space will be synchronously stored in the file system.

[0110] Specifically, when the log file is full, the msync function can be called to actively synchronize all log data in the memory space to the file system. The msync function is defined as follows:

[0111] msync(addr,file_size,MS_SYNC)

[0112] Among them, MS_SYNC indicates the use of synchronous writing mode.

[0113] The technical solution of the embodiment of the present invention controls the log collection application to create and start a thread pool, and calls a preset channel creation function according to each thread to create a log collection channel; controls each business application to call a preset connection creation function so that each business application is connected to the log collection channel corresponding to the log collection application; controls each business application to call a preset message sending function to send log data to the log collection application; controls the log collection application to call a preset message receiving function to receive the log data sent by each business application; controls the log collection application to call a preset message reply function to feed back the reply message corresponding to the log data to the business application through the corresponding log collection channel; and persistently stores the log data obtained by the log collection application in the file system through memory mapping. The technical solution of the embodiment of the present invention has the following beneficial effects: ① The implementation method of log data transmission (message transmission) is a basic function of the QNX operating system, and there is no compatibility problem caused by version differences; ② The message transmission mechanism is used to copy data directly between processes without queuing, and the performance of message transmission is close to the memory bandwidth of the underlying hardware, and the data transmission efficiency is higher; ③ Message transmission is a blocking call, and there is no need to add a complex synchronization mechanism, and the log data writing process is simpler; ④ It has a core source code of the log function, and the function can be reduced according to actual needs to reduce the system resource consumption of the log system; ⑤ The memory mapping mechanism is used to write to the memory instead of writing to the file, which reduces the number of kernel memory copies and reduces the kernel overhead.

[0114] Embodiment 4

[0115] Figure 8 This is a structural diagram of a log processing device provided by Embodiment 4 of the present invention. Figure 8 As shown, the device comprises:

[0116] The channel establishing module 41 is used to establish a log collection channel between the log collection application and at least one business application according to the thread pool of the log collection application, wherein each thread in the thread pool corresponds to a business application.

[0117] The data acquisition module 42 is used to acquire the log data sent by the business application in the log collection application according to the log collection channel.

[0118] The data storage module 43 is used to synchronously store the log data to the file system according to a preset memory mapping strategy.

[0119] The technical solution of the embodiment of the present invention is to establish a log collection channel between a log collection application and at least one business application according to the thread pool of the log collection application through a channel establishment module, wherein each thread in the thread pool corresponds to a business application; the data acquisition module obtains the log data sent by the business application in the log collection application according to the log collection channel; the data storage module synchronously stores the log data to the file system according to a preset memory mapping strategy. The embodiment of the present invention sets up a separate log collection application, uses the thread pool of the log collection application to establish a log collection channel with each business application, enables the log collection application to obtain the log data sent by each business application through the log collection channel, and synchronously stores the log data to the file system according to a preset memory mapping strategy, thereby realizing persistent storage of log data without calling an additional log storage service, and having strong system compatibility.

[0120] Further, based on the above-mentioned embodiment of the invention, the channel establishing module 41 includes:

[0121] A channel creation unit, used to control the log collection application to call a preset channel creation function according to each thread to create the same number of log collection channels as the business application;

[0122] The channel connection unit is used to control each business application to call a preset connection creation function according to the channel identifier of each log collection channel and the thread identifier of each business application corresponding to the thread, so that each business application is connected to the log collection channel corresponding to the log collection application.

[0123] Further, based on the above-mentioned embodiment of the invention, the data acquisition module 42 includes:

[0124] The log data sending unit is used to control each business application to call a preset message sending function to send the log data to the log collection application through the corresponding log collection channel.

[0125] The log data receiving unit is used to control the log collection application to call a preset message receiving function to receive the log data sent by each business application through the corresponding log collection channel.

[0126] The response feedback unit is used to control the log collection application to call a preset message response function to feed back the response message corresponding to the log data to the business application through the corresponding log collection channel.

[0127] Further, based on the above-mentioned embodiment of the invention, the data storage module 43 includes:

[0128] The log file creation unit is used to control the log collection application to create a log file in the file system.

[0129] The memory mapping unit is used to call a preset memory mapping function to map the log file to the memory space of the log collection application.

[0130] The log data writing memory unit is used to control the log collection application to write the received log data into the memory space.

[0131] The data synchronization unit is used to synchronize and store all log data in the memory space to the file system when the amount of log data in the memory space reaches a preset capacity threshold.

[0132] Furthermore, based on the above-mentioned embodiment of the invention, the log processing device further includes:

[0133] The thread blocking state setting module is used to set the thread corresponding to the business application to a blocking state after the business application calls the preset message sending function, and release the blocking state of the corresponding thread after the corresponding business application receives the response message fed back by the log collection application.

[0134] Further, based on the above-mentioned embodiment of the invention, the input parameters of the preset message sending function include at least: a connection identifier of the log collection application, a pointer to the log data to be sent, the size of the log data to be sent, a pointer to the reply data, and the size of the reply data;

[0135] The input parameters of the preset message receiving function include at least: a channel identifier of the log collection channel, a pointer to the reply data, and a size of the reply data;

[0136] The input parameters of the preset message reply function include at least: a thread identifier of a thread corresponding to the business application, a pointer to the reply data, and a size of the reply data.

[0137] Further, based on the above-mentioned embodiments of the invention, the log processing device is applied to the QNX operating system when executing the log processing method provided by any embodiment of the present invention.

[0138] The log processing device provided in the embodiment of the present invention can execute the log processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0139] Embodiment 5

[0140] Fig. 9A schematic diagram of an electronic device 50 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0141] like Fig. 9 As shown, the electronic device 50 includes at least one processor 51, and a memory connected to the at least one processor 51 in communication, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 51 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 52 or the computer program loaded from the storage unit 58 to the random access memory (RAM) 53. In the RAM 53, various programs and data required for the operation of the electronic device 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0142] A number of components in the electronic device 50 are connected to the I / O interface 55, including: an input unit 56, such as a keyboard, a mouse, etc.; an output unit 57, such as various types of displays, speakers, etc.; a storage unit 58, such as a disk, an optical disk, etc.; and a communication unit 59, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0143] The processor 51 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 51 executes the various methods and processes described above, such as a log processing method.

[0144] In some embodiments, the log processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 50 via the ROM 52 and / or the communication unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the log processing method described above may be performed. Alternatively, in other embodiments, the processor 51 may be configured to execute the log processing method in any other appropriate manner (e.g., by means of firmware).

[0145] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0147] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0149] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0150] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0151] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0152] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log processing method, characterized in that: The method comprises: Establishing a log collection channel between the log collection application and at least one business application according to the thread pool of the log collection application, wherein the log collection application is an independent application program, and each thread in the thread pool corresponds to one of the business applications; Acquiring the log data sent by the business application in the log collection application according to the log collection channel; Synchronously storing the log data to the file system according to a preset memory mapping strategy; The step of establishing a log collection channel between the log collection application and at least one business application according to the thread pool of the log collection application includes: Controlling the log collection application to create and start the thread pool, wherein the thread pool includes at least the same number of threads as the business application; Control the log collection application to call a preset channel creation function according to each of the threads to create the same number of log collection channels as the business applications; According to the channel identifier of each log collection channel and the thread identifier of each thread corresponding to the business application, control each business application to call a preset connection creation function so that each business application is connected to the log collection channel corresponding to the log collection application; The step of synchronously storing the log data to the file system according to a preset memory mapping strategy includes: Controlling the log collection application to create a log file under the file system; Calling a preset memory mapping function to map the log file to the memory space of the log collection application; Controlling the log collection application to write the received log data into the memory space; When the amount of log data in the memory space reaches a preset capacity threshold, all the log data in the memory space are synchronously stored in the file system.

2. The method according to claim 1, characterized in that The acquiring the log data sent by the business application in the log collection application according to the log collection channel includes: Control each of the business applications to call a preset message sending function to send the log data to the log collection application through the corresponding log collection channel; Controlling the log collection application to call a preset message receiving function to receive the log data sent by each business application through the corresponding log collection channel; The log collection application is controlled to call a preset message response function to feed back a response message corresponding to the log data to the business application through the corresponding log collection channel.

3. The method according to claim 2, characterized in that Also includes: After the business application calls the preset message sending function, the thread corresponding to the business application is set to a blocked state until the corresponding business application receives the response message fed back by the log collection application, and then the blocking state of the corresponding thread is released.

4. The method according to claim 2, characterized in that: The input parameters of the preset message sending function include at least: a connection identifier of the log collection application, a pointer to the log data to be sent, the size of the log data to be sent, a pointer to the reply data, and the size of the reply data; The input parameters of the preset message receiving function include at least: the channel identifier of the log collection channel, a pointer to the reply data, and the size of the reply data; The input parameters of the preset message response function include at least: a thread identifier of a thread corresponding to the business application, a pointer to reply data, and a size of the reply data.

5. The method according to any one of claims 1 to 4, characterized in that: The method is applied to the QNX operating system.

6. A log processing device, characterized in that: The device comprises: A channel establishment module, used to establish a log collection channel between the log collection application and at least one business application according to the thread pool of the log collection application, wherein the log collection application is an independent application program, and each thread in the thread pool corresponds to one of the business applications; A data acquisition module, used for acquiring the log data sent by the business application in the log collection application according to the log collection channel; A data storage module, used for synchronously storing the log data in a file system according to a preset memory mapping strategy; The channel establishment module includes: A thread pool starting unit, used to control the log collection application to create and start the thread pool, wherein the thread pool includes at least the same number of threads as the business application; A channel creation unit, used for controlling the log collection application to call a preset channel creation function according to each of the threads to create the same number of log collection channels as the business applications; A channel connection unit, used to control each of the business applications to call a preset connection creation function according to the channel identifier of each of the log collection channels and the thread identifier of each of the business applications corresponding to the thread, so that each of the business applications is connected to the log collection channel corresponding to the log collection application; The data storage module comprises: A log file creation unit, used to control the log collection application to create a log file under the file system; A memory mapping unit, used for calling a preset memory mapping function to map the log file to the memory space of the log collection application; A log data writing memory unit is used to control the log collection application to write the received log data into the memory space; The data synchronization unit is used to synchronize and store all the log data in the memory space to the file system when the amount of log data in the memory space reaches a preset capacity threshold.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log processing method according to any one of claims 1 to 5 when executed.

Images