Data acquisition methods and apparatus, electronic devices and storage media

Abstract

This application provides a method, apparatus, electronic device, and storage medium for data acquisition. The method includes: acquiring associated data corresponding to the execution of the target operation when it is determined that a target object has performed a target operation; assembling the associated data within a target message entity; sending the target message entity to a message middleware via a target thread; and consuming the associated data within the message middleware via a consumption engine upon receiving an instruction to view the associated data, thereby obtaining the operation record corresponding to the target object. This application solves the problem in related technologies where timely problem tracing is not possible, leading to low efficiency in operation and maintenance troubleshooting.

Description

Technical Field

[0001] This application relates to the field of big data processing, and in particular to a method and apparatus for data acquisition, an electronic device, and a storage medium. Background Technology

[0002] The data development platform provides users with an integrated execution process for job development, saving, testing, running, submitting, and publishing. Current data development platforms support multi-user collaboration, which can lead to arbitrary operations by users. However, because current platforms do not support recording the entire process of user job execution, timely problem tracing is not possible when issues occur, resulting in low efficiency in maintenance and troubleshooting.

[0003] Therefore, when problems occur in the data development platform of related technologies, there is a problem that the source of the problem cannot be traced in a timely manner, resulting in low efficiency in operation and maintenance troubleshooting. Summary of the Invention

[0004] This application provides a method and apparatus for data acquisition, an electronic device, and a storage medium to at least solve the problem in related technologies where the inability to trace the source of problems in a timely manner leads to low efficiency in operation and maintenance troubleshooting.

[0005] According to one aspect of the embodiments of this application, a method for data acquisition is provided, the method comprising:

[0006] If it is determined that the target object has performed the target operation, obtain the associated data corresponding to the execution of the target operation;

[0007] The associated data is assembled into the target message entity;

[0008] The target message entity is sent to the message middleware via the target thread;

[0009] Upon receiving an instruction to view the associated data, the associated data within the message middleware is consumed through the consumption engine to obtain the operation record corresponding to the target object.

[0010] According to another aspect of the embodiments of this application, a data acquisition apparatus is also provided, the apparatus comprising:

[0011] The first acquisition unit is used to acquire the associated data corresponding to the execution of the target operation when it is determined that the target object has performed the target operation;

[0012] An assembly unit is used to assemble the associated data into a target message entity;

[0013] The sending unit is used to send the target message entity to the message middleware via the target thread;

[0014] The consumption unit is used to consume the associated data in the message middleware through the consumption engine when it receives an instruction to view the associated data, so as to obtain the operation record corresponding to the target object.

[0015] Optionally, the acquisition unit includes:

[0016] The first acquisition module is used to acquire operation request information sent by the target object through the target page;

[0017] The recording module is used to call the corresponding application programming interface according to the operation request information and record the target operation corresponding to the execution of the application programming interface.

[0018] The second acquisition module is used to acquire the associated data according to the target operation.

[0019] Optionally, the device further includes:

[0020] The second acquisition unit is used to acquire the operation record display page after the operation record corresponding to the target object is obtained;

[0021] The determining unit is used to determine the operation behavior of the target object by displaying the associated data on the operation record display page, wherein the operation behavior is used to characterize the behavior trajectory of the target object.

[0022] Optionally, the device further includes:

[0023] The establishment unit is used to establish a database management cluster before obtaining the operation record corresponding to the target object;

[0024] The storage unit is used to send the consumed associated data to the database management cluster for storage.

[0025] Optionally, the transmitting unit includes:

[0026] An acquisition module is used to acquire a preset threshold, wherein the preset threshold is used to indicate the maximum value that the associated data is allowed to be lost, and the maximum value is 0;

[0027] The first sending module is used to send the target message entity to the message middleware via an asynchronous thread when it is determined that the value at which the loss of the associated data is allowed is greater than the preset threshold.

[0028] Optionally, the transmitting unit includes:

[0029] The second sending module is used to send the target message entity to the message middleware via a synchronization thread when it is determined that the allowable value of the associated data loss is equal to the preset threshold.

[0030] Optionally, the associated data includes: operation time, operation type, the target object's account, and operation content.

[0031] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein the memory is used to store a computer program; and the processor is used to execute the method steps of any of the above embodiments by running the computer program stored in the memory.

[0032] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the method steps of any of the above embodiments when running.

[0033] In this embodiment, when it is determined that the target object has performed a target operation, the associated data corresponding to the target operation is obtained; the associated data is assembled into a target message entity; the target message entity is sent to the message middleware through the target thread; and when an instruction to view the associated data is received, the associated data in the message middleware is consumed through the consumption engine to obtain the operation record corresponding to the target object. Since this embodiment encapsulates the associated data corresponding to the target operation performed by the target object within the target message entity and sends it to the message middleware, the target operation is decoupled from the main process, allowing the target object to be unaware of the entire recording process. When it is necessary to view the target operation of the target object, it is only necessary to directly consume the associated data in the message middleware to obtain the entire process of the target object performing the job operation on the data development platform. This allows for timely error location, improves the work efficiency of operation and maintenance personnel in troubleshooting, and solves the problem in related technologies where data development platforms cannot be traced back to their source in a timely manner, resulting in low efficiency in operation and maintenance troubleshooting. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of the hardware environment for an optional data acquisition method according to an embodiment of the present invention;

[0037] Figure 2 This is a flowchart illustrating an optional data acquisition method according to an embodiment of this application;

[0038] Figure 3 This is a schematic diagram of the overall processing flow of an optional data acquisition method according to an embodiment of this application;

[0039] Figure 4 This is a structural block diagram of an optional data acquisition device according to an embodiment of this application;

[0040] Figure 5 This is a structural block diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0041] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0042] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] The data development platform provides users with an integrated execution workflow for job development, saving, testing, running, submitting, and publishing. Users can drag and drop plugins from the left-hand plugin list to create a job, configure dependency rules, set resource configurations, configure alarm rules, and schedule settings. Within the job, users process business data by writing code or using structured query language. After development, users need to save and submit the job to generate a new version. The versioned job workflow is then published to the test or production environment for periodic execution. However, currently, the data development platform does not support recording the entire process of user job execution. If problems occur on the platform, timely problem tracing is not possible, leading to inefficient troubleshooting and maintenance.

[0044] To address the aforementioned problems, according to one aspect of an embodiment of this application, a data acquisition method is provided. Optionally, in this embodiment, the above-described data acquisition method can be applied to, for example... Figure 1 In the hardware environment shown. For example... Figure 1 As shown, terminal 102 may include memory 104, processor 106, and display 108 (optional component). Terminal 102 can communicate with server 112 via network 110. Server 112 can provide services (such as application services) to the terminal or clients installed on the terminal. Database 114 can be set up on or independently of server 112 to provide data storage services to server 112. In addition, server 112 may run a processing engine 116, which can be used to execute the steps performed by server 112.

[0045] Optionally, terminal 102 may be, but is not limited to, a terminal capable of computing data, such as a mobile terminal (e.g., mobile phone, tablet computer), laptop computer, PC (Personal Computer), etc. The aforementioned network may include, but is not limited to, a wireless network or a wired network. The wireless network includes Bluetooth, Wi-Fi (Wireless Fidelity), and other networks that enable wireless communication. The aforementioned wired network may include, but is not limited to, a wide area network (WAN), a metropolitan area network (MAN), and a local area network (LAN). The aforementioned server 112 may include, but is not limited to, any hardware device capable of computing.

[0046] Furthermore, in this embodiment, the data acquisition method described above can also be applied to, but is not limited to, a powerful independent processing device without requiring data interaction. For example, the processing device can be, but is not limited to, a powerful terminal device; that is, the various operations in the data acquisition method described above can be integrated into a single independent processing device. The above is merely an example, and no limitation is made in this embodiment.

[0047] Optionally, in this embodiment, the above-described data acquisition method can be executed by server 112, by terminal 102, or by both server 112 and terminal 102. The data acquisition method of this embodiment can also be executed by a client installed on terminal 102.

[0048] Taking the server running on the data development platform as an example, Figure 2 This is a flowchart illustrating an optional data acquisition method according to an embodiment of this application, such as... Figure 2 As shown, the process of this method may include the following steps:

[0049] Step S201: If it is determined that the target object has performed the target operation, obtain the associated data corresponding to the execution of the target operation.

[0050] Optionally, in this embodiment, each time the target object sends a request to the server, the server can detect that the target object has performed a target operation on the data development platform. At this time, the data development platform performs a tracking operation in the service layer code to obtain all the associated data corresponding to the target operation and generate a message accordingly. Here, the object refers to the user, and in subsequent embodiments, the meaning of "receive" also refers to the user. For example, the target object is the user currently performing the target operation.

[0051] The associated data may include the operation time, operation type, operation content, and the current account of the target object performing the target operation. The target operation may be a click, drag, etc. Data tracking refers to the related technologies and implementation processes for capturing, processing, and sending specific user behaviors or events. In this embodiment, it refers to the process of capturing the behaviors performed by the target object and the events triggered.

[0052] Step S202: Assemble the associated data within the target message entity.

[0053] Optionally, in this embodiment of the application, the acquired associated data is assembled or encapsulated into a target message entity, wherein the target message entity is a message entity object, the function of which is to treat the associated data as a whole object for subsequent operations.

[0054] Step S203: Send the target message entity to the message middleware via the target thread.

[0055] Optionally, in this embodiment, the target thread is used to send the target message entity to the message middleware, so that the associated data is not directly stored but written into the message middleware, which can improve the overall system performance of the data development platform. The message middleware is a middleware technology composed of message passing mechanisms or message queue patterns. It utilizes efficient and reliable message passing mechanisms for platform-independent data exchange and integrates distributed systems based on data communication. The target thread can be understood as an actual unit of operation within a process; the target thread enables the sending of the target message entity.

[0056] Step S204: Upon receiving an instruction to view associated data, the associated data within the message middleware is consumed through the consumption engine to obtain the operation record corresponding to the target object.

[0057] Optionally, when the server of the data development platform of this application needs to view the above-mentioned related data, it can start a consumer group (such as a consumer engine) to consume the related data in the message middleware, and then obtain all the operation records corresponding to the target object. The operation records are consistent with the related data and also include the operation time, operation type, operation content, etc. of the target object performing the target operation.

[0058] In this embodiment, when it is determined that the target object has performed a target operation, the associated data corresponding to the target operation is obtained; the associated data is assembled into a target message entity; the target message entity is sent to the message middleware through the target thread; and when an instruction to view the associated data is received, the associated data in the message middleware is consumed through the consumption engine to obtain the operation record corresponding to the target object. Since this embodiment encapsulates the associated data corresponding to the target operation performed by the target object within the target message entity and sends it to the message middleware, the target operation is decoupled from the main process, allowing the target object to be unaware of the entire recording process. When it is necessary to view the target operation of the target object, it is only necessary to directly consume the associated data in the message middleware to obtain the entire process of the target object performing the job operation on the data development platform. This allows for timely error location, improves the work efficiency of operation and maintenance personnel in troubleshooting, and solves the problem in related technologies where data development platforms cannot be traced back to their source in a timely manner, resulting in low efficiency in operation and maintenance troubleshooting.

[0059] As an optional embodiment, when it is determined that the target object has performed the target operation, obtaining the associated data corresponding to the execution of the target operation includes:

[0060] Obtain the operation request information sent by the target object through the target page;

[0061] Invoke the corresponding application programming interface (API) based on the operation request information, and record the target operation corresponding to the execution of the API.

[0062] Retrieve related data based on the target operation.

[0063] Optionally, the operation request information initiated by the target object to the data development platform through the target page on the front end is obtained. The operation request information may include saving, running, testing, submitting, publishing, etc. Then, the application interface of the data development platform called according to the operation request information is obtained, and the target operation corresponding to the front end calling the application interface is recorded. Based on the target operation, all related data can be obtained.

[0064] In this embodiment of the application, by recording the target operation by recording the front-end calling the application interface, all data of the target object performing the target operation can be obtained more quickly.

[0065] As an optional embodiment, after obtaining the operation record corresponding to the target object, the method further includes:

[0066] Get the operation record display page;

[0067] The operation log display page shows related data to determine the operation behavior of the target object, where the operation behavior is used to characterize the behavioral trajectory of the target object.

[0068] Optionally, the data development platform includes an operation log page, or operation log display page, which shows all related data. Based on this displayed data, the target object's operational behavior can be determined. Besides the target object itself, other objects can also view the target object's or other objects' operational behavior through the operation log display page. Operational behavior refers to the target object's behavioral trajectory. Based on this trajectory, the target object's behavioral habits and operational tendencies can be determined, and further, the initial and final operations performed by the target object when executing the target operation can be identified.

[0069] In this embodiment of the application, the operation records of a specific object can be easily viewed through the operation record display page, which enables faster problem tracing and improves operation and maintenance efficiency when a problem occurs.

[0070] As an optional embodiment, before obtaining the operation record corresponding to the target object, the method further includes:

[0071] Establish a database management cluster;

[0072] The associated data after consumption is sent to the database management cluster for storage.

[0073] Optionally, this application embodiment establishes a database management cluster, which can be an ES (Elasticsearch) cluster or a MySQL database management cluster, etc., and sends the associated data consumed by the consumption engine to any database management cluster for storage.

[0074] In this embodiment of the application, the associated data after consumption is written into the database management cluster to facilitate the search and viewing of the associated data.

[0075] As an optional embodiment, sending the target message entity to the message middleware via the target thread includes:

[0076] Obtain a preset threshold, where the preset threshold is used to indicate the maximum value that associated data can be lost, and the maximum value is 0;

[0077] If the value at which the loss of associated data is allowed is determined to be greater than a preset threshold, the target message entity is sent to the message middleware via an asynchronous thread; or,

[0078] If the allowed value of associated data loss is equal to a preset threshold, the target message entity is sent to the message middleware via a synchronization thread.

[0079] Optionally, in this embodiment, when sending the target message entity using the target thread, the type of thread can be selected according to actual needs, such as whether to send it asynchronously or synchronously. More specifically, a preset threshold is first set, where the preset threshold is the maximum value that allows for data loss of associated data during transmission, and the value of this maximum value is set to 0.

[0080] If the value at which the loss of associated data is allowed is greater than the preset threshold, it indicates that the current scenario allows for the loss of associated data. In this case, an asynchronous thread can be used to send the target message entity to the message middleware. If the value at which the loss of associated data is allowed is equal to the preset threshold, it indicates that the current scenario does not allow for the loss of associated data. In this case, a synchronous thread can be used to send the target message entity to the message middleware.

[0081] In this embodiment, the thread type of the target thread is determined based on the number of associated data that can be lost during transmission, thereby improving transmission efficiency while meeting the requirements of the scenario.

[0082] As an alternative embodiment, such as Figure 3As shown in the diagram, the source data used when performing a target operation on the data development platform with audit data as the target object is the data itself. More specifically, it obtains front-end UI operations, which call the API interface of the data development platform. Then, the data after the operation is assembled through the audit data model and sent to the RocketMQ message middleware. Subsequently, consumers 1 and 2 call the poll() function to consume the data in the RocketMQ message middleware and send the consumed data to Elasticsearch (ES). Users can also view the data stored in ES through the data development platform. Additionally, the data development platform has an operation log page where users can query ES via the API interface to obtain a series of historical operation behaviors on the audit data.

[0083] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0085] According to another aspect of the embodiments of this application, an apparatus for implementing the above-described data acquisition method is also provided. Figure 4 This is a structural block diagram of an optional data acquisition device according to an embodiment of this application, such as... Figure 4 As shown, the device may include:

[0086] The first acquisition unit 401 is used to acquire the associated data corresponding to the execution of the target operation when it is determined that the target object has performed the target operation.

[0087] Assembly unit 402, connected to first acquisition unit 401, is used to assemble associated data into target message entity;

[0088] The sending unit 403, connected to the assembly unit 402, is used to send the target message entity to the message middleware through the target thread;

[0089] The consumer unit 404, connected to the sending unit 403, is used to consume the associated data in the message middleware through the consumer engine when it receives an instruction to view associated data, and obtain the operation record corresponding to the target object.

[0090] It should be noted that the first acquisition unit 401 in this embodiment can be used to perform the above step S201, the assembly unit 402 in this embodiment can be used to perform the above step S202, the sending unit 403 in this embodiment can be used to perform the above step S203, and the consumption unit 404 in this embodiment can be used to perform the above step S204.

[0091] Through the above modules, since the associated data corresponding to the target operation performed by the target object is encapsulated in the target message entity and sent to the message middleware in this embodiment of the application, the target operation is decoupled from the main process, so that the target object is unaware of the entire recording process. When it is necessary to view the target operation of the target object, it is only necessary to directly consume the associated data in the message middleware to obtain the entire process of the target object performing the job operation on the data development platform, so as to locate errors in time, improve the work efficiency of operation and maintenance personnel in troubleshooting problems, and solve the problem in related technologies that when problems occur in the data development platform, the source of the problem cannot be traced in time, resulting in low efficiency of operation and maintenance troubleshooting.

[0092] As an optional embodiment, the acquisition unit includes:

[0093] The first acquisition module is used to acquire operation request information sent by the target object through the target page;

[0094] The recording module is used to call the corresponding application programming interface based on the operation request information and record the target operation corresponding to the execution of the application programming interface.

[0095] The second acquisition module is used to acquire related data based on the target operation.

[0096] As an optional embodiment, the device further includes:

[0097] The second acquisition unit is used to acquire the operation record display page after obtaining the operation record corresponding to the target object;

[0098] The determination unit is used to display related data on the operation record display page and determine the operation behavior of the target object, wherein the operation behavior is used to characterize the behavioral trajectory of the target object.

[0099] As an optional embodiment, the device further includes:

[0100] Establishment unit, used to establish a database management cluster before obtaining the operation record corresponding to the target object;

[0101] The storage unit is used to send the associated data after consumption to the database management cluster for storage.

[0102] As an optional embodiment, the transmitting unit includes:

[0103] The acquisition module is used to acquire a preset threshold, wherein the preset threshold is used to indicate the maximum value that associated data can be lost, and the maximum value is 0;

[0104] The first sending module is used to send the target message entity to the message middleware via an asynchronous thread when the value at which the loss of associated data is allowed is greater than a preset threshold.

[0105] As an optional embodiment, the transmitting unit includes:

[0106] The second sending module is used to send the target message entity to the message middleware via a synchronous thread when it is determined that the allowable value of associated data loss is equal to a preset threshold.

[0107] As an optional implementation, the associated data includes: operation time, operation type, target account, and operation content.

[0108] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1 The hardware environment shown can be implemented through software or hardware, and the hardware environment includes the network environment.

[0109] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described data acquisition method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0110] Figure 5 This is a structural block diagram of an optional electronic device according to an embodiment of this application, such as... Figure 5As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504. The processor 501, communication interface 502, and memory 503 communicate with each other via the communication bus 504.

[0111] Memory 503 is used to store computer programs;

[0112] When processor 501 executes a computer program stored in memory 503, it performs the following steps:

[0113] Once it is determined that the target object has performed the target operation, obtain the associated data corresponding to the execution of the target operation;

[0114] Assemble the associated data within the target message entity;

[0115] The target message entity is sent to the message middleware via the target thread;

[0116] When an instruction to view related data is received, the related data within the message middleware is consumed through the consumption engine to obtain the operation record corresponding to the target object.

[0117] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0118] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0119] The memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0120] As an example, such as Figure 5 As shown, the memory 503 may include, but is not limited to, the first acquisition unit 401, assembly unit 402, transmission unit 403, and consumption unit 404 in the data acquisition device. Furthermore, it may include, but is not limited to, other module units in the data acquisition device, which will not be elaborated upon in this example.

[0121] The processors mentioned above can be general-purpose processors, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; they can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0122] In addition, the aforementioned electronic device also includes a display for displaying the results of data acquisition.

[0123] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0124] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. The device that implements the above data acquisition method can be a terminal device, such as a smartphone (e.g., Android phone, iOS phone), tablet computer, PDA, mobile Internet Device (MID), PAD, etc. Figure 5 This does not limit the structure of the aforementioned electronic devices. For example, the terminal device may also include components that are more advanced than those described above. Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0125] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0126] According to another aspect of the embodiments of this application, a storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to execute program code for a data acquisition method.

[0127] Optionally, in this embodiment, the storage medium may be located on at least one of the network devices in the network shown in the above embodiment.

[0128] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

[0129] Once it is determined that the target object has performed the target operation, obtain the associated data corresponding to the execution of the target operation;

[0130] Assemble the associated data within the target message entity;

[0131] The target message entity is sent to the message middleware via the target thread;

[0132] When an instruction to view related data is received, the related data within the message middleware is consumed through the consumption engine to obtain the operation record corresponding to the target object.

[0133] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated in this embodiment.

[0134] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0135] According to another aspect of the embodiments of this application, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the data acquisition method steps in any of the above embodiments.

[0136] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0137] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the data acquisition methods of the various embodiments of this application.

[0138] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0140] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the solution provided in this embodiment, depending on actual needs.

[0141] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0142] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method of data acquisition, characterized by, The method comprises: In the case where it is determined that the target object has performed a target operation, the associated data corresponding to the performance of the target operation is acquired, comprising: acquiring operation request information issued by the target object through a target page, wherein the operation request information is used to request the performance of an operation on a data development platform, and the operation request information comprises saving, running, testing, submitting and publishing; calling a corresponding application program interface according to the operation request information, recording the target operation corresponding to the performance of the application program interface; and acquiring the associated data according to the target operation; The associated data is assembled in a target message entity; The target message entity is sent to a message middleware through a target thread, comprising: acquiring a preset threshold, wherein the preset threshold is used to indicate a maximum value of the allowed loss of the associated data, and the value of the maximum value is 0; in the case where it is determined that the value of the allowed loss of the associated data is greater than the preset threshold, the target message entity is sent to the message middleware through an asynchronous thread, wherein the value of the allowed loss of the associated data greater than the preset threshold indicates that the loss of the associated data is allowed; or in the case where it is determined that the value of the allowed loss of the associated data is equal to the preset threshold, the target message entity is sent to the message middleware through a synchronous thread, wherein the value of the allowed loss of the associated data equal to the preset threshold indicates that the loss of the associated data is not allowed; When an instruction to view the associated data is received, the associated data in the message middleware is consumed through a consumption engine to obtain operation records corresponding to the target object.

2. The method of claim 1, wherein, After the operation records corresponding to the target object are obtained, the method further comprises: An operation record display page is acquired; The associated data is displayed by using the operation record display page to determine the operation behavior of the target object, wherein the operation behavior is used to represent the behavior track of the target object.

3. The method of claim 1, wherein, Before the operation records corresponding to the target object are obtained, the method further comprises: A database management cluster is established; The consumed associated data is sent to the database management cluster for storage.

4. The method according to any one of claims 1 to 3, characterized in that, The associated data comprises operation time, operation type, the account of the target object and operation content.

5. An apparatus for data acquisition, characterized by The device comprises: A first acquisition unit is configured to acquire, in the case where it is determined that a target object has performed a target operation, associated data corresponding to the performance of the target operation, comprising: acquiring operation request information issued by the target object through a target page, wherein the operation request information is used to request the performance of an operation on a data development platform, and the operation request information comprises saving, running, testing, submitting and publishing; calling a corresponding application program interface according to the operation request information, recording the target operation corresponding to the performance of the application program interface; and acquiring the associated data according to the target operation; An assembly unit is configured to assemble the associated data in a target message entity; The sending unit is configured to send the target message entity to the message middleware through a target thread, including: obtaining a preset threshold, wherein the preset threshold is used to indicate a maximum value of allowed loss of the associated data, and the value of the maximum value is 0; in a case where it is determined that a value of allowed loss of the associated data is greater than the preset threshold, the target message entity is sent to the message middleware through an asynchronous thread, wherein the value of allowed loss of the associated data greater than the preset threshold indicates that the loss of the associated data is allowed; or in a case where it is determined that the value of allowed loss of the associated data is equal to the preset threshold, the target message entity is sent to the message middleware through a synchronous thread, wherein the value of allowed loss of the associated data equal to the preset threshold indicates that the loss of the associated data is not allowed; The consuming unit is configured to, when receiving an instruction of viewing the associated data, consume the associated data in the message middleware through a consuming engine to obtain an operation record corresponding to the target object.

6. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein, The processor, the communication interface and the memory complete mutual communication through the communication bus, and the processor is configured to execute the method steps in any one of claims 1 to 4 by running the computer program stored on the memory. The memory is configured to store a computer program. The processor is configured to execute the method steps in any one of claims 1 to 4 by running the computer program stored on the memory.

7. A computer readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is configured to execute the method steps in any one of claims 1 to 4 when running.

Images