Data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data processing method and device, electronic equipment and storage medium, relates to the technical field of data processing, in particular to the technical field of cloud computing and big data, comprising: creating a first conversion task for each data source type; obtaining a plurality of logs of all products, classifying all the obtained logs according to data source types to obtain a plurality of log sets, and the first conversion task is used for extracting data of a set type from the log set of the corresponding data source type for data conversion; according to the data source type, performing the corresponding first conversion task on the log set to obtain a first data table, and the first data table is used for storing all data after data conversion.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to the fields of cloud computing and big data technology. Background Technology

[0002] Currently, when processing data in a data warehouse, all log data is first merged into a large data table, a corresponding transformation task is generated for the product, and data extraction and transformation are performed on the large data table. Summary of the Invention

[0003] This disclosure provides a data processing method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, a data processing method is provided, comprising:

[0005] Create a first transformation task for each data source type;

[0006] The system retrieves multiple logs from all products, categorizes all logs according to data source type, and obtains multiple log sets. The first transformation task is used to extract data of a set type from the log set of the corresponding data source type and perform data transformation.

[0007] Based on the data source type, a corresponding first transformation task is performed on the log set to obtain a first data table, which is used to store all the transformed data.

[0008] According to another aspect of this disclosure, a data processing apparatus is provided, comprising:

[0009] The processing module is used to create a first transformation task for each data source type;

[0010] The acquisition module is used to acquire multiple logs from all products, classify all acquired logs according to data source type, and obtain multiple log sets. The first conversion task is used to extract data of a set type from the log set of the corresponding data source type and perform data conversion.

[0011] The transformation module is used to perform a first transformation task on the log set according to the data source type to obtain a first data table, which is used to store all the data after transformation.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described above.

[0016] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in any of the preceding claims.

[0017] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a schematic flowchart of a data processing method provided according to an embodiment of the present disclosure;

[0021] Figure 2 This is a flowchart illustrating another data processing method provided according to an embodiment of the present disclosure;

[0022] Figure 3 This is a flowchart illustrating another data processing method provided according to an embodiment of the present disclosure;

[0023] Figure 4 This is a flowchart illustrating a data bad block processing method provided according to an embodiment of this disclosure;

[0024] Figure 5 This is a flowchart illustrating another data bad block processing method provided according to an embodiment of this disclosure;

[0025] Figure 6 This is a schematic diagram of the structure of a data warehouse provided according to an embodiment of this disclosure;

[0026] Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided according to an embodiment of the present disclosure;

[0027] Figure 8 This is a block diagram of an electronic device used to implement the data processing method of the embodiments of this disclosure. Detailed Implementation

[0028] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] Currently, when processing data in a data warehouse, all source log data is first merged into a large data table, and then corresponding transformation tasks are generated to transform the data in the large table. However, when products are added or removed, the number and configuration of the generated transformation tasks need to be modified, resulting in excessively high system maintenance costs.

[0030] To reduce system maintenance costs, such as Figure 1 As shown, one embodiment of this disclosure provides a data processing method, the method comprising:

[0031] Step 101: Create a first transformation task for each data source type.

[0032] A first transformation task is created for each data source type. The first transformation task is an ETL (Extract-Transform-Load) task, which is used to extract, transform, and load data from the logs. In this embodiment, the data source type is the source of the logs, such as the client, front-end, and back-end. In other embodiments, the data source type can also be the type of device, such as mobile terminal, desktop computer terminal, and server terminal.

[0033] For example, if a user has multiple products, and all products are integrated with the client and front-end, then two first conversion tasks need to be created to handle the client's logs and the front-end's logs respectively.

[0034] Step 102: Obtain multiple logs from all products, classify all obtained logs according to data source type to obtain multiple log sets, and the first transformation task is used to extract data of a set type from the log set of the corresponding data source type for data transformation.

[0035] Retrieve multiple logs from all products and categorize all logs according to data source type;

[0036] For example, if a user has multiple products, and all products are connected to both the client and the front-end, then we can obtain multiple logs from all products and categorize these logs according to the client and the front-end to obtain the client log set and the front-end log set.

[0037] The first transformation task is used to extract data of a specified type from the log set of the corresponding data source type and perform data transformation.

[0038] Each log entry contains multiple types of data, such as business data, user behavior data, and error data. The data type must include at least business data, but can also include other types of data, depending on the specific requirements.

[0039] Step 103: Based on the data source type, perform the corresponding first transformation task on the log set to obtain a first data table, which is used to store all the data after data transformation.

[0040] The first transformation task is used to extract and transform data from the log set of the corresponding data source type to obtain the first data table corresponding to the data source type.

[0041] For example, the first transformation task for processing client logs is used to extract and transform data from the client log set to obtain the first data table corresponding to the client. Similarly, the first transformation task for processing front-end log sets is used to extract and transform data from the front-end logs to obtain the first data table corresponding to the front-end.

[0042] The first data table contains all the converted data and adds a product type field. When using the data in the first data table, you can use this field to determine which product's log the data belongs to.

[0043] After generating the first data table, store the first data table.

[0044] In the data processing method disclosed herein, logs are categorized according to data source type. After categorization, each data source type has a corresponding log set. Due to the configuration of the first transformation task, the first data table after data transformation contains a product type field. When using the first data table, the data can be distinguished from which product it belongs based on the product type field in the first data table. Therefore, the first transformation task can be created based on the number of data source types, instead of creating transformation tasks based on the number of products. The number of products will frequently increase and decrease, while the data source type is fixed and will not change. Therefore, regardless of whether the number of products increases or decreases, there is no need to modify the number and configuration of transformation tasks, which greatly reduces the cost of system maintenance.

[0045] like Figure 2 As shown, one embodiment of this disclosure provides another data processing method, the method comprising:

[0046] Step 201: Create a second transformation task for each data source type. The second transformation task is used to extract user behavior data from the log set of the corresponding data source type and perform data transformation.

[0047] Create a second transformation task for each data source type. The second transformation task is an ETL task. Unlike the configuration of the first transformation task, the second transformation task extracts user behavior data from the log set of the corresponding data source type and performs data transformation.

[0048] Step 202: Based on the data source type, perform the corresponding second transformation task on the log set to obtain a second data table, which is used to store the transformed user behavior data.

[0049] The second transformation task is used to extract and transform user behavior data from logs of the corresponding data source type, resulting in a second data table corresponding to the data source type. The second data table is used to store the transformed user behavior data.

[0050] For example, the second transformation task for processing client logs is used to extract and transform data from the client log set to obtain the second data table corresponding to the client. Similarly, the second transformation task for processing front-end logs is used to extract and transform data from the front-end log set to obtain the second data table corresponding to the front-end.

[0051] Step 203: Perform user behavior detection based on the second data table of the log set to obtain specific types of user behavior data.

[0052] User behavior detection is performed on the second data table to obtain specific types of user behavior data. These specific types of user behavior data can be at least one of the following: user cheating behavior data, consumption behavior data, operation behavior data, etc.

[0053] Step 204: Based on the data identifier of the specific type of user behavior data in the second data table of the log set, mark the corresponding data in the first data table of the log set.

[0054] Based on the data identifier in the second data table of the log set, the corresponding data in the first data table of the log set is marked according to the specific type of user behavior data;

[0055] For example, if a specific type of user behavior data is cheating behavior data, and a certain piece of data in the cheating behavior data is {"Data Identifier": 0011, "Cheating Behavior Type": "Too Many Requests"}, then based on this piece of data, the data in the first data table that also has a data identifier of 0011 will be added with a cheating behavior identifier.

[0056] By performing user behavior detection on the data in the second data table, specific types of user behavior data are obtained. Then, this specific type of user behavior data is merged with the first data table according to the data identifier. This can effectively mark some cheating data in the first data table, ensuring data security and reducing the risk to users. Alternatively, it can obtain users' consumption behavior data so that users can understand their own consumption habits and improve the user experience.

[0057] like Figure 3 As shown, one embodiment of this disclosure provides another data processing method, the method comprising:

[0058] Step 301: Execute the first transformation task corresponding to each log set in parallel.

[0059] Step 302: Execute the second transformation task corresponding to each log set in parallel.

[0060] Currently, when processing data, because the transformation tasks are created one-to-one with the products, and there is only one large data table to be processed, multiple transformation tasks can only extract and transform data from one large data table. When the previous transformation task is extracting and transforming data, the next transformation task cannot be performed simultaneously. It can only continue after the previous transformation task is completed, resulting in very low efficiency in processing data.

[0061] In the data processing method provided in this disclosure, when there are multiple first transformation tasks or second transformation tasks, when data extraction and data transformation are performed on log sets of the same data source type using the first transformation task or second transformation task corresponding to the data source type, since each first transformation task or second transformation task has a corresponding log set of the data source type, multiple first transformation tasks or second transformation tasks can be executed in parallel, which can significantly improve the efficiency of data processing.

[0062] like Figure 4 As shown, one embodiment of this disclosure provides a method for handling bad data blocks, the method comprising:

[0063] After obtaining the first data table, step 401 involves performing a bad block detection on the first data table.

[0064] To perform bad block detection on the first data table, any detection method capable of detecting bad blocks can be used.

[0065] Step 402: In response to the presence of bad data blocks in the first data table, perform a first transformation task on the log set corresponding to the first data table to obtain a new first data table.

[0066] The first transformation task, which uses the same data source type as the log set corresponding to the first data table containing bad data blocks, re-extracts and transforms the data in the log set to obtain a new first data table.

[0067] After generating the first data table, a bad block detection process is performed. If a bad block is detected, it is highly likely that the bad block was caused during the data transformation process. In this case, the log set to which the bad block belongs needs to be re-extracted and transformed to generate a new first data table. This allows the system to detect bad blocks and re-extract and transform the data immediately, without having to wait until the first data table containing bad blocks is used. This ensures the timeliness of the data, improves the availability of the first data table, and enhances the user experience.

[0068] like Figure 5 As shown, one embodiment of this disclosure provides another method for handling bad data blocks, the method comprising:

[0069] After obtaining the first data table, step 501 involves receiving a bad block request, which includes the bad block address and log priority.

[0070] After receiving a user's query request, the third-party application will retrieve the data required by the user from the first data table stored on the system. Data with bad blocks will be discovered when the query is performed. The third-party application will then obtain the bad block address and log priority of the data and send a bad block request to the system. The bad block address can indicate which part of the log set the bad block belongs to. The log priority is a pre-marked log that can indicate the importance of the log.

[0071] Step 502: In response to the log priority being greater than or equal to a preset priority threshold, a portion of the data in the log set is determined based on the bad block address.

[0072] If the log priority is greater than or equal to the preset priority threshold, the bad block will be determined based on its address, and if there are multiple logs, the log priorities of all of them will be considered.

[0073] In this embodiment, the log set can be divided into multiple parts for processing. In this case, the bad block address indicates which part of the log set the bad block belongs to. The bad block is determined to be in the log set based on the bad block address.

[0074] Step 503: Based on the determined data source type of the partial data in the log set, perform the corresponding first transformation task on the partial data in the log set.

[0075] The first transformation task, which uses the same data source type as the log set to which the bad data block belongs, re-extracts and transforms the data in the log set containing the bad data block.

[0076] Step 504: Update the corresponding data in the first data table based on the data obtained from performing the first conversion task.

[0077] Based on the data obtained after re-extraction and transformation, and the data identifiers of these data, update the data with the same data identifiers in the original first data table.

[0078] When a third-party application queries the first data table, it may find bad data blocks. In this case, it sends a bad data block request to the system, which then extracts and transforms the data corresponding to the bad data block in the log. The transformed data is then used to update the data in the first data table that have the same data identifier, improving system stability. This ensures that the data containing bad data blocks is extracted and transformed before being sent to the user, guaranteeing data availability and improving the user experience.

[0079] In one possible implementation, historical data is collected before acquiring multiple logs of the product.

[0080] Historical data includes the amount of data processed in each time period, the number of queries made by third-party applications to the first data table stored in the system, and other historical data.

[0081] The transmission speed threshold is obtained by using the model to predict the historical request data.

[0082] The model is used to predict historical request data to obtain transmission speed thresholds. These thresholds include the transmission speeds that should be set for different time periods, such as the transmission speeds on holidays and weekdays.

[0083] Specifically, a linear regression model can be used to predict historical data to obtain a transmission speed threshold, and then a random forest model can be used to predict historical data to obtain another transmission speed threshold. The average of the transmission speed thresholds obtained from the two models is then taken to obtain the final transmission speed threshold. Other models can also be used for prediction, which can make full use of the predictive capabilities of multiple models.

[0084] The transmission speed threshold is used to set the transmission speed for acquiring multiple logs of the product and sending data to third-party applications.

[0085] Because the amount of data the system needs to process and the number of queries from third-party applications to the first data table stored in the system vary depending on the time period, the system may crash if the amount of data to be processed or the number of queries to the first data table stored in the system is too large during a certain period. By setting transmission speed thresholds for retrieving multiple logs from the product and sending data to third-party applications, the system can avoid processing too much data at the same time or limiting the transmission speed of data sent to third-party applications when the amount of data to be processed or the number of queries to the first data table stored in the system is very large. This significantly improves the stability of the system and reduces the risk of system crashes.

[0086] In one possible implementation, when performing data extraction and transformation on the logs using a first transformation task or a second transformation task, the method further includes:

[0087] If a log has a high priority, it will be assigned to a high-performance device for processing. A high priority log indicates that it is of great importance and has a strong time sensitivity. Assigning the log to a high-performance device for processing can significantly improve the efficiency of processing high-priority logs and further improve the user experience.

[0088] In one possible implementation, after obtaining the first data table, when the system receives a query request, the method further includes:

[0089] For query requests sent by third-party applications, determine which product the request belongs to. If the device of the product to which the request belongs is handling a large number of requests, determine whether there are other products with a smaller number of requests being handled in the account to which the product belongs. If so, then assign the user's request to the other product with a smaller number of requests being handled.

[0090] For example, if an account contains product A and product B, and a user request for product A is received, and the device for product A is currently processing a large number of requests while the device for product B is currently processing a small number of requests, then the user request for product A will be assigned to the device for product B for processing.

[0091] By distributing requests from products belonging to devices with high data processing volumes to other products with lower data processing volumes, a scaling mechanism for idle time reuse is established. This prevents the allocation of tasks to devices with high data processing volumes, avoiding the risk of device crashes and improving system stability. On the other hand, distributing requests to devices with lower data processing volumes reduces the idle time of those devices, significantly improving system utilization.

[0092] In one possible implementation, the data processing method provided in this disclosure can be applied to a data warehouse, such as... Figure 6 As shown, the data warehouse includes a log layer, a data layer, and a database;

[0093] Create a first transformation task in advance for each data source type at the data layer.

[0094] After the first transformation task is created, the log layer is used to obtain multiple logs from the product, and all the obtained logs are classified according to the data source type to obtain multiple log sets.

[0095] Then, the data layer performs the corresponding first transformation task on the log set according to the data source type to obtain the first data table.

[0096] After obtaining the first data table, store it in the database.

[0097] One embodiment of this disclosure provides a data processing apparatus, such as... Figure 7 As shown, the device includes:

[0098] Processing module 10 is used to create a first transformation task for each data source type;

[0099] The acquisition module 20 is used to acquire multiple logs from all products, classify all acquired logs according to data source type, and obtain multiple log sets. The first conversion task is used to extract data of a set type from the log set of the corresponding data source type and perform data conversion.

[0100] The conversion module 30 is used to perform a first conversion task on the log set according to the data source type to obtain a first data table, which is used to store all the data after data conversion.

[0101] The processing module 10 is further configured to create a second transformation task for each data source type, wherein the second transformation task is used to extract user behavior data from the log set of the corresponding data source type for data transformation.

[0102] The conversion module 30 is further configured to perform a corresponding second conversion task on the log set according to the data source type to obtain a second data table, the second data table being used to store user behavior data after data conversion;

[0103] The processing module 10 is also used to perform user behavior detection based on the second data table of the log set to obtain user behavior data of a specific type;

[0104] The processing module 10 is further configured to mark the corresponding data in the first data table of the log set according to the data identifier of the specific type of user behavior data in the second data table of the log set.

[0105] The conversion module 30 is also used to execute the first conversion task corresponding to each log set in parallel;

[0106] The conversion module 30 is also used to execute the second conversion task corresponding to each log set in parallel.

[0107] The processing module 10 is further configured to perform bad block detection on the first data table;

[0108] The conversion module 30 is further configured to, in response to the presence of bad data blocks in the first data table, perform a first conversion task on the log set corresponding to the first data table to obtain a new first data table.

[0109] The processing module 10 is further configured to receive a bad block request, wherein the bad block request includes the bad block address and log priority;

[0110] The processing module 10 is further configured to, in response to the log priority being greater than or equal to a preset priority threshold, determine a portion of the data in the log set based on the bad block address;

[0111] The conversion module 30 is further configured to perform a corresponding first conversion task on a portion of the data in the log set according to the determined data source type of the portion of the data in the log set;

[0112] The processing module 10 is further configured to update the corresponding data in the first data table based on the data obtained from performing the first conversion task.

[0113] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0114] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0115] Figure 8 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0116] like Figure 8 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0117] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0118] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0119] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0121] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0123] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0124] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0125] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0126] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, comprising: Create a first transformation task for each data source type; The system retrieves multiple logs from all products, categorizes all logs according to data source type, and obtains multiple log sets. The first transformation task is used to extract data of a set type from the log set of the corresponding data source type and perform data transformation. According to the data source type, the corresponding first transformation task is performed on the log set to obtain a first data table, which is used to store all the data after data transformation. Receive a bad block request, which includes the bad block address and log priority. The bad block request is sent by a third-party application when it finds bad blocks in the first data table. In response to the log priority being greater than or equal to a preset priority threshold, a portion of the data in the log set is determined based on the bad block address. Based on the determined data source type of a portion of the data in the log set, perform a corresponding first transformation task on the portion of the data in the log set; Based on the data obtained from performing the first transformation task, update the corresponding data in the first data table; This includes, before retrieving multiple logs from the product, the following: Collect historical data, which includes the amount of data processed in each time period and / or the number of queries to the first data table; The historical data is used to predict the transmission speed threshold using a model; The transmission speed threshold is used to set the transmission speed for acquiring multiple logs of the product and sending data to third-party applications.

2. The method according to claim 1, further comprising: A second transformation task is created for each data source type. The second transformation task is used to extract user behavior data from the log set of the corresponding data source type and perform data transformation. Based on the data source type, a corresponding second transformation task is performed on the log set to obtain a second data table, which is used to store user behavior data after data transformation. User behavior detection is performed based on the second data table of the log set to obtain specific types of user behavior data; Based on the data identifier in the second data table of the log set for the specific type of user behavior data, the corresponding data in the first data table of the log set is marked.

3. The method according to claim 2, further comprising: The first transformation task corresponding to each log set is executed in parallel. The second transformation task corresponding to each log set is executed in parallel.

4. The method according to claim 1, further comprising, after obtaining the first data table: Perform bad block detection on the first data table; In response to the presence of bad data blocks in the first data table, a first transformation task is performed on the log set corresponding to the first data table to obtain a new first data table.

5. A data processing apparatus, comprising: The processing module is used to create a first transformation task for each data source type; The acquisition module is used to acquire multiple logs from all products, classify all acquired logs according to data source type, and obtain multiple log sets. The first conversion task is used to extract data of a set type from the log set of the corresponding data source type and perform data conversion. The conversion module is used to perform a first conversion task on the log set according to the data source type to obtain a first data table, which is used to store all the data after data conversion. The processing module is also used to receive a bad block request, which includes the bad block address and log priority. The processing module is also configured to, in response to the log priority being greater than or equal to a preset priority threshold, determine a portion of the data in the log set based on the bad block address; The conversion module is further configured to perform a corresponding first conversion task on a portion of the data in the log set according to the determined data source type of the portion of the data in the log set; The processing module is further configured to update the corresponding data in the first data table based on the data obtained from performing the first conversion task; This includes, before retrieving multiple logs from the product, the following: Collect historical data; The transmission speed threshold is obtained by using the model to predict historical request data; The transmission speed threshold is used to set the transmission speed for acquiring multiple logs of the product and sending data to third-party applications.

6. The apparatus according to claim 5, comprising: The processing module is also used to create a second transformation task for each data source type, the second transformation task being used to extract user behavior data from the log set of the corresponding data source type for data transformation; The conversion module is also used to perform a corresponding second conversion task on the log set according to the data source type to obtain a second data table, which is used to store user behavior data after data conversion. The processing module is also used to perform user behavior detection based on the second data table of the log set to obtain specific types of user behavior data; The processing module is also used to mark the corresponding data in the first data table of the log set according to the data identifier of the specific type of user behavior data in the second data table of the log set.

7. The apparatus according to claim 5, comprising: The conversion module is also used to execute the first conversion task corresponding to each log set in parallel; The conversion module is also used to execute the second conversion task corresponding to each log set in parallel.

8. The apparatus according to claim 5, comprising: The processing module is also used to perform bad block detection on the first data table; The conversion module is further configured to, in response to the presence of bad data blocks in the first data table, perform a first conversion task on the log set corresponding to the first data table to obtain a new first data table.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-4.

Images