Methods, devices, equipment, media and products for processing data

Abstract

This disclosure provides methods, devices, equipment, media, and products for processing data, relating to the field of big data technology, specifically data processing technology. A specific implementation scheme comprises: obtaining data to be processed; determining a data extraction category corresponding to the data to be processed; updating the data to be processed based on the data extraction category to obtain updated data to be processed; and determining target aggregated data based on the updated data to be processed. This implementation method can improve data computation efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, specifically the field of data processing technology. Background Art

[0002] At present, with the continuous development of the Internet, the scale of Internet users in China is growing rapidly, resulting in the existence of massive amounts of data that need to be processed on the Internet.

[0003] Typically, storage and computing resources can be increased to cope with the growing volume of data. However, when there is a large amount of duplicate data, this approach will cause repeated storage and computing of the duplicate data, resulting in low computing efficiency and wasted resources. Summary of the Invention

[0004] The present disclosure provides a method, apparatus, device, medium, and product for processing data.

[0005] According to one aspect of the present disclosure, a method for processing data is provided, including: obtaining data to be processed; determining a data extraction category corresponding to the data to be processed; updating the data to be processed based on the data extraction category to obtain updated data to be processed; and determining target aggregated data based on the updated data to be processed.

[0006] According to another aspect of the present disclosure, a device for processing data is provided, including: a data acquisition unit, configured to acquire data to be processed; a category determination unit, configured to determine a data extraction category corresponding to the data to be processed; a data update unit, configured to update the data to be processed based on the data extraction category to obtain updated data to be processed; and a data aggregation unit, configured to determine target aggregated data based on the updated data to be processed.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above methods for processing data.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any one of the above methods for processing data.

[0009] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements any one of the above methods for processing data when executed by a processor.

[0010] According to the technology disclosed herein, a method for processing data is provided, which can improve data calculation efficiency.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0014] Figure 2 is a flow chart of one embodiment of a method for processing data according to the present disclosure;

[0015] Figure 3 is a schematic diagram of an application scenario of the method for processing data according to the present disclosure;

[0016] Figure 4 is a flowchart of another embodiment of a method for processing data according to the present disclosure;

[0017] Figure 5 is a structural diagram of an embodiment of an apparatus for processing data according to the present disclosure;

[0018] Figure 6 The present invention is a block diagram of an electronic device for implementing the method for processing data according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0021] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0022] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can obtain data to be processed and send it to server 105 via network 104, so that server 105 generates target aggregated data corresponding to the data to be processed. Terminal devices 101, 102, and 103 can respond to data query instructions, determine target query data that matches the data query instruction based on the target aggregated data returned by server 105, and output the target query data.

[0023] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to mobile phones, computers, tablets, and the like. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is given here.

[0024] The server 105 may be a server that provides various services. For example, the server 105 may receive the data to be processed sent by the terminal devices 101, 102, and 103 through the network 104, and determine the data extraction category corresponding to the data to be processed, and update the data to be processed based on the data extraction category to obtain the updated data to be processed, and determine the target aggregated data based on the updated data to be processed, and return the target aggregated data to the terminal devices 101, 102, and 103 through the network 104, so that the terminal devices 101, 102, and 103 can perform data query based on the target aggregated data.

[0025] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitations are given here.

[0026] It should be noted that the method for processing data provided in the embodiment of the present disclosure can be executed by the terminal devices 101, 102, 103, or by the server 105. The device for processing data can be set in the terminal devices 101, 102, 103, or in the server 105. The embodiment of the present disclosure does not limit this.

[0027] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0028] Continue to refer Figure 2 , shows a process 200 of an embodiment of a method for processing data according to the present disclosure. The method for processing data of this embodiment includes the following steps:

[0029] Step 201: Obtain data to be processed.

[0030] In this embodiment, the execution subject (such as Figure 1 The server 105 or the terminal devices 101, 102, 103 in the embodiment can obtain the data to be processed that needs to be processed from the local storage or the electronic device with which the connection is established in advance. The data to be processed can be a partition table that needs to be processed, or can be data in other forms that need to be processed, which is not limited in this embodiment. Usually, there are multiple data to be processed. By processing the data to be processed, the data to be processed can be deduplicated and aggregated, and then the corresponding query and storage operations can be performed based on the deduplicated and aggregated data, which can reduce the amount of data calculation. In addition, corresponding date data and data identifiers can be stored in each data to be processed. The date data can be a partition date, the latest date, etc., and the data identifier can be a unique identifier generated for the data.

[0031] Step 202: Determine the data extraction category corresponding to the data to be processed.

[0032] In this embodiment, the execution entity can respond to different data processing requirements and select a data extraction category that matches the data processing requirements. For example, for a data processing requirement that prioritizes data integrity, the data extraction category corresponding to the data processing requirement can be determined to be a category that retains duplicate records. For a data processing requirement that prioritizes data calculation speed, the data extraction category corresponding to the data processing requirement can be determined to be a category that removes duplicate records. In particular, in the case where the data to be processed has corresponding date data, the duplicate records can be data generated by the same data identifier on different dates.

[0033] In some optional implementations of this embodiment, determining the data extraction category corresponding to the data to be processed may include: determining data processing requirement information corresponding to the data to be processed; determining a priority data processing attribute from the data processing requirement information; in response to determining that the priority data processing attribute is data calculation speed, determining the data extraction category corresponding to the data to be processed to be a first category that does not retain duplicate data; in response to determining that the priority data processing attribute is data integrity, determining the data extraction category corresponding to the data to be processed to be a second category that retains duplicate data. By implementing this optional implementation, the data extraction category can be adaptively determined based on different data processing requirements, thereby improving the flexibility of data processing.

[0034] Step 203: Based on the data extraction category, the data to be processed is updated to obtain updated data to be processed.

[0035] In this embodiment, the execution entity may update the data to be processed based on the data extraction method corresponding to the data extraction category to obtain updated data to be processed. For the first category in which duplicate data is not retained, the update method for updating the data to be processed may be a deduplication-based update. For the second category in which duplicate data is retained, the update method for updating the data to be processed may be a direct update.

[0036] Furthermore, if the data to be processed has corresponding date data, the update frequency of the data to be processed may be at least one day apart. Preferably, the execution entity may update the data to be processed every day to obtain updated data to be processed.

[0037] Step 204: Determine target aggregated data based on the updated data to be processed.

[0038] In this embodiment, the data to be processed may correspond to initial aggregated data. After the data to be processed is updated to obtain the updated data to be processed, the initial aggregated data may be updated accordingly to obtain target aggregated data. The target aggregated data may be data obtained by aggregating data identifiers in the data to be processed according to date data in the data to be processed. Optionally, the target aggregated data may be in the form of an aggregation result table, or in any other form that can reflect the data aggregation results, which is not limited in this embodiment.

[0039] Continue to see Figure 3 , which shows a schematic diagram of an application scenario of the method for processing data according to the present disclosure. Figure 3In the application scenario, the execution subject can obtain data record 301, where data record 301 can be the above-mentioned data to be processed. Afterwards, the execution subject can further determine whether it is necessary to retain the duplicate data in the data record 301. If it is not necessary to retain the duplicate data, the above-mentioned data extraction category is determined to be the first category, and based on the real-time update of the data record 301, a single-partition full table 302 is established, and the update results are aggregated based on the single-partition full table to obtain an aggregated result table 304. Among them, the single-partition full table 302 refers to a partition table corresponding to only one partition. In addition, if it is necessary to retain duplicate data, the above-mentioned data extraction category is determined to be the second category, and based on the real-time update of the data record 301, a partition table 303 is established, and the update results are aggregated based on the partition table 303 to obtain an aggregated result table 304. Among them, the partition table 303 refers to a partition table corresponding to multiple partitions.

[0040] The data processing method provided by the above-mentioned embodiments of the present disclosure determines the data extraction category of the data to be processed, performs different update processing on the data to be processed according to the different data extraction categories, and performs data combination on the updated data to be processed to obtain target aggregated data. By performing data calculation on the target aggregated data, data calculation efficiency can be improved.

[0041] Continue to see Figure 4 , which shows a process 400 of another embodiment of a method for processing data according to the present disclosure. Figure 4 As shown, the method for processing data in this embodiment may include the following steps:

[0042] Step 401: Obtain data to be processed.

[0043] In this embodiment, for the detailed description of step 401 , please refer to the detailed description of step 201 , which will not be repeated here.

[0044] Step 402: Determine the data extraction category corresponding to the data to be processed.

[0045] In this embodiment, for the detailed description of step 402 , please refer to the detailed description of step 202 , which will not be repeated here.

[0046] Step 403: In response to the data extraction category being the preset first category, a first identifier of data to be updated is determined.

[0047] In this embodiment, the data to be processed includes at least partition date data, latest date data, and data identifier. The partition date data may be the date when the partition is generated, the latest date may be the date when the data most recently appears, and the data identifier may be a unique identifier for specific data. For example, the data to be processed may be as shown in Table 1:

[0048] Table 1

[0049] Partition Date Latest Date Data Identification 2021 / 12 / 01 2021 / 11 / 30 Logo 1 2021 / 12 / 01 2021 / 11 / 30 Logo 2 2021 / 12 / 01 2021 / 12 / 01 Logo 3

[0050] The preset first category may be a data extraction category that does not retain duplicate data. When the data extraction category is the preset first category, a first identifier of data to be updated is determined. The first identifier of data to be updated may be an identifier corresponding to new data generated during the current update cycle. The current update cycle is preferably updated daily. Furthermore, the first identifier of data to be updated may be an identifier of data that already exists in the data to be processed, for example, identifier 1, identifier 2, or identifier 3. Furthermore, the first identifier of data to be updated may also be an identifier of data that does not exist in the data to be processed, for example, identifier 4.

[0051] Step 404 : in response to the presence of the first data identifier to be updated in the data to be processed, the partition date data and the latest date data corresponding to the first data identifier to be updated are updated to obtain updated data to be processed.

[0052] In this embodiment, after obtaining the first data identifier to be updated, the execution subject can match the first data identifier to be updated with each data identifier in the data to be processed to determine whether the first data identifier to be updated exists in the data to be processed. If so, the partition date data and the latest date data corresponding to the first data identifier to be updated are updated. Among them, the partition date data and the latest date data corresponding to the first data identifier to be updated can be updated to the current date. For example, in the case where the data to be processed is the data in Table 1, if the first data identifier to be updated is identifier 2, it is determined that the first data identifier to be updated exists in the data to be processed. At this time, the updated data to be processed can be shown in Table 2:

[0053] Table 2

[0054] Partition Date Latest Date Data Identification 2021 / 12 / 02 2021 / 11 / 30 Logo 1 2021 / 12 / 02 2021 / 12 / 02 Logo 2 2021 / 12 / 02 2021 / 12 / 01 Logo 3

[0055] As shown in Table 2, the latest date and partition date corresponding to identifier 2 are both updated to the current date. Optionally, when updating the data to be processed, the partition dates corresponding to the data identifiers in the data to be processed that do not belong to the first data identifier to be updated can also be modified to the current date to ensure that the updated data to be processed is a single-partition partition table.

[0056] Step 405 , in response to the first data identifier to be updated not existing in the data to be processed, the first data identifier to be updated, the partition date data corresponding to the first data identifier to be updated, and the latest date data corresponding to the first data identifier to be updated are added to the data to be processed to obtain updated data to be processed.

[0057] In this embodiment, if the first data identifier to be updated does not exist in the data to be processed, the first data identifier to be updated, the partition date data corresponding to the first data identifier to be updated, and the latest date data corresponding to the first data identifier to be updated are directly added to the data to be processed. For example, if the data to be processed is the data in Table 1, and the first data identifier to be updated is identifier 4, it is determined that the first data identifier to be updated does not exist in the data to be processed. In this case, the updated data to be processed can be shown in Table 3:

[0058] Table 3

[0059] Partition Date Latest Date Data Identification 2021 / 12 / 02 2021 / 11 / 30 Logo 1 2021 / 12 / 02 2021 / 12 / 02 Logo 2 2021 / 12 / 02 2021 / 12 / 01 Logo 3 2021 / 12 / 02 2021 / 12 / 02 Logo 4

[0060] As shown in Table 3, it is possible to add identifier 4, the partition date corresponding to identifier 4, and the latest date. Optionally, when updating the data to be processed, the partition dates corresponding to the data identifiers in the data to be processed that do not belong to the first data identifier to be updated can be modified to the current date to ensure that the updated data to be processed is a single-partition partition table.

[0061] Step 406 : In response to the data extraction category being the preset second category, a second identifier of data to be updated is determined.

[0062] In this embodiment, the data to be processed includes at least partition date data and data identifier. The partition date data may be the date when the partition is generated, and the data identifier may be a unique identifier for specific data. For example, the data to be processed may be as shown in Table 4:

[0063] Table 4

[0064] Partition Date Data Identification 2021 / 11 / 30 Logo 1 2021 / 11 / 30 Logo 2 2021 / 12 / 01 Logo 3

[0065] The date of generating the partition here may be the time when the latest data is generated, that is, the partition date here is equivalent to the latest date mentioned above.

[0066] The preset second category may be a data extraction category for retaining duplicate data. If the data extraction category is the preset second category, a second identifier of data to be updated is determined. The second identifier of data to be updated may be an identifier corresponding to new data generated during the current update cycle. The current update cycle is preferably updated daily.

[0067] Step 407 : Add a second identifier of data to be updated and partition date data corresponding to the second identifier of data to be updated to the data to be processed to obtain updated data to be processed.

[0068] In this embodiment, the execution entity can directly add the second data identifier to be updated and the partition date data corresponding to the second data identifier to be updated to the data to be processed. For example, if the data to be processed is the data in Table 4, and the second data identifier to be updated is identifier 2 and identifier 4, then identifier 2 and the partition date corresponding to identifier 2, and identifier 4 and the partition date corresponding to identifier 4 can be added to the data to be processed. In this case, the updated data to be processed can be shown in Table 5:

[0069] Table 5

[0070] Partition Date Data Identification 2021 / 11 / 30 Logo 1 2021 / 11 / 30 Logo 2 2021 / 12 / 01 Logo 3 2021 / 12 / 02 Logo 4 2021 / 12 / 02 Logo 2

[0071] And, optionally, after obtaining the updated data to be processed based on the data extraction category of the second category, the updated data to be processed can be used as a basis to generate a single-partition full table corresponding to the first category, and then the single-partition full table of the first category can be used to generate an aggregated result table, thereby realizing the integrated use of the two categories.

[0072] Step 408: Determine the initial aggregated data corresponding to the data to be processed.

[0073] In this embodiment, the data to be processed can be updated according to a preset data update cycle, preferably daily. For the data to be processed, corresponding initial aggregated data can be provided. After the data to be processed is updated, the initial aggregated data can be updated accordingly to determine the target aggregated data. For example, the initial aggregated data corresponding to the data to be processed shown in Table 1 and Table 2 can be as shown in Table 6:

[0074] Table 6

[0075] Partition Date Latest Date Aggregate results 2021 / 12 / 01 2021 / 11 / 30 2 2021 / 12 / 01 2021 / 12 / 1 1

[0076] Step 409 : Determine target aggregated data based on the initial aggregated data and the updated data to be processed.

[0077] In this embodiment, the execution entity may aggregate the updated data to be processed, and use the aggregated data to update the initial aggregated data to obtain target aggregated data.

[0078] In some optional implementations of this embodiment, determining the target aggregate data based on the initial aggregate data and the updated data to be processed may include: aggregating the data identifiers in the updated data to be processed to obtain identification aggregation information corresponding to each data identifier; determining the updated aggregate data based on the date data and identification aggregation information in the updated data to be processed; and determining the target aggregate data based on the initial aggregate data and the updated aggregate data.

[0079] In this embodiment, the execution entity can first determine the updated data to be processed after the update, then aggregate the data identifiers corresponding to each updated data at the data identifier granularity to obtain the identifier aggregation information corresponding to each data identifier. Based on the date data and the identifier aggregation information, the updated aggregate data to be updated is generated. The updated aggregate data is then added to the initial aggregate data to obtain the target aggregate data. For example, the target aggregate data for Table 6 can be shown in Table 7:

[0080] Table 7

[0081] Partition Date Latest Date Aggregate results 2021 / 12 / 02 2021 / 11 / 30 1 2021 / 12 / 02 2021 / 12 / 01 1 2021 / 12 / 02 2021 / 12 / 02 2 2021 / 12 / 01 2021 / 11 / 30 2 2021 / 12 / 01 2021 / 12 / 1 1

[0082] The target aggregate data can include the partition date, the latest date, and the aggregation result. The partition date and latest date are described above and will not be repeated here. The aggregation result refers to the number of data identifiers.

[0083] Step 410 : In response to the data query instruction, determine target query data matching the data query instruction from the target aggregated data.

[0084] In this embodiment, the execution entity can respond to a data query instruction based on human-computer interaction with the user and search for corresponding target query data based on the target aggregate data. The data query instruction is used to query the data volume for a specified date. For example, if the target aggregate data is as shown in Table 7, the data volume can be queried for a partition date of 2021 / 12 / 02 and the latest date is from 2021 / 11 / 30 to 2021 / 12 / 2. In this case, the data volume is 4.

[0085] Step 411: Output target query data.

[0086] In this embodiment, the execution subject can output the target query data to complete the data query task.

[0087] For example, the method for processing data provided by the above embodiment of the present disclosure can be specifically used to count the number of users within a certain date range. In this case, the updated data to be processed is as shown in Table 8:

[0088] Table 8

[0089] Field Name Field Type Chinese meaning Example event_day date Partition Date 2021 / 12 / 01 last_day date Latest Date 2021 / 11 / 30 os string operating system android uid string User ID bd2021001 type int Access Type 0 impressions 1 click pv int Number of visits 20

[0090] The execution subject may determine the data identifier based on the operating system, user identifier, and access type. The number of accesses may be the sum of the number of accesses corresponding to the data identifier.

[0091] And, the target aggregate data obtained at this time can be shown in Table 9:

[0092] Table 9

[0093] Field Name Field Type Chinese meaning Example event_day date Partition Date 2021 / 12 / 01 last_day date Latest Date 2021 / 11 / 30 os string operating system android type int Access Type 0 impressions 1 click pv int Number of visits 30 uv int Number of users 10

[0094] The aggregated results may include the aforementioned operating system, access type, number of accesses, and number of users.

[0095] Furthermore, using the method for processing data provided in the above embodiment, when the data extraction category is the first category, the data calculation amount is as follows:

[0096] c1=n1+(1+n2)*n2 / 2

[0097] Among them, c1 is the data calculation amount, n1 is the number of data identifiers in the data to be processed, and n2 is the number of dates corresponding to the data to be processed.

[0098] For the case where the data extraction category is the second category, the data calculation amount is as follows:

[0099]

[0100] c2=c+(1+n2)*n2 / 2

[0101] Wherein, c is the total amount of data in the data to be processed, A is the set of data identifiers in the data to be processed, and D is the set of dates in the data to be processed.

[0102] For data query, the computational complexity is as follows:

[0103]

[0104] Among them, r1 is the computational cost of data query, s(d j ) indicates the latest date is d j The amount of data.

[0105] It can be seen that using the first category as the data extraction category can reduce the amount of data calculation, and using the second category as the data extraction category can ensure data integrity. In addition, using these two categories for data query can achieve data query with lower computational complexity.

[0106] The method for processing data provided by the above-mentioned embodiment of the present disclosure can also utilize the second category that retains duplicate data and the first category that does not retain duplicate data to obtain updated data to be processed according to different data processing requirements, thereby improving the accuracy of determining the updated data to be processed. In addition, in the first category, only the latest date data can be updated for the existing data identifier, thereby further reducing the amount of data calculation. In addition, when generating target aggregated data, aggregated data can be directly generated based on the updated data to be processed, and then the historical initial aggregated data can be retained on the basis of the aggregated data, thereby improving the generation accuracy of the target aggregated data. In addition, using the target aggregated data for data query can reduce the amount of calculation of the data query, thereby improving the efficiency of the data query.

[0107] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for processing data. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to electronic devices such as terminal devices and servers.

[0108] like Figure 5 As shown, the apparatus 500 for processing data in this embodiment includes: a data acquiring unit 501 , a category determining unit 502 , a data updating unit 503 and a data aggregating unit 504 .

[0109] The data acquisition unit 501 is configured to acquire data to be processed.

[0110] The category determination unit 502 is configured to determine the data extraction category corresponding to the data to be processed.

[0111] The data updating unit 503 is configured to update the data to be processed based on the data extraction category to obtain updated data to be processed.

[0112] The data aggregation unit 504 is configured to determine target aggregated data based on the updated data to be processed.

[0113] In some optional implementations of this embodiment, the data to be processed includes at least partition date data, latest date data and data identifier; and the data update unit 503 is further configured to: in response to the data extraction category being the preset first category, determine the first data identifier to be updated; in response to the presence of the first data identifier to be updated in the data to be processed, update the partition date data and latest date data corresponding to the first data identifier to be updated to obtain updated data to be processed.

[0114] In some optional implementations of this embodiment, the data update unit 503 is further configured to: in response to the absence of the first data identifier to be updated in the data to be processed, add the first data identifier to be updated, the partition date data corresponding to the first data identifier to be updated, and the latest date data corresponding to the first data identifier to be updated to the data to be processed, to obtain updated data to be processed.

[0115] In some optional implementations of this embodiment, the data to be processed includes at least partition date data and a data identifier; and the data update unit 503 is further configured to: determine a second data identifier to be updated in response to the data extraction category being a preset second category; add the second data identifier to be updated and the partition date data corresponding to the second data identifier to be updated to the data to be processed, to obtain updated data to be processed.

[0116] In some optional implementations of this embodiment, the data aggregation unit 504 is further configured to: determine initial aggregated data corresponding to the data to be processed; and determine target aggregated data based on the initial aggregated data and the updated data to be processed.

[0117] In some optional implementations of this embodiment, the data aggregation unit 504 is further configured to: aggregate the data identifiers in the updated data to be processed to obtain identifier aggregation information corresponding to each data identifier; determine the updated aggregate data based on the date data and identifier aggregation information in the updated data to be processed; and determine the target aggregate data based on the initial aggregate data and the updated aggregate data.

[0118] In some optional implementations of this embodiment, it further includes: a data query unit, configured to determine target query data matching the data query instruction from the target aggregate data in response to the data query instruction; and output the target query data.

[0119] It should be understood that the units 501 to 504 described in the apparatus 500 for processing data are respectively the same as those in the reference Figure 2 Therefore, the operations and features described above for the method for processing data are also applicable to the apparatus 500 and the units contained therein, and will not be described in detail here.

[0120] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0121] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0122] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0123] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0124] The computing unit 601 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as methods for processing data. For example, in some embodiments, the method for processing data can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for processing data described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the method for processing data by any other appropriate means (e.g., by means of firmware).

[0125] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0126] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0131] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0132] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0133] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for processing data, comprising: Get the data to be processed; Determining data processing requirement information corresponding to the data to be processed; determining a priority data processing attribute from the data processing requirement information; In response to the priority data processing attribute being data calculation speed, determining the data extraction category corresponding to the data to be processed as a first category that does not retain duplicate data; in response to the priority data processing attribute being data integrity, determining the data extraction category corresponding to the data to be processed as a second category that retains duplicate data; Based on the data extraction category, the data to be processed is updated to obtain updated data to be processed, including: for the first category in which duplicate data is not retained, the data to be processed is updated after deduplication; for the second category in which duplicate data is retained, the data to be processed is updated directly; Target aggregated data is determined based on the updated data to be processed.

2. The method according to claim 1, wherein The data to be processed at least includes partition date data, latest date data and data identification; as well as The updating of the data to be processed based on the data extraction category to obtain updated data to be processed includes: In response to the data extraction category being the preset first category, determining a first identifier of data to be updated; In response to the presence of the first data identifier to be updated in the data to be processed, the partition date data and the latest date data corresponding to the first data identifier to be updated are updated to obtain the updated data to be processed.

3. The method according to claim 2, further comprising: In response to the fact that the first data identifier to be updated does not exist in the data to be processed, the first data identifier to be updated, the partition date data corresponding to the first data identifier to be updated, and the latest date data corresponding to the first data identifier to be updated are added to the data to be processed to obtain the updated data to be processed.

4. The method according to claim 1, wherein The data to be processed at least includes partition date data and data identifier; as well as The updating of the data to be processed based on the data extraction category to obtain updated data to be processed includes: In response to the data extraction category being the preset second category, determining a second identifier of data to be updated; The second data identifier to be updated and the partition date data corresponding to the second data identifier to be updated are added to the data to be processed to obtain the updated data to be processed.

5. The method according to claim 1, wherein The determining target aggregated data based on the updated data to be processed includes: Determining initial aggregated data corresponding to the data to be processed; The target aggregated data is determined based on the initial aggregated data and the updated data to be processed.

6. The method according to claim 5, wherein: The determining the target aggregated data based on the initial aggregated data and the updated data to be processed includes: Aggregating the data identifiers in the updated data to be processed to obtain identifier aggregation information corresponding to each data identifier; Determining updated aggregated data based on the date data in the updated data to be processed and the identification aggregate information; The target aggregate data is determined based on the initial aggregate data and the updated aggregate data.

7. The method according to any one of claims 1 to 6, further comprising: In response to a data query instruction, determining target query data matching the data query instruction from the target aggregated data; The target query data is output.

8. A device for processing data, comprising: a data acquisition unit, configured to acquire data to be processed; a category determining unit, configured to determine data processing requirement information corresponding to the data to be processed; determining a priority data processing attribute from the data processing requirement information; In response to the priority data processing attribute being data calculation speed, determining the data extraction category corresponding to the data to be processed as a first category that does not retain duplicate data; in response to the priority data processing attribute being data integrity, determining the data extraction category corresponding to the data to be processed as a second category that retains duplicate data; a data updating unit configured to update the data to be processed in a manner of deduplication followed by updating for the first category of data without retaining duplicate data; For the second category of retaining duplicate data, the updating method for updating the data to be processed is direct updating; The data aggregation unit is configured to determine target aggregated data based on the updated data to be processed.

9. The device according to claim 8, wherein The data to be processed at least includes partition date data, latest date data and data identification; and The data updating unit is further configured to: In response to the data extraction category being the preset first category, determining a first identifier of data to be updated; In response to the presence of the first data identifier to be updated in the data to be processed, the partition date data and the latest date data corresponding to the first data identifier to be updated are updated to obtain the updated data to be processed.

10. The device according to claim 9, wherein The data updating unit is further configured to: In response to the fact that the first data identifier to be updated does not exist in the data to be processed, the first data identifier to be updated, the partition date data corresponding to the first data identifier to be updated, and the latest date data corresponding to the first data identifier to be updated are added to the data to be processed to obtain the updated data to be processed.

11. The device according to claim 8, wherein The data to be processed at least includes partition date data and data identifier; and The data updating unit is further configured to: In response to the data extraction category being the preset second category, determining a second identifier of data to be updated; The second data identifier to be updated and the partition date data corresponding to the second data identifier to be updated are added to the data to be processed to obtain the updated data to be processed.

12. The device according to claim 8, wherein The data aggregation unit is further configured to: Determining initial aggregated data corresponding to the data to be processed; The target aggregated data is determined based on the initial aggregated data and the updated data to be processed.

13. The device according to claim 12, wherein The data aggregation unit is further configured to: Aggregating the data identifiers in the updated data to be processed to obtain identifier aggregation information corresponding to each data identifier; Determining updated aggregated data based on the date data in the updated data to be processed and the identification aggregate information; The target aggregate data is determined based on the initial aggregate data and the updated aggregate data.

14. The apparatus according to any one of claims 8 to 13, further comprising: a data query unit configured to, in response to a data query instruction, determine target query data matching the data query instruction from the target aggregated data; The target query data is output.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images