Method and device for filtering data, equipment and medium

Abstract

The invention provides a method, a device, equipment and a medium for filtering data. In one method, a plurality of first codes for a first field in a plurality of data records in a first data set are respectively determined using a coding function. A plurality of second codes of a second field in a plurality of data records in the second data set are respectively determined using the coding function, the first field and the second field relating to the same data item. A first subset of the first data set and a second subset of the second data set are respectively determined based on the filtering condition, the plurality of first codes, and the plurality of second codes. With the exemplary implementation of the present disclosure, data can be filtered in a simple and efficient manner, thereby providing data with higher quality for subsequent training of a machine learning model.

Description

Technical Field

[0001] Exemplary implementations of the present disclosure generally relate to data processing, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for filtering data. Background Art

[0002] Machine learning technology has been widely used in various application environments, and the acquired sample data can be used to train the machine learning model. A large amount of sample data can be continuously acquired, so it is difficult to directly load the sample data into the memory of the computing device and perform the training process, which leads to unsatisfactory performance of the training process. At this time, it is expected to perform filtering on the acquired massive sample data so as to select a part of the sample data to perform the training process, thereby improving the efficiency of the process of acquiring the machine learning model. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for filtering data is provided. In the method, a plurality of first codes of a first field in a plurality of data records in a first data set are determined respectively using a coding function. A plurality of second codes of a second field in a plurality of data records in a second data set are determined respectively using a coding function, wherein the first field and the second field relate to the same data item. Based on a filtering condition, the plurality of first codes, and the plurality of second codes, a first subset of the first data set and a second subset of the second data set are determined respectively.

[0004] In a second aspect of the present disclosure, a device for filtering data is provided. The device includes: a first encoding module, configured to use an encoding function to respectively determine multiple first encodings of a first field in multiple data records in a first data set. A second encoding module, configured to use an encoding function to respectively determine multiple second encodings of a second field in multiple data records in a second data set, wherein the first field and the second field relate to the same data item. A filtering module, configured to respectively determine a first subset of the first data set and a second subset of the second data set based on a filtering condition, multiple first encodings, and multiple second encodings.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the electronic device executes the method according to the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the first aspect of the present disclosure.

[0007] It should be understood that the content described in this content section is not intended to limit the key features or important features of the implementation of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the following, in conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent. In the accompanying drawings, the same or similar figure labels represent the same or similar elements, wherein:

[0009] Figure 1 A block diagram showing an application environment according to an exemplary implementation of the present disclosure is shown;

[0010] Figure 2 A block diagram for filtering data according to some implementations of the present disclosure is shown;

[0011] Figure 3 A block diagram illustrating execution encoding according to some implementations of the present disclosure is shown;

[0012] Figure 4A and Figure 4B Block diagrams of performing filtering based on different filtering conditions according to some implementations of the present disclosure are respectively shown;

[0013] Figure 5 A block diagram showing the relationship between filtering results of different filtering conditions according to some implementations of the present disclosure;

[0014] Figure 6 A block diagram illustrating selecting different filtering conditions based on the number of data records in a subset according to some implementations of the present disclosure;

[0015] Figure 7 A block diagram showing an algorithm for calculating a connection rate according to some implementations of the present disclosure;

[0016] Figure 8 A block diagram showing an algorithm for generating a benchmark for comparison according to some implementations of the present disclosure;

[0017] Fig. 9 A block diagram illustrating comparison using different data sets as benchmarks according to some implementations of the present disclosure;

[0018] Fig.10 A flowchart showing a method for filtering data according to some implementations of the present disclosure is shown;

[0019] Fig.11 A block diagram showing an apparatus for filtering data according to some implementations of the present disclosure; and

[0020] Fig.12 A block diagram of a device capable of implementing multiple implementations of the present disclosure is shown. DETAILED DESCRIPTION

[0021] The implementation of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain implementations of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the implementations described herein. On the contrary, these implementations are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and implementations of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] In the description of the implementation methods of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "an implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". The following may also include other explicit and implicit definitions. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0023] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0024] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0025] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0026] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly related to the time when the event occurs or the condition is satisfied. For example, in some cases, the subsequent action may be executed immediately when the event occurs or the condition is satisfied; while in other cases, the subsequent action may be executed some time after the event occurs or the condition is satisfied.

[0029] Example Environment

[0030] In the field of machine learning, the quality of training data plays a vital role in the entire training process and directly affects the performance of machine learning models. Therefore, the demand for high-quality training data is increasing. Furthermore, as the amount of data increases, the training time and scale of machine learning models are also increasing, resulting in the inability to obtain a machine learning model with the expected effect within the expected time by directly using the acquired raw data for training.

[0031] See also Figure 1 Describe an application environment according to an example implementation of the present disclosure. For ease of description, in the context of the present disclosure, a recommendation system based on machine learning technology is used as an example to describe the process of data filtering. Alternatively and / or additionally, the technical solution according to the present disclosure can be implemented in other application environments. Figure 1 A block diagram 100 of an application environment according to an exemplary implementation of the present disclosure is shown. Figure 1 As shown, multiple data sets may be involved in the recommendation system 110 : a user data set 110 , an event data set 120 , and a product data set 130 .

[0032] The recommendation system 110 herein may, for example, recommend commodities (and / or other objects) to users. For example, the user data set 110 may include data on the attributes of the user, such as a user identifier (ID), etc. It is understandable that the user ID mentioned in the present application is anonymized. The commodity data set 130 may, for example, include data on the attributes of commodities, such as a commodity identifier, a commodity name, etc. Further, the event data set 120 may include data related to interaction events (e.g., evaluations, etc.) between users and commodities, such as event identifiers, user identifiers, commodity identifiers, etc.

[0033] Furthermore, the training process may involve multiple data sets with associated relationships. For example, the user data set 110 and the event data set 120 are associated through a user identifier, and the commodity data set 130 and the event data set 120 are associated through a commodity identifier. For example, the user data set 110 may include user data of users 1-100, and the event data set 120 may include related events of users 1-100. Assuming that the machine learning model is expected to obtain the associated relationship between the user and the commodity of interest, if a subset of the user data set 110 and the event data set 120 is simply selected as training data, the data in each subset may not accurately reflect the above-mentioned associated relationship.

[0034] For example, if the subset of user data set 110 includes user data of users 1-50, and the subset of event data set 120 includes related events of users 51-100, then the machine learning model cannot learn the association between the user and the product of interest. At this time, it is desirable to perform filtering on the acquired massive sample data so as to select a portion of the sample data for training, thereby improving the efficiency of the process of acquiring the machine learning model.

[0035] Summary of Data Filtering

[0036] In order to at least partially address the deficiencies in the prior art, according to an exemplary implementation of the present disclosure, a method for filtering data is proposed. In summary, an encoding function can be used to perform encoding on the relevant data of a field in a data set, and a matching encoding can be selected based on a predetermined filtering condition so as to select a portion of data records from the data set to generate a subset of the data set. In this way, the purpose of data filtering can be achieved, thereby reducing the amount of data to be processed, thereby improving the performance of subsequent operation processes. Multiple data sets involving fields of the same data item can be processed in the same manner, so that the association relationship between the data of the multiple data sets can be retained while performing data filtering.

[0037] See also Figure 2Describes an overview of an exemplary implementation according to the present disclosure, Figure 2 A block diagram 200 for filtering data according to some implementations of the present disclosure is shown. Figure 2 As shown, encoding operations may be performed on data fields in multiple data sets to be processed. For example, the encoding function 230 may be used to respectively determine multiple first encodings of the first field in multiple data records in the first data set 210. Further, the encoding function 230 may be used to respectively determine multiple second encodings of the second field in multiple data records in the second data set 220. In the context of the recommendation system 110, the first data set 210 may, for example, represent the user data set 110, and the second data set 220 may, for example, represent the event data set 120.

[0038] It should be understood that since the machine learning model is intended to learn the association between different data, the first field in the first data set 210 to be processed and the second field in the second data set 220 may relate to the same data item. Specifically, in the recommendation system, both the first field and the second field may indicate a user identifier. At this time, the machine learning model can learn the association between users and events via the user identifier as an intermediary between the two data sets, that is, a join operation can be performed between the two data sets.

[0039] According to an exemplary implementation of the present disclosure, in order to reduce the amount of data to be processed, a first subset 212 of the first data set 210 and a second subset 222 of the second data set 220 may be determined based on the filtering condition 240, the plurality of first codes, and the plurality of second codes. In other words, data records that meet the filtering condition 240 may be selected to generate the first subset 212 and the second subset 222, respectively. In this way, a portion of data records may be selected from the user data set 110 and the event data set 120 including a large amount of data to serve as the data basis for the subsequent training process.

[0040] It should be understood that, since the data records in the two subsets are determined by the encoding function 230 and the filtering condition 240, the connection between the "user identifier" fields in the data records in the two subsets can be retained. That is, for the user data including a certain user identifier in the first subset 212, the event data associated with the user identifier will be included in the second subset 222. At the same time, for the event data including a certain user identifier in the second subset 222, the user data associated with the user identifier is also included in the first subset 212.

[0041] By using the example implementation of the present disclosure, it is possible to ensure that the data records in the first subset 212 and the second subset 222 obtained retain the association relationship between the original massive data records, thereby ensuring that the association relationship extracted in the subsequent processing process can reflect the relevant distribution of the original massive data records. Further, since the amount of data in the first subset 212 and the second subset 222 at this time will be lower than the original amount of data, the data records in each subset can be loaded into the memory of the computing device in the subsequent processing process, thereby improving the performance of the subsequent processing.

[0042] Detailed process of data filtering

[0043] An overview of an example implementation according to the present disclosure has been described, and more details of data filtering are described below with reference to the accompanying drawings. In a recommendation system, a machine learning model can be used to determine the association between users and commodities. For example, a user data set may include multiple fields, such as user ID, user registration time, and so on. As mentioned above, this information is obtained with user authorization. The event data set 120 may include multiple fields, such as event ID, user ID, commodity ID, event time, and so on. The commodity data set 130 may include multiple fields, such as commodity ID, commodity name, shelf time, and so on. It should be understood that the commodities herein may indicate real commodities (e.g., food, beverages, etc.) or virtual commodities (e.g., audio, video, games, items in a virtual world, and so on).

[0044] Here, one field in the user dataset 110 and another field in the event dataset 120 both indicate the user ID (that is, the “user ID” is used as a connection field), and thus the user dataset 110 and the event dataset 120 can be connected via the “user ID”. Further, one field in the product dataset 130 and another field in the event dataset 120 both indicate the product ID (that is, the “product ID” is used as a connection field), and thus the product dataset 130 and the event dataset 120 can be connected via the “product ID”.

[0045] In the application environment of the recommendation system, each data record in the first data set 210 and the second data set 220 described above may include historical data obtained in the recommendation system. Specifically, the multiple fields in the first data set 210 respectively relate to multiple attributes of objects in the recommendation system, and the multiple fields in the second data set 220 respectively relate to multiple attributes of events in the recommendation system.

[0046] Specifically, the first data set 210 may indicate the user data set 110, and the second data set 220 may indicate the event data set 120. At this time, both data sets include a "user ID". Alternatively and / or additionally, the first data set 210 may indicate the commodity data set 130, and the second data set 220 may indicate the event data set 120. At this time, both data sets include a "commodity ID". Therefore, the determined first subset 212 and second subset 222 can be used as training data for a machine learning model, thereby extracting the association relationship between users and commodities. It should be understood that the first data set 210 and second data set 220 described above are merely schematic, and the order of the two data sets can be exchanged. At this time, the order of the first subset 212 and the second subset 222 will also be exchanged.

[0047] It should be understood that the various data sets described above are merely illustrative. Alternatively and / or additionally, each data set may include more, fewer, or different fields, as long as the two data sets to be processed include fields indicating the same data items.

[0048] According to an exemplary implementation of the present disclosure, an encoding operation may be performed on the connection fields in each data set to determine the corresponding encoding. Figure 3 A block diagram 300 of performing encoding according to some implementations of the present disclosure is shown. Figure 3 As shown, for example, the encoding function 230 may be applied to a field 310 including “User ID” in order to determine a corresponding encoding 320 .

[0049] It should be understood that a variety of encoding functions (e.g., hash functions) currently known and / or to be developed in the future may be used. For example, encoding function 230 may map a user ID represented by a string to a number represented in a decimal format. At this time, the obtained number may, for example, include a number with the last digit being "0"-"9", a number with the second-to-last digit being "0"-"9", and so on. Specifically, encoding 320 may include: number 331 (the last two digits are "00"), number 332 (the last two digits are "01"), ..., number 333 (the last two digits are "09"), number 341 (the last two digits are "10"), number 342 (the last two digits are "11"), ..., number 343 (the last two digits are "19"), and so on.

[0050] Further, the data records in the corresponding subset can be selected based on whether the code matches the predetermined filtering condition 240. In the initial stage, the first subset can be set to an empty set. Further, in the process of determining the first subset 212, for a given first code among multiple first codes, if it is determined that the given first code satisfies the filtering condition 240, the first subset can be updated based on the first data record corresponding to the given first code among multiple data records in the first data set. It should be understood that the filtering condition 240 can be pre-set, for example, the filtering condition 240 can represent: the last digit in the code is "1", or the last two digits in the code are "11", and so on.

[0051] Figure 4A and Figure 4B Block diagrams 400A and 400B respectively show filtering performed based on different filtering conditions according to some implementations of the present disclosure. Figure 4A As shown, at this time, the filtering condition 410 is set to: the last bit in the code is "1". Figure 4A As shown, the last digits of the numbers 332 and 342 are both “1”, thus satisfying the filtering condition 410 , and the data records corresponding to the numbers 332 and 342 can be added to the first subset 212 .

[0052] It should be understood that the specific content of the filtering condition can be set based on the specific application environment. For example, the filtering condition 420 can represent: the last digit in the code is "9". Figure 4B As shown, the last digit of numbers 333 and 343 is “9”, thus satisfying the filtering condition 240 , and data records corresponding to numbers 333 and 343 can be added to the first subset 212 .

[0053] In the above example, the last digit of the code may involve ten numbers from "0" to "9", and only the data record corresponding to one number is selected. At this time, the amount of data after filtering will become 1 / 10 of the original data amount. In this way, data records can be easily extracted from the original data set including massive data, thereby reducing the amount of data to be processed later without affecting the data distribution law.

[0054] According to an example implementation of the present disclosure, the second subset may be determined in a similar manner. In the initial stage, the second subset may be set to an empty set. Further, for a given second encoding in the plurality of second encodings, if it is determined that the given second encoding satisfies the filtering condition, the second subset is updated based on a second data record corresponding to the given second encoding in the plurality of data records in the second data set.

[0055] It should be understood that the encoding function 230 and the filtering condition 240 described above are merely illustrative and not restrictive. Alternatively and / or additionally, in the process of processing the first data set 210 and the second data set 220, the encoding function 230 may map the string format to hexadecimal, and may choose to retain the data records with the last digit being "0"-"1", in which case the amount of data after filtering will become 2 / 16=1 / 8 of the original data amount.

[0056] According to an example implementation of the present disclosure, the first data set 210 and the second data set 220 may include the same or different amounts of data. Generally speaking, the amount of user data in a recommendation system is usually less than the amount of event data. For example, a user can perform different interactions on one or more products and generate different events. For example, user 1 can perform different operations such as evaluation and forwarding on product 1, and different events can be generated. For another example, user 1 can perform evaluation operations on different products 1, product 2, etc., and different events can also be generated.

[0057] According to an example implementation of the present disclosure, the filter condition 240 may include a single predetermined condition. In other words, the same predetermined condition (e.g., the first condition) may be used to determine both the first subset 212 and the second subset 222. In this way, the two subsets may be determined in a unified manner, thereby facilitating the simplification of the operational flow of data filtering.

[0058] Specifically, the encoding of each data record in the original data set can be compared with the predetermined condition to determine whether to add the data record to the subset. As the processing continues, the number of data records in the subset will continue to increase. According to an example implementation of the present disclosure, a predetermined threshold can be set, and before the amount of data in the subset reaches the threshold or when the threshold is reached, the predetermined condition is used to determine the data records in the subset. It should be understood that the predetermined threshold can be represented in a variety of ways. For example, the number of data records in the subset (e.g., 100,000, or other numerical values) can be used to represent the predetermined threshold. Alternatively and / or additionally, the amount of data in the subset (e.g., 1GB, or other numerical values) can be used to represent the predetermined threshold.

[0059] It should be understood that the difference in the amount of data in each original data set may result in the difference in the amount of data in each filtered subset. According to an example implementation of the present disclosure, in order to facilitate post-processing, the amount of data in the two data sets to be processed may be first determined, and the data set including fewer data records may be used as the first data set, and the data set including more data records may be used as the second data set. At this time, the number of the first subset of the multiple data records in the first data set is not less than the number of the second subset of the multiple data records in the second data set.

[0060] The purpose of distinguishing the amount of data in each data set is to ensure that the amount of data in each subset does not exceed a predetermined threshold. For example, the second data set can be processed first, and before determining that the amount of data in the second subset exceeds the predetermined threshold, the predetermined condition is used to generate two subsets. At this time, generally speaking, the amount of data in both subsets is lower than the predetermined threshold, and the two data sets can be processed further using a single predetermined condition to generate corresponding subsets.

[0061] According to an exemplary implementation of the present disclosure, the filter condition 240 may include a plurality of different conditions. Figure 5 A block diagram 500 is shown showing the relationship between the filtering results of different filtering conditions according to some implementations of the present disclosure. Figure 5 As shown, the filtering condition 240 may include a first condition 510 and a second condition 520. Here, different filtering conditions may correspond to different filtering results, for example, the first condition 510 corresponds to the first result 512 and the second condition 520 corresponds to the second result 522. As shown, the range of the first result 512 may be larger, and the range of the second result 522 may be smaller, that is, the second result 522 may be a subset of the first result 512.

[0062] Using the example implementation of the present disclosure, as the filtering process continues, when the amount of data in the subset exceeds or is about to exceed a predetermined threshold, the filtering condition can be dynamically switched. For example, a more stringent second condition 520 that produces fewer results can be selected to prevent the situation where the amount of data in the subset is too large and difficult to be loaded into the memory of the computing device.

[0063] Figure 6 A block diagram 600 is shown for selecting different filter conditions based on the number of data records in a subset according to some implementations of the present disclosure. Figure 6 As shown, the coordinate axis 610 represents the number of data records in the subset, the left side of the coordinate axis 610 may represent the origin "0", and the threshold number 620 may have a pre-specified value (eg, 100,000, or other values). Figure 6 In the example, when the number of data records in the subset is within the range of 0 to a threshold number 620, the filtering process can be performed based on the first condition 510; when the number of data records in the subset reaches the threshold number 620, the filtering process can be performed based on the second condition 520.

[0064] According to an example implementation of the present disclosure, if the number of data records in both subsets is lower than a predetermined threshold, the first subset and the second subset are determined based on the first condition. If the number of data records in either of the two subsets is not lower than the predetermined threshold, the first subset and the second subset are determined based on the second condition. That is, a conditional switch 630 may be performed when the threshold number 620 is met so as to switch from the first condition 510 to the second condition 520. In this way, the results from the various filtering conditions may be compatible, thereby reducing the number of data records to be added to the subset based on the satisfaction of the previous filtering condition.

[0065] According to an example implementation of the present disclosure, after the conditional switch 630, the data records in each determined subset can be rearranged. For example, each subset can be filtered based on the new second condition 520. Assuming that the first condition 510 is to retain the data records whose last digit of the code is "1", and the second condition 520 is to retain the data records whose last two digits of the code are "11", then only the data records whose last two digits of the code are "11" can be retained in the subset, that is, the data records that do not meet the second condition 520 (for example, the data records whose last two digits of the code are "01", "21", "31", ..., "91") are removed. At this time, the amount of data in the subset will become 1 / 10 of the original. Although the above technical solution will result in additional workload, it can ensure that the amount of data does not exceed the predetermined threshold, thereby ensuring the performance of subsequent processing.

[0066] It should be understood that as an online streaming method, the exact amount of data to be passed in is uncertain. By using the conditional switch 630 described above, dynamically configured filtering rules can be implemented. For example, a re-hash method can be used to limit the amount of data in the filtered subset.

[0067] According to an example implementation of the present disclosure, in order to adapt to the memory limitations of the computing device, the rehash threshold can be set to 100,000-200,000. For example, for the first 100,000 data records, no filtering rules may be executed; when the amount of data in the subset reaches 100,000, an example rule may be dynamically configured to retain only data records whose last digit of the code is "1". All previously acquired data will be recalculated and hashed, and only hashed data ending with "1" will be retained. At this point, the amount of data in the subset will be reduced to 1 / 10 of the original. As more data arrives, when the amount of data increases to 100,000 again, more stringent filtering conditions can be used. For example, only data records whose last two digits of the code are "11" can be retained, and so on.

[0068] According to an example implementation of the present disclosure, it should be ensured that the level of the filter condition applied to the first data set is less than or equal to the level of the filter condition applied to the second data set. In other words, the filter condition applied to a data set including more data records is more stringent than that of a data set including fewer data records. For example, if the filter condition applied to the user data set is to retain the last two digits "11", and the filter condition applied to the event data set is to retain the last digit "1", then the event subset may involve all users ending with "21", which will result in a sampling connection rate result of 0%, which in turn leads to an error. At this point, it can be specified that the amount of data in the first data set should be less than the amount of data in the second data set. If the above specification is not met, an alarm message can be provided.

[0069] During the data filtering process, the data in each data set may change. For example, a new registered user may appear, a new event may appear, and a new product may appear. At this time, a similar method can be used to process the newly added data. Using the example implementation of the present disclosure, the latest incoming data can be continuously processed as the recommendation system runs, thereby providing real-time dynamic processing capabilities.

[0070] According to an exemplary implementation of the present disclosure, if it is determined that the third data record is added to the target data set in the first data set and the second data set, a third encoding of a third field in the third data record can be determined using an encoding function, where the third field relates to a data item. Further, if it is determined that the third encoding satisfies a predetermined filtering condition, the third data record is added to a target subset corresponding to the target data set in the first subset and the second subset.

[0071] Specifically, assuming that user n is a newly registered user, a new data record (for example, represented as data record 1) will appear in the user data set 110. If the code of user n meets the filtering condition 240 described above, the new data record 1 can be added to the first subset. For another example, assuming that user 1 has recently evaluated product 1 again, a new data record 2 will appear in the event data set 120. If the code of user 1 meets the filtering condition 240 described above, the new data record 2 can be added to the second subset.

[0072] It should be understood that the amount of data of the first subset 212 and the second subset 222 obtained by the method described above will be much smaller than the original first data set 210 and the second data set 220. At this time, the first subset 212 and the second subset 222 can be used as training data and loaded into the memory of the computing device to train the machine learning model. Using the example implementation of the present disclosure, the training data can be loaded into the memory with a higher access speed of the computing device during the training process, thereby improving the training speed. Compared with the conventional technical solution of performing training using a large amount of original data, it is not necessary to perform data exchange between the memory with a higher access speed and the external low-speed memory of the computing device, thereby improving the training performance.

[0073] Figure 7 A block diagram 700 of an algorithm for calculating a connection rate according to some implementations of the present disclosure is shown. Figure 7 As shown, the input data of Algorithm 1 are the event table E including event data and the user table U including user data. At line 1, the function set() can be used to set the subset users of the user data, and the count count is set to 0. At lines 2-3, each user u in the user table U can be processed in a loop, and the user identifier (user_id) of each user can be added to the subset users. At lines 4-6, each event e in the event table E can be processed in a loop, and the count count of the users in the subset users appearing in the event table can be determined. At line 7, the connection rate count / len(E) can be returned.

[0074] Compared with the existing offline batch processing method, the proposed data processing scheme can process incoming data in real time. Furthermore, the amount of filtered data is kept within a reasonable range, so it can be loaded into the memory of the computing device to improve processing efficiency. Specifically, a filtering task can be started for each data set, and the filtering results can be updated regularly (usually 10 seconds or other values). When the threshold number is set to 100,000, using the rehash method described above, the total number of filtering results will not exceed the following limit:

[0075]

[0076] According to an example implementation of the present disclosure, tests can be performed on the following two open datasets to verify the advantages of the technical solution of the present disclosure: MovieLens and RetailRocket2. Both datasets contain relatively high amounts of data, and the original datasets always produce a 100% connection rate. A mutation algorithm can be executed to intentionally introduce some errors in order to verify the performance of the data filtering technical solution of the present disclosure.

[0077] Figure 8 A block diagram 800 of an algorithm for generating a benchmark for comparison according to some implementations of the present disclosure is shown. Figure 8 As shown, the input data of Algorithm 2 is the user table U including user data, and the mutation rate m. At line 1, the subset is set to empty, and at lines 2-7, each user in the user table U is processed in a loop. Specifically, the flag is set to 1 with a probability of m%. If the flag is 1, the user ID that does not appear in the user table U is added to the subset users; otherwise, the current user u is added to the subset users. In this way, a benchmark for comparison can be generated, that is, the subset users can include m% of the original user data.

[0078] Specifically, 5 mutations ranging from 10% to 50% can be performed on each dataset.

[0079] Fig. 9 Block diagram 900 illustrating a comparison utilizing different data sets as a benchmark according to some implementations of the present disclosure. Fig. 9 The horizontal axis in represents the mutation rate, and the vertical axis represents the accuracy rate. Fig. 9 As can be seen in FIG, for different data sets, the detected connection rate matches the ratio of connectable goods / users. It can be seen that the data distribution in the subsets obtained using the example implementation of the present disclosure is consistent with the data distribution in the original data set, so the training process performed based on these subsets is also accurate and reliable.

[0080] By utilizing the example implementation of the present disclosure, the data filtering process can control the amount of training data within a reasonable range while ensuring the quality of the training data, thereby increasing the speed of subsequent training processes and improving the accuracy of machine learning models.

[0081] Example Process

[0082] Fig.10 A flow chart of a method 1000 for filtering data according to some implementations of the present disclosure is shown. At block 1010, a plurality of first codes of a first field in a plurality of data records in a first data set are determined using a coding function. At block 1020, a plurality of second codes of a second field in a plurality of data records in a second data set are determined using a coding function, wherein the first field and the second field relate to the same data item. At block 1030, a first subset of the first data set and a second subset of the second data set are determined based on a filtering condition, the plurality of first codes, and the plurality of second codes.

[0083] According to an example implementation of the present disclosure, determining the first subset includes: for a given first encoding among multiple first encodings, in response to determining that the given first encoding satisfies a filtering condition, updating the first subset based on a first data record corresponding to the given first encoding among multiple data records in the first data set.

[0084] According to an example implementation of the present disclosure, determining the second subset includes: for a given second encoding among multiple second encodings, in response to determining that the given second encoding satisfies a filtering condition, updating the second subset based on a second data record corresponding to the given second encoding among multiple data records in the second data set.

[0085] According to an example implementation of the present disclosure, the filtering condition includes a first condition, and determining the first subset and the second subset includes: in response to determining that a first subset number of data records in the first subset and a second subset number of data records in the second subset are both lower than a predetermined threshold, determining the first subset and the second subset based on the first condition.

[0086] According to an example implementation of the present disclosure, the filtering condition further includes a second condition, and determining the first subset and the second subset includes: in response to determining that any one of the first subset quantity and the second subset quantity is not lower than a predetermined threshold, determining the first subset and the second subset based on the second condition.

[0087] According to an example implementation of the present disclosure, determining the first subset based on the second condition includes at least any of the following: removing data records that do not meet the second condition from the first subset; and processing data records in the first data set that have not been processed based on the second condition.

[0088] According to an example implementation of the present disclosure, the second result from the second filter condition is a subset of the first result from the first filter condition, and determining the first subset and the second subset includes: determining the first subset based on the first condition, and determining the second subset based on the second condition.

[0089] According to an example implementation of the present disclosure, a first number of the plurality of data records in the first data set is not greater than a second number of the plurality of data records in the second data set.

[0090] According to an example implementation of the present disclosure, the method 1000 further includes: in response to determining that a third data record is added to a target data set in the first data set and the second data set, using an encoding function, determining a third encoding of a third field in the third data record, the third field relating to a data item; and in response to determining that the third encoding satisfies a filtering condition, adding the third data record to a target subset of the first subset and the second subset corresponding to the target data set.

[0091] According to an example implementation of the present disclosure, each data record in the first data set and the second data set includes historical data obtained in the recommendation system, multiple fields in the first data set respectively relate to multiple attributes of an object in the recommendation system, and multiple fields in the second data set respectively relate to multiple attributes of an event associated with the object in the recommendation system.

[0092] According to an exemplary implementation of the present disclosure, the object includes at least any one of the following: a user and an item in a recommendation system.

[0093] According to an example implementation of the present disclosure, the method 1000 further includes: loading the first subset and the second subset into a memory of a computing device to train a machine learning model.

[0094] Example devices and equipment

[0095] Fig.11 A block diagram of a device 1100 for filtering data according to some implementations of the present disclosure is shown. The device 1100 includes: a first encoding module 1110, configured to use an encoding function to respectively determine a plurality of first encodings of a first field in a plurality of data records in a first data set; a second encoding module 1120, configured to use an encoding function to respectively determine a plurality of second encodings of a second field in a plurality of data records in a second data set, wherein the first field and the second field relate to the same data item; and a filtering module 1130, configured to respectively determine a first subset of the first data set and a second subset of the second data set based on a filtering condition, the plurality of first encodings, and the plurality of second encodings.

[0096] According to an example implementation of the present disclosure, the first encoding module 1110 includes: a first update module, configured to update a first subset based on a first data record corresponding to the given first encoding among multiple first encodings in response to determining that the given first encoding satisfies a filtering condition.

[0097] According to an example implementation of the present disclosure, the second encoding module 1120 includes: a second update module, configured to update the second subset based on second data records corresponding to the given second encoding among multiple data records in the second data set in response to determining that the given second encoding satisfies a filtering condition.

[0098] According to an example implementation of the present disclosure, the filtering condition includes a first condition, and the filtering module includes: a first filtering module configured to determine the first subset and the second subset based on the first condition in response to determining that a first subset number of data records in the first subset and a second subset number of data records in the second subset are both lower than a predetermined threshold.

[0099] According to an example implementation of the present disclosure, the filtering condition further includes a second condition, and the filtering module includes: a second filtering module, configured to determine the first subset and the second subset based on the second condition in response to determining that any one of the first subset quantity and the second subset quantity is not lower than a predetermined threshold.

[0100] According to an example implementation of the present disclosure, the second filtering module includes: a removal module configured to remove data records that do not meet the second condition from the first subset; and a processing module configured to process the unprocessed data records in the first data set based on the second condition.

[0101] According to an example implementation of the present disclosure, the second result from the second filter condition is a subset of the first result from the first filter condition, and determining the first subset and the second subset includes: determining the first subset based on the first condition, and determining the second subset based on the second condition.

[0102] According to an example implementation of the present disclosure, a first number of the plurality of data records in the first data set is not greater than a second number of the plurality of data records in the second data set.

[0103] According to an example implementation of the present disclosure, the device 1100 further includes: an encoding determination module, configured to, in response to determining that a third data record is added to a target data set in the first data set and the second data set, determine a third encoding of a third field in the third data record using an encoding function, wherein the third field relates to a data item; and an adding module, configured to, in response to determining that the third encoding satisfies a filtering condition, add the third data record to a target subset corresponding to the target data set in the first subset and the second subset.

[0104] According to an example implementation of the present disclosure, each data record in the first data set and the second data set includes historical data obtained in the recommendation system, multiple fields in the first data set respectively relate to multiple attributes of an object in the recommendation system, and multiple fields in the second data set respectively relate to multiple attributes of an event associated with the object in the recommendation system.

[0105] According to an exemplary implementation of the present disclosure, the object includes at least any one of the following: a user and an item in a recommendation system.

[0106] According to an example implementation of the present disclosure, the apparatus 1100 further includes: a loading module configured to load the first subset and the second subset into the memory of the computing device in order to train the machine learning model.

[0107] Fig.12 1 shows a block diagram of a device 1200 capable of implementing multiple implementations of the present disclosure. It should be understood that Fig.12 The computing device 1200 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the implementations described herein. Fig.12 The computing device 1200 shown may be used to implement the methods described above.

[0108] like Fig.12 As shown, computing device 1200 is in the form of a general-purpose computing device. Components of computing device 1200 may include, but are not limited to, one or more processors or processing units 1210, memory 1220, storage device 1230, one or more communication units 1240, one or more input devices 1250, and one or more output devices 1260. Processing unit 1210 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 1220. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device 1200.

[0109] The computing device 1200 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device 1200, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1220 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1230 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the computing device 1200.

[0110] The computing device 1200 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig.12As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 1220 may include a computer program product 1225 having one or more program modules that are configured to perform various methods or actions of various implementations of the present disclosure.

[0111] The communication unit 1240 enables communication with other computing devices via a communication medium. Additionally, the functions of the components of the computing device 1200 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 1200 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0112] Input device 1250 may be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1260 may be one or more output devices, such as a display, speaker, printer, etc. Computing device 1200 may also communicate with one or more external devices (not shown) through communication unit 1240 as needed, such as storage devices, display devices, etc., communicate with one or more devices that allow a user to interact with computing device 1200, or communicate with any device that allows computing device 1200 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0113] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, the computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0114] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0115] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0116] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0117] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0118] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for filtering data, comprising: Using the encoding function, respectively determine a plurality of first encodings of the first fields in the plurality of data records in the first data set; Using the encoding function, respectively determine a plurality of second encodings of second fields in a plurality of data records in a second data set, wherein the first field and the second field relate to the same data item; as well as Based on the filtering condition, the plurality of first codes, and the plurality of second codes, a first subset of the first data set and a second subset of the second data set are determined respectively.

2. The method of claim 1 , wherein determining the first subset comprises: For a given first encoding among the multiple first encodings, in response to determining that the given first encoding satisfies the filtering condition, the first subset is updated based on a first data record among the multiple data records in the first data set that corresponds to the given first encoding.

3. The method of claim 2, wherein determining the second subset comprises: For a given second encoding among the plurality of second encodings, in response to determining that the given second encoding satisfies the filtering condition, the second subset is updated based on a second data record among the plurality of data records in the second data set that corresponds to the given second encoding.

4. The method of claim 1, wherein the filtering condition comprises a first condition, and determining the first subset and the second subset comprises: In response to determining that a first subset number of data records in the first subset and a second subset number of data records in the second subset are both below a predetermined threshold, the first subset and the second subset are determined based on the first condition.

5. The method of claim 4, wherein the filtering condition further comprises a second condition, and determining the first subset and the second subset comprises: In response to determining that any one of the first subset quantity and the second subset quantity is not lower than the predetermined threshold, the first subset and the second subset are determined based on the second condition.

6. The method according to claim 5, wherein determining the first subset based on the second condition comprises at least any one of the following: remove data records that do not meet the second condition from the first subset; and The data records in the first data set that have not been processed are processed based on the second condition.

7. The method of claim 5, wherein the second results from the second filter condition are a subset of the first results from the first filter condition, and determining the first subset and the second subset comprises: The first subset is determined based on the first condition, and the second subset is determined based on the second condition. 8 . The method of claim 5 , wherein a first number of the plurality of data records in the first data set is not greater than a second number of the plurality of data records in the second data set.

9. The method according to claim 1, further comprising: In response to determining that a third data record is added to a target data set of the first data set and the second data set, determining, using the encoding function, a third encoding of a third field in the third data record, the third field relating to the data item; as well as In response to determining that the third code satisfies the filtering condition, the third data record is added to a target subset corresponding to the target data set among the first subset and the second subset.

10. A method according to claim 1, wherein each data record in the first data set and the second data set includes historical data obtained in a recommendation system, multiple fields in the first data set respectively relate to multiple attributes of an object in the recommendation system, and multiple fields in the second data set respectively relate to multiple attributes of an event associated with the object in the recommendation system. The method according to claim 10 , wherein the object comprises at least any one of the following: a user and an item in the recommendation system.

12. The method according to claim 1, further comprising: The first subset and the second subset are loaded into a memory of a computing device to train a machine learning model.

13. A device for filtering data, comprising: A first encoding module is configured to respectively determine a plurality of first encodings of first fields in a plurality of data records in a first data set using an encoding function; A second encoding module is configured to respectively determine a plurality of second encodings of second fields in a plurality of data records in a second data set using the encoding function, wherein the first field and the second field relate to the same data item; as well as The filtering module is configured to determine a first subset of the first data set and a second subset of the second data set respectively based on a filtering condition, the plurality of first codes, and the plurality of second codes.

14. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 12 when executed by the at least one processing unit.

15. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 12.

Images