Data filtering method and system, electronic device, and storage medium

Abstract

The application discloses a data filtering method and system, an electronic device and a storage medium. The method comprises the following steps: receiving data to be filtered, and determining a label value of the data; inputting the data into a first filter, and determining an expected value and a tail value; the expected value is used for representing the label value determined based on data continuity, and the tail value is used for representing the earliest effective label value in a second filter; if the label value is equal to the expected value, filtering the data through the first filter to determine target data after filtering; the first filter is used for filtering data with continuous label values; or, if the label value is not equal to the expected value and the label value is greater than or equal to the tail value, inputting the data into the second filter, filtering the data through the second filter to determine the target data after filtering; and the second filter is used for filtering data corresponding to non-continuous label values. The embodiment of the application is beneficial to improving the data processing efficiency. The application can be widely applied in the field of network communication technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network communication, and in particular to a data filtering method and system, an electronic device and a storage medium. BACKGROUND

[0002] Duplicate data filtering application is a tool for identifying and deleting duplicate records in a data set. It can be applied to various data types, including text, numbers, images, etc. Duplicate data refers to the existence of multiple identical or similar records in a data set, which may be caused by data input errors, system failures or other reasons. The main goal of duplicate data filtering application is to improve data quality and accuracy. By detecting and deleting duplicate records, data redundancy can be reduced, and data analysis and processing efficiency can be improved. In related technologies, duplicate data filtering applications usually identify duplicate records based on algorithms and techniques. These algorithms can compare different records in a data set and determine their similarity according to a predefined similarity measure. Once duplicate records are identified, the application can take appropriate action, such as deleting, merging or marking duplicate records. However, the filtering method in the related art usually reduces the forwarding performance of the data, affecting the processing of the data. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide an efficient data filtering method and system, an electronic device and a storage medium.

[0004] To achieve the above-mentioned purpose, one aspect of an embodiment of the present application provides a data filtering method, which comprises: receiving data to be filtered and determining the label value of the data; inputting the data into a first filter to determine an expected value and a tail value; the expected value is used to represent the label value determined based on data continuity, and the tail value is used to represent the earliest valid label value in a second filter; if the label value is equal to the expected value, the data is filtered by the first filter to determine the target data after filtering; the first filter is used to filter data with continuous label values; or, if the label value is not equal to the expected value and the label value is greater than or equal to the tail value, the data is input into the second filter, and the data is filtered by the second filter to determine the target data after filtering; the second filter is used to filter data corresponding to non-continuous label values. By labeling the data, the label value of the data is determined, and then the first filter and the second filter are used to filter the data. No additional deployment is required, and the original network data packet is not damaged. Therefore, the embodiment of the present application is beneficial to improving the data processing efficiency.

[0005] In some embodiments, the method provided by the embodiments of the present application, the secondary filter comprises a plurality of filtering nodes, the filtering nodes comprise boundary records and marking units; the filtering processing of the data by the secondary filter comprises:

[0006] determining whether the label value is registered in the secondary filter according to the boundary record and the label value;

[0007] if the label value is registered in the secondary filter, determining a first boundary corresponding to the label value; and filtering the data according to the first marking unit to determine target data; wherein the first filtering node comprises a first boundary record and a first marking unit;

[0008] Alternatively, if the label value is not registered in the secondary filter, registering a second filtering node in the secondary filter according to the label value.

[0009] In some embodiments, the method provided by the embodiments of the present application, the second filtering node comprises a second boundary record and a second marking unit; the registering of the second filtering node in the secondary filter according to the label value comprises:

[0010] determining a first boundary value according to a current expected value;

[0011] determining a second boundary value according to the label value; the first boundary value and the second boundary value are two interval endpoints of the second boundary record;

[0012] determining a second marking unit according to the first boundary value and the second boundary value.

[0013] In some embodiments, the method provided by the embodiments of the present application, the filtering processing of the data according to the first marking unit comprises:

[0014] if the label value has been registered in the first marking unit, discarding the data, and determining that the target data does not include the data;

[0015] Alternatively, if the label value has not been registered in the first marking unit, registering the label value in the first marking unit according to the first boundary value and the label value, and determining that the target data includes the data.

[0016] In some embodiments, the method provided by the embodiments of the present application, the filtering node further comprises a timer, and the timer starts timing from the establishment of the filtering node; the method further comprises:

[0017] if the duration of the timer is greater than a preset duration, deleting the filtering node; the preset duration is used to represent the aging time of the filtering node.

[0018] In some embodiments, the method provided by the embodiments of the present application further comprises:

[0019] If the tag value is equal to the history value, discarding the data and determining that the target data does not include the data.

[0020] Or, if the tag value is less than the tail value, discarding the data and determining that the target data does not include the data.

[0021] In some embodiments, the method provided by the embodiments of the present application further comprises:

[0022] If the secondary filter is not empty, determining that the tail value is the earliest valid tag value in the secondary filter.

[0023] Or, if the secondary filter is empty, determining that the tail value is an expected value.

[0024] To achieve the above object, another aspect of the embodiments of the present application provides a data filtering system, which comprises:

[0025] A first module is configured to receive data to be filtered and determine a tag value of the data.

[0026] A second module is configured to input the data into a primary filter, determine an expected value and a tail value; the expected value is used to represent a tag value determined based on data continuity, and the tail value is used to represent an earliest valid tag value in a secondary filter.

[0027] A third module is configured to, if the tag value is equal to the expected value, filter the data by the primary filter to determine target data after filtering; the primary filter is used to filter data with continuous tag values.

[0028] Or,

[0029] A fourth module is configured to, if the tag value is not equal to the expected value and the tag value is greater than or equal to the tail value, input the data into the secondary filter, filter the data by the secondary filter to determine target data after filtering; the secondary filter is used to filter data corresponding to non-continuous tag values.

[0030] To achieve the above object, another aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor; the memory stores a computer program; and the processor executes the computer program to implement the above method.

[0031] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0032] The embodiment of the present application has at least the following advantages: the data filtering method provided by the embodiment of the present application includes: receiving data to be filtered, and determining a label value of the data; inputting the data into a first filter, and determining an expected value and a tail value; the expected value is used to represent the label value determined based on data continuity, and the tail value is used to represent the earliest valid label value in a second filter; if the label value is equal to the expected value, the data is filtered by the first filter to determine target data after filtering; the first filter is used to filter data with continuous label values; or, if the label value is not equal to the expected value, and the label value is greater than or equal to the tail value, the data is input into the second filter, and the data is filtered by the second filter to determine target data after filtering; the second filter is used to filter data corresponding to non-continuous label values. The embodiment of the present application determines the label value of the data by marking the data, and then filters the data by the first filter and the second filter, without additional deployment, and without damaging the original network data packets. Therefore, the embodiment of the present application is beneficial to improving data processing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a flowchart of an embodiment of the data filtering method provided by the present application;

[0034] Figure 2 is an architectural schematic diagram of an embodiment of the filter provided by the present application;

[0035] Figure 3 is an architectural schematic diagram of an embodiment of the second filter provided by the present application;

[0036] Figure 4 is a flowchart of an embodiment of the filtering process of the second filter provided by the present application;

[0037] Figure 5 is a flowchart of another embodiment of the filtering process of the second filter provided by the present application;

[0038] Figure 6 is a flowchart of an embodiment of the filtering process of the first filter provided by the present application;

[0039] Figure 7 is a flowchart of an embodiment of the data filtering process provided by the present application;

[0040] Figure 8is a filtering effect diagram of an embodiment of the data filtering process provided by the present application;

[0041] Figure 9 is a structural schematic diagram of the data filtering system provided by the embodiment of the present application;

[0042] Figure 10 is a hardware structural schematic diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description relates to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0044] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0045] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0047] Before the embodiments of the present application are described in detail, first, some nouns and terms involved in the embodiments of the present application are described, and the nouns and terms involved in the embodiments of the present application are applicable to the following explanations.

[0048] Ring queue: a ring-shaped data storage structure.

[0049] bitmap: a data structure based on bit storage.

[0050] In the related art, the duplicate data filtering application is a tool for identifying and deleting duplicate records in a data set. It can be applied to various data types, including text, numbers, images, etc. Duplicate data refers to the existence of multiple identical or similar records in a data set, which may be caused by data input errors, system failures or other reasons. The main goal of duplicate data filtering application is to improve data quality and accuracy. By detecting and deleting duplicate records, data redundancy can be reduced, and data analysis and processing efficiency can be improved. In addition, duplicate data filtering application can also help protect the consistency and integrity of data, ensuring that the information in the data set is accurate and reliable. Duplicate data filtering applications are usually based on algorithms and techniques to identify duplicate records. These algorithms can compare different records in a data set and determine their similarity based on predefined similarity metrics. However, the above algorithms affect the forwarding performance of data, resulting in reduced processing efficiency of data.

[0051] It can be understood that once duplicate records are identified, the application can take appropriate actions such as deleting, merging or marking duplicate records. Duplicate data filtering applications have wide applications in various fields, including database management, data cleaning, data mining and data analysis. They can help organizations and individuals better manage and utilize data resources, improve data quality and credibility. In many application scenarios, duplicate data filtering is crucial to ensure that the terminal does not receive duplicate data.

[0052] Therefore, in order to achieve filtering of duplicate data without losing data forwarding performance, the data filtering method provided in the embodiments of the present application realizes the functions of filtering of duplicate data of network packets, elimination of timeout data, etc.

[0053] The data filtering method provided in the embodiments of the present application relates to the technical field of network communication. The data filtering method provided in the embodiments of the present application can be applied in a terminal, can be applied in a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, etc., but is not limited thereto; the server end can be configured as a standalone physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms, and the server can also be a node server in a blockchain network; the software can be an application that implements the data filtering method, etc., but is not limited to the above forms.

[0054] The application is operable in a variety of general purpose or special purpose computer systems environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0055] It should be noted that in each of the specific embodiments of the present application, when it is necessary to process relevant data related to the identity or characteristics of the user according to user information, user behavior data, user history data, and user location information, the user's permission or consent is obtained first, and the collection, use, and processing of these data comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user is obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally operate is obtained.

[0056] Figure 1 is an optional flowchart of the data filtering method provided by the embodiments of the present application. Figure 2 The method in can include but is not limited to including steps S100 to S400.

[0057] Step S100, receiving data to be filtered, and determining a label value of the data;

[0058] Step S200, inputting the data into a first filter, and determining an expected value and a tail value; the expected value is used to represent the label value determined based on data continuity, and the tail value is used to represent the earliest valid label value in a second filter;

[0059] Step S300, if the label value is equal to the expected value, filtering the data by the first filter to determine target data after filtering; the first filter is used to filter data with continuous label values;

[0060] Alternatively, in step S400, if the label value is not equal to the expected value, and the label value is greater than or equal to the tail value, the data is input into the secondary filter, and the data is filtered by the secondary filter to determine the filtered target data; the secondary filter is used to filter the data corresponding to non-continuous label values.

[0061] In this embodiment, the data tag value is used to distinguish different data. In some embodiments, the network data tagging method in this embodiment can be: the tag data is attached after the network Layer 2 data frame; that is, the tag value in this embodiment can be determined after the network Layer 2 data frame; this does not destroy the original network data packet format and improves the efficiency of data filtering. It is understood that the prerequisite for data filtering is to tag the data packets, and then at the receiving end, multiple data streams are aggregated into one data stream, duplicate data is removed, and timed-out data is removed. In order to mark the uniqueness of the data packets, by way of example, the present invention tags the data, and the same data will be tagged with the same tag (the same ID) to mark the uniqueness of the data. The data format of the tag is: message ID (4 bytes). The message ID uses an unsigned 32-bit integer data type. It should be noted that the first-level filter in this embodiment is a circular data structure. The first-level filter includes the expected value and the tail value to realize the recording and filtering of tag values.

[0062] It is understood that the data filtering method in the embodiments of the present invention refers to... Figure 2 The filter structure provided in this embodiment, as shown, consists of two parts: a primary filter and a secondary filter. The primary filter filters continuous data, i.e., data with consecutive data tag values, such as a current data value of 10 followed by a tag value of 11. This type of data is filtered first by the primary filter. The secondary filter in this embodiment filters non-continuous data, i.e., data with non-continuous data tags. For example, if the current data tag value is 10 and the next is 20, it indicates that 10 data points between the two tags have not been filtered. In this case, this embodiment filters this data through the secondary filter. This embodiment achieves data filtering through a two-level filter structure. The filtering method provided in this embodiment is implemented in user space, requiring no additional deployment or maintenance, thus improving data processing efficiency.

[0063] The data after being labeled is filtered by the data filtering module, and then enters a first filter. The first filter is composed of an expected value counter (expert), a history value counter (last), and a tail value counter (tail). The first filter compares the data label value with the expected value, and if they are the same, the expected value is increased, and the next data is filtered. If the label value is equal to the expected value, it is determined that the target data includes the data, which is used to represent the data currently being filtered in the embodiment of the application.

[0064] In some embodiments, referring to the architecture of the second filter shown in Figure 3 and the filtering process of the second filter shown in Figure 4 The second filter includes a plurality of filtering nodes, and each filtering node includes a boundary record and a marking unit. The filtering process of the second filter includes the following steps.

[0065] In step S410, it is determined whether the label value is registered in the second filter according to the boundary record and the label value.

[0066] In step S420, if the label value is registered in the second filter, a first boundary corresponding to the label value is determined, and the data is filtered according to the first marking unit to determine the target data. The first filtering node includes the first boundary record and the first marking unit.

[0067] Alternatively, in step S430, if the label value is not registered in the second filter, a second filtering node is registered in the second filter according to the label value.

[0068] In some possible implementations, the boundary record in the embodiment of the application is used to represent the boundary of the label value recorded by the filtering node, and the marking unit is used to mark the label value recorded by the filtering node. The second filter in the application can include a plurality of filtering nodes, and the embodiment of the application can logically control X filtering nodes through a ring queue in the second filter, as shown in Figure 3

[0069] The second filter in the embodiment of the application is used to filter data that cannot be filtered by the first filter. As described above, when the received data label value is not the expected value and is greater than the tail value, the data will enter the second filter. As shown in Figure 2 The second filter can be composed of filtering nodes and a ring queue, wherein the filtering node is composed of a boundary record, a timer, and a bitmap marking unit. The ring queue SlaveRring is used to manage the filtering nodes. As shown in Figure 5 ​In an embodiment, the secondary filter includes a boundary record, which is composed of a maximum tag value (min, i.e., the first boundary value in the embodiment) and a minimum tag value (max, i.e., the second boundary value in the embodiment), for recording the boundary value of the filter node. A timer records the aging time of the node, and the filter node is removed when the aging time exceeds a certain time. A bitmap marking unit is a 128-bit bitmap, which marks the data tag value recorded in the current filter node. For example, 1 indicates that the data has been filtered, and 0 indicates that the data has not been filtered. In an embodiment, after the data enters the secondary filter, it is checked whether the data is registered in the filter node. If not, the filter node information needs to be registered first, otherwise, the data tag value is used to filter in the filter node. After the data is filtered by the primary filter and the secondary filter, the repeated data in a certain time range is filtered. It can be understood that if the tag value is registered in the secondary filter, the first boundary to which the tag value belongs is determined, and the data is filtered according to the first marking unit corresponding to the first boundary. If the tag value is not registered in the secondary filter, a new second filter node is registered, which is used to represent the unregistered filter node in the embodiment.

[0070] In some embodiments, the method provided by the embodiment of the application, the second filter node includes a second boundary record and a second marking unit; and registering the second filter node in the secondary filter according to the tag value includes:

[0071] According to the current expected value, the first boundary value is determined;

[0072] According to the tag value, the second boundary value is determined; the first boundary value and the second boundary value are two interval endpoints of the second boundary record;

[0073] According to the first boundary value and the second boundary value, the second marking unit is determined.

[0074] In some possible implementations, the functions of the secondary filter include node registration, data filtering, and timeout aging. When the data enters the secondary filter, it is first determined whether the filter node needs to be registered, and the relevant information is registered, so that the subsequent data tag filtering can be completed. It can be understood that if the current tag value is not registered in the secondary filter, a free filter node (i.e., the second filter node in the embodiment, which is a free filter node) needs to be applied from the ring queue first, and then the data registration is completed. The data registration needs to register the following information:

[0075] The minimum boundary value (i.e., the first boundary value in the embodiment of the present application): the current expected value (not received). The maximum boundary value (i.e., the second boundary value in the embodiment of the present application): the received data tag value. The timer value: the receiving time of the current data. The bitmap flag (i.e., the second flag unit in the embodiment of the present application): the data tag registration is completed in the table, and the registration value = setBitmap (tag value - minimum boundary value).

[0076] When entering the secondary filter, it is checked whether the data tag has been registered. If registered, the data tag is matched with the registered node boundary value one by one to determine the first boundary corresponding to the tag value, and then subsequent filtering processing is performed.

[0077] In some embodiments, the method provided by the embodiment of the present application comprises:

[0078] If the tag value has been registered in the first flag unit, the data is discarded, and it is determined that the target data does not include the data.

[0079] Or, if the tag value has not been registered in the first flag unit, the tag value is registered in the first flag unit according to the first boundary value and the tag value, and it is determined that the target data includes the data.

[0080] In some possible implementation manners, when a suitable filtering node is matched, it is first checked whether the data is registered in the bitmap registration table. That is, if the first filtering node is matched, it is judged by the first flag unit of the first filtering node whether the current tag value has been registered. If not registered, the value is registered; otherwise, the data is discarded, and the secondary filtering of the data is completed.

[0081] In some embodiments, the method provided by the embodiment of the present application, the filtering node further comprises a timer, the timer starts timing from the establishment of the filtering node, and the method further comprises:

[0082] If the duration of the timer is greater than a preset duration, the filtering node is deleted; the preset duration is used to represent the aging time of the filtering node.

[0083] In some possible implementation manners, since the data to be filtered itself has timeliness, the aging problem of the filtering node needs to be considered. When the timer is timed out, the filtering node is deleted from the ring queue SlaveRing. First, when the data tag is registered, the receiving time of the data is registered, so that each filtering node has its maximum timeout time. The secondary filter deletes the timeout node in the ring queue every certain time (for example, 500 ms), and completes the aging of the data.

[0084] Therefore, the secondary filtering of the embodiment of the present application filters data slightly complicatedly, and the core idea is: registering information first, matching information then, and checking data aging periodically.

[0085] In some embodiments, the method provided by the embodiment of the present application includes a history value, the history value being a valid tag value received last time and not filtered, and the method further includes:

[0086] If the tag value is equal to the history value, discarding the data and determining that the target data does not include the data;

[0087] Or, if the tag value is less than the tail value, discarding the data and determining that the target data does not include the data.

[0088] In some possible implementation manners, if the data tag and the history value are the same, the data is discarded, indicating that the current data is repeated with the history data; if the data tag is greater than the expected value, the secondary filtering is entered; if the data tag is less than the history value and greater than or equal to the tail value, the secondary filtering is entered; if the data tag is less than the tail value, the data is lost (indicating that the current data is timed out). As can be seen, the primary filter can filter most of the repeated data. Only a small part of data enters the secondary filter due to link delay.

[0089] The primary filter in the embodiment of the present application only needs to use three 32-bit unsigned integer data (expert, last, tail) in program design. Referring to Figure 6 An embodiment shown in the figure, the expected value counter (expert) in the embodiment of the present application: the expected value is a data tag value that the primary filter wants to receive, when the received data tag and the expected value are the same, the data is discarded and the value is incremented. The history value counter (last): the history value refers to the last received valid data tag value that is not filtered. The tail counter (tail): the tail value refers to the last data tag value to be filtered. If the secondary filter is empty, the value is the expected value, otherwise it is the last valid data tag value of the secondary filter node.

[0090] The reason why the primary filter in the embodiment of the present application uses 32-bit unsigned integer data is that it is a ring data structure itself. When the maximum value is reached, it starts counting from 0. And the difference between the previous value and the later value is always the difference between the two numbers. It is equivalent to a ring counter.

[0091] In some embodiments, the method provided by the embodiment of the present application further includes:

[0092] If the secondary filter is not empty, the tail value is determined to be the earliest valid tag value in the secondary filter;

[0093] Or, if the secondary filter is empty, the tail value is determined to be the expected value.

[0094] The following, with reference to Figure 7 The scheme of the embodiment of the present application is described and explained in detail in combination with a specific application example as shown in the following:

[0095] The first process shows that the primary filter has started filtering data, and the secondary filter has registered 2 filtering nodes (i.e. node0 and node1), at this time, the tail value is the first boundary corresponding to the node0 filtering node registered earliest. Figure 7 The second process in the figure shows that a filtering node (the node0 filtering node) has failed and is removed, at this time, the tail value is the first boundary corresponding to the node1 filtering node registered earliest; and the expected value of the primary filter has overflowed, according to the characteristics of the ring data structure, the expected value is redefined as the tag value from the tag 0. Figure 7 The third process in the figure shows that the node1 filtering node of the secondary filter has failed and is removed, there is no valid filtering node in the secondary filter, at this time, the secondary filter does not work, therefore, the tail value is the expected value. At this time, the primary filter normally filters data. Referring to Figure 8 As shown in the figure, the data filtering method provided by the embodiment of the present application filters data, Figure 8 In the figure, different data is represented by different symbols, the same data is discarded by the primary filter and the secondary filter, and non-repeated data is obtained. Figure 7 In the figure, different data is represented by different symbols.

[0096] It can be understood that, at the data sending side, the data marking application software is integrated to complete marking of data. At the data receiving side, the data filter software is integrated, which completes filtering of repeated data and invalid data out of time. After the two are integrated, the data filtering function is completed.

[0097] The embodiment of the present application can implement the above method based on the linux user state, specifically, the network data marking manner: the tag data is marked behind the network layer 2 data frame; the specific format of the data tag: device identification+state identification+frame ID+magic number.

[0098] Compared with the existing kernel state implementation, the embodiment of the present application improves the portability and deployability of the multi-transmit and selective-receive function, improves the product value, and reduces the maintenance and deployment cost. Once the software fails, it only needs to be restarted, without the need to restart the entire system, and the network can be quickly recovered. While marking the data with a tag, the original network layer 2 data packet is not damaged.

[0099] The data filtering method provided in the embodiments of the present application comprises the following steps: receiving data to be filtered, and determining a label value of the data; inputting the data into a first filter, and determining an expected value and a tail value; the expected value is used to represent the label value determined based on data continuity, and the tail value is used to represent the earliest valid label value in a second filter; if the label value is equal to the expected value, filtering the data by using the first filter to determine target data after filtering; the first filter is used to filter data with continuous label values; or, if the label value is not equal to the expected value and the label value is greater than or equal to the tail value, inputting the data into the second filter, filtering the data by using the second filter to determine the target data after filtering; the second filter is used to filter data corresponding to non-continuous label values. The embodiments of the present application determine the label value of the data by marking the data, and then filter the data by using the first filter and the second filter, without the need for additional deployment and without damaging the data packets of the original network, and therefore, the embodiments of the present application are beneficial to improving the data processing efficiency.

[0100] Please refer to Figure 9 The embodiments of the present application also provide a data filtering system, which can implement the above data filtering method, and the system comprises:

[0101] The first module 810 is configured to receive data to be filtered, and determine a label value of the data.

[0102] The second module 820 is configured to input the data into a first filter, and determine an expected value and a tail value; the expected value is used to represent the label value determined based on data continuity, and the tail value is used to represent the earliest valid label value in a second filter.

[0103] The third module 830 is configured to, if the label value is equal to the expected value, filter the data by using the first filter to determine target data after filtering; the first filter is used to filter data with continuous label values.

[0104] Or,

[0105] The fourth module 840 is configured to, if the label value is not equal to the expected value and the label value is greater than or equal to the tail value, input the data into the second filter, filter the data by using the second filter to determine the target data after filtering; the second filter is used to filter data corresponding to non-continuous label values.

[0106] In some embodiments, the system provided in the embodiments of the present application further comprises a timer, which starts timing from the establishment of the filtering node; the system further comprises a sixth module configured to delete the filtering node if the length of time of the timer is greater than a preset length of time; the preset length of time is used to represent the aging time of the filtering node.

[0107] In some embodiments, the system provided by the embodiments of the present application includes a first filter, the first filter includes a history value, the history value being a last received and unfiltered valid tag value; the system further includes a seventh module configured to discard the data and determine that the target data does not include the data if the tag value is equal to the history value.

[0108] Alternatively, if the tag value is less than the tail value, the data is discarded and it is determined that the target data does not include the data.

[0109] In some embodiments, the system provided by the embodiments of the present application further includes an eighth module configured to determine the tail value as the earliest valid tag value in the second filter if the second filter is not empty.

[0110] Alternatively, if the second filter is empty, the tail value is determined as the expected value.

[0111] It can be understood that the contents in the above method embodiments are applicable to the system embodiments, the system embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0112] The embodiments of the present application further provide an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the above data filtering method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0113] It can be understood that the contents in the above method embodiments are applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0114] Please refer to Figure 10 , Figure 10 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0115] The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0116] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the data filtering method of the embodiments of the present application.

[0117] The input / output interface 903 is configured to realize information input and output.

[0118] The communication interface 904 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0119] The bus 905 is configured to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0120] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize the communication connection between the device.

[0121] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-mentioned data filtering method.

[0122] It can be understood that the contents in the above method embodiments are applicable to the present storage medium embodiments. The functions implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved by the present storage medium embodiments are the same as those of the above method embodiments.

[0123] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor. These remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0124] The embodiments described in the specification are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0125] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0126] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0127] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0128] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0129] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0130] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed mutual ones can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0131] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0132] In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0133] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0134] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A data filtering method, characterized in that, The method includes: Receive the data to be filtered and determine the label value of the data; The data is input into the first-level filter to determine the expected value and the tail value; the expected value is used to characterize the label value determined based on data continuity, and the tail value is used to characterize the earliest valid label value in the second-level filter; If the label value equals the expected value, the data is filtered by the first-level filter to determine the filtered target data; the first-level filter is used to filter data with continuous label values. If the label value is not equal to the expected value, and the label value is greater than or equal to the tail value, the data is input into the secondary filter, and the data is filtered by the secondary filter to determine the filtered target data; the secondary filter is used to filter the data corresponding to non-continuous label values.

2. The method according to claim 1, characterized in that, The secondary filter includes several filtering nodes, each including a boundary record and a tagging unit; the filtering process of the data through the secondary filter includes: Based on the boundary record and the tag value, determine whether the tag value is registered in the secondary filter; If the tag value is registered in the secondary filter, a first boundary corresponding to the tag value is determined; and the data is filtered according to the first marking unit to determine the target data; wherein, the first filtering node includes a first boundary record and a first marking unit; If the tag value is not registered in the secondary filter, a second filter node is registered in the secondary filter based on the tag value.

3. The method according to claim 2, characterized in that, The second filter node includes a second boundary record and a second tagging unit; registering the second filter node in the secondary filter according to the tag value includes: Determine the first boundary value based on the current expected value; The second boundary value is determined based on the label value; the first boundary value and the second boundary value are the two interval endpoints of the second boundary record; The second marking unit is determined based on the first boundary value and the second boundary value.

4. The method according to claim 2, characterized in that, The filtering process based on the first tagging unit includes: If the tag value has already been registered in the first tagging unit, discard the data and determine that the target data does not include the data. If the tag value has not been registered in the first tagging unit, the tag value is registered in the first tagging unit based on the first boundary value and the tag value, and the target data is determined to include the data.

5. The method according to claim 2, characterized in that, The filtering node further includes a timer, the timer starting to count when the filtering node is created; the method further includes: If the duration of the timer is greater than the preset duration, the filter node is deleted; the preset duration is used to characterize the aging time of the filter node.

6. The method according to any one of claims 1 to 5, characterized in that, The primary filter includes historical values, which are the previously received, unfiltered, valid tag values; the method further includes: If the label value is equal to the historical value, discard the data, and determine that the target data does not include the data. If the tag value is less than the tail value, the data is discarded, and it is determined that the target data does not include the data.

7. The method according to any one of claims 1 to 5, characterized in that, The method further includes: If the secondary filter is not empty, the tail value is determined to be the earliest valid label value in the secondary filter; If the secondary filter is empty, the tail value is determined to be the expected value.

8. A data filtering system, characterized in that, The system includes: The first module is used to receive the data to be filtered and determine the label value of the data; The second module is used to input the data into the first-level filter to determine the expected value and the tail value; the expected value is used to characterize the label value determined based on data continuity, and the tail value is used to characterize the earliest valid label value in the second-level filter; The third module is used to filter the data through the first-level filter if the label value is equal to the expected value, and to determine the filtered target data; the first-level filter is used to filter data with continuous label values. The fourth module is used to input the data into the secondary filter if the label value is not equal to the expected value and the label value is greater than or equal to the tail value, and to filter the data through the secondary filter to determine the filtered target data; the secondary filter is used to filter the data corresponding to non-continuous label values.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images