Data filtering method, electronic equipment and storage medium

Abstract

The invention discloses a data filtering method, electronic equipment and a storage medium, and relates to the technical field of data processing. The data filtering method comprises the following steps: acquiring an initial filtering field and to-be-transmitted data in a to-be-transmitted message, and performing standardization operation on the initial filtering field to obtain a to-be-filtered field; filtering the to-be-filtered field based on a preset dictionary tree and a preset bitmap, and determining a target transmission message; and converting the target transmission message into a standard message through a preset operator protocol, and transmitting the standard message to an operator, so that the operator sends the to-be-transmitted data to target terminal equipment. According to the embodiment of the invention, memory compression and efficient comparison query under a multi-dimensional filtering condition or a large data volume filtering condition are realized, and compared with common methods such as external database query, Bloom filter and hash table search, IO interaction delay is reduced, query accuracy is enhanced, and memory use space is saved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data filtering method, electronic equipment and storage medium. Background Art

[0002] In today's business environment, companies are facing increasingly fierce market competition and evolving customer demands. SMS, as a low-cost, timely, and high-reach marketing and promotional tool, has been recognized by many companies as a crucial component of their customer acquisition and retention strategies. SMS blacklist filtering, as a fundamental component of SMS delivery, can help companies effectively mitigate risks, reduce customer complaints, and improve the accuracy and efficiency of SMS notifications, marketing promotions, and production emergency response, thereby reducing service costs and improving service quality.

[0003] As a financial institution, commercial banks provide SMS services across provinces nationwide, covering a wide range of services, including account activation, marketing, notifications, and verification codes. However, some customers may not want to receive certain types of SMS (for example, some users may not want marketing messages but want account activation messages; some customers may only be interested in marketing in a certain area but not in other areas). Furthermore, when a carrier outage occurs in a particular city, SMS messages may not reach customers in a timely manner, necessitating subsequent on-demand resupply. For example, if a carrier outage occurs in a particular province or city, all SMS messages from that carrier are unable to be delivered to customers. When banks provide reimbursement SMS after the carrier is restored, they typically only provide account activation messages and some notifications, such as debit card account activation messages and agreed repayment notifications, but not verification code messages. Verification code messages are time-sensitive, and expired verification code messages are useless and can cause inconvenience to users. Currently, flexible conditional blacklist message filtering typically relies on external databases or middleware. Conditional filtering involves frequent input / output (IO) interactions, and multiple similar conditions may be repeatedly compared, resulting in long response times. Improving processing efficiency relies on horizontal capacity expansion. Therefore, how to filter messages efficiently has become a problem that needs to be solved urgently. Summary of the Invention

[0004] The present invention provides a data filtering method, electronic device and storage medium to solve the problems in the prior art of frequent data filtering interactions, repeated comparisons of multiple similar conditions, resulting in long response times, and the reliance on horizontal expansion to enhance processing efficiency.

[0005] According to one aspect of the present invention, a data filtering method is provided, wherein the method comprises:

[0006] Acquire an initial filtering field and data to be transmitted in a message to be transmitted, and perform a standardization operation on the initial filtering field to obtain a field to be filtered;

[0007] filter the to-be-filtered field based on a preset dictionary tree and a preset bitmap, to determine a target transmission message;

[0008] convert the target transmission message into a standard message through a preset operator protocol, and transmit the standard message to an operator, so that the operator sends the to-be-transmitted data to a target terminal device.

[0009] According to another aspect of the present application, a data filtering device is provided, wherein the device comprises:

[0010] a field determining module configured to acquire an initial filtering field and to-be-transmitted data in a to-be-transmitted message, and perform a standardization operation on the initial filtering field to obtain a to-be-filtered field;

[0011] a data filtering module configured to filter the to-be-filtered field based on a preset dictionary tree and a preset bitmap, to determine a target transmission message;

[0012] a data transmission module configured to convert the target transmission message into a standard message through a preset operator protocol, and transmit the standard message to an operator, so that the operator sends the to-be-transmitted data to a target terminal device.

[0013] According to another aspect of the present application, an electronic device is provided, wherein the electronic device comprises:

[0014] at least one processor; and

[0015] a memory connected to the at least one processor in communication; wherein

[0016] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute a data filtering method according to any one of the embodiments of the present application.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, wherein the computer readable storage medium stores computer instructions for enabling a processor to implement a data filtering method according to any one of the embodiments of the present application when executed by the processor.

[0018] The technical solution of the embodiment of the present invention obtains the initial filter field and the data to be transmitted in the message to be transmitted, performs a standardization operation on the initial filter field to obtain the field to be filtered, filters the field to be filtered based on a preset dictionary tree and a preset bitmap, determines the target transmission message, converts the target transmission message into a standard message through a preset operator protocol, and transmits the standard message to the operator so that the operator sends the data to be transmitted to the target terminal device, thereby realizing memory compression and efficient comparison query under multi-dimensional filtering conditions or large data volume filtering conditions. Compared with commonly used external database query, Bloom filter and hash table search methods, it reduces IO interaction delay, enhances query accuracy, and saves memory usage space.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a flow chart of a data filtering method provided according to the first embodiment of the present invention;

[0022] Figure 2 This is an example diagram of a first target field provided according to the first embodiment of the present invention;

[0023] Figure 3 This is an example diagram of a second target field provided according to the first embodiment of the present invention;

[0024] Figure 4 This is an example diagram of a character string provided according to the first embodiment of the present invention;

[0025] Figure 5 This is a flow chart of a data filtering method provided according to the second embodiment of the present invention;

[0026] Figure 6 This is an example diagram of a character string provided according to the second embodiment of the present invention;

[0027] Figure 7 This is a structural diagram of a data filtering system provided according to a third embodiment of the present invention;

[0028] Figure 8This is a workflow diagram of a data filtering system provided according to the third embodiment of the present invention;

[0029] Figure 9 This is a workflow diagram of a conditional parsing unit provided according to the third embodiment of the present invention;

[0030] Figure 10 This is a schematic structural diagram of a building unit provided according to Embodiment 3 of the present invention;

[0031] Figure 11 This is a structural diagram of a data filtering device provided according to a fourth embodiment of the present invention;

[0032] Figure 12 The figure is a schematic structural diagram of an electronic device for implementing a data filtering method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] The acquisition, storage, use, and processing of data in this application's technical solution comply with relevant national laws and regulations.

[0036] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0037] Example 1

[0038] Figure 1 This is a flow chart of a data filtering method provided according to the first embodiment of the present invention. This embodiment is applicable to the case where a message to be transmitted is filtered. The method can be executed by a data filtering device. The data filtering device can be implemented in the form of hardware and / or software. The data filtering device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0039] S110 , obtaining an initial filtering field and data to be transmitted in a message to be transmitted, and performing a standardization operation on the initial filtering field to obtain a field to be filtered.

[0040] The message to be transmitted can be understood as a message that needs to be transmitted, containing the specific content to be conveyed and necessary control information. Generally speaking, the message to be transmitted can include an initial filtering field and data to be transmitted. In actual applications, the message to be transmitted can be a message generated according to business needs. For example, the message to be transmitted can be a text message. In one embodiment, the message to be transmitted can be a message transmitted upstream or a pre-stored message to be resent. The initial filtering field can be understood as the field information used for filtering, and the data to be transmitted can be understood as the key data information to be transmitted in the message to be transmitted. Standardization operation refers to the process of performing standardized execution on the message to be transmitted. For example, standardization operation may include but is not limited to determining whether the input filtering condition contains the logical symbol "OR". If so, splitting the filtering condition into multiple conditions containing only "AND". The field to be filtered can be understood as the initial filtering field after the standardization operation. In one embodiment, the field to be filtered can include one or more fields.

[0041] In an embodiment, the initial filter field and the data to be transmitted in the message to be transmitted can be extracted according to the field position or field information. The logic symbol field contained in the initial filter field is determined. If the logic symbol "OR" exists, the initial filter field is split into multiple conditions containing only "AND". If the logic symbol "OR" does not exist, the initial filter field is used as the field to be filtered, so that each field to be filtered can be filtered one by one.

[0042] S120: Filter the to-be-filtered fields based on the preset dictionary tree and the preset bitmap to determine a target transmission message.

[0043] The pre-set dictionary tree, also known as a prefix tree, is a variant of the hash tree and can be used to count, sort, and store large numbers of strings (but not limited to strings). It is often used by search engines for word frequency statistics. It utilizes common prefixes to reduce query time and minimize unnecessary string comparisons, resulting in higher query efficiency than a hash tree.

[0044] The preset bitmap can be understood as a bitmap that pre-stores the filtering conditions. It is a compact data structure used to represent the state of elements (presence or absence) in a set or sequence of fixed size. The bitmap uses a bit array to represent the state of elements in the set, and each element corresponds to a bit, thereby achieving efficient space and time performance. In one embodiment, the preset bitmap may include a multi-index map and a single-index map, and both the multi-index map and the single-index map can be a bitmap. The target transmission message can be understood as a data message generated by splicing the filtered field to be filtered and the data to be transmitted.

[0045] In an embodiment, the to-be-filtered field can be filtered according to a preset dictionary tree and a preset bitmap, respectively, to obtain the filtered to-be-filtered field as the target field, and the target field and the data to be transmitted can be concatenated to form the target transmission message. In actual operation, the to-be-filtered field can be filtered according to the preset dictionary tree first, and if the to-be-filtered field is not fully matched, the to-be-filtered field after the preset dictionary tree is filtered by the preset bitmap; or alternatively, the to-be-filtered field can be filtered according to the preset bitmap first, and if the to-be-filtered field is not fully matched, the to-be-filtered field after the preset dictionary tree is filtered by the preset dictionary tree, thereby improving filtering efficiency.

[0046] S130: Convert the target transmission message into a standard message through a preset operator protocol, and transmit the standard message to the operator, so that the operator sends the data to be transmitted to the target terminal device.

[0047] The preset operator protocol can be understood as a communication transmission protocol adopted by the operator, such as Hyper Text Transfer Protocol (HTTP), etc. The target terminal device can be understood as a user who needs to receive standard messages.

[0048] In an embodiment, the target transmission message may be converted into a standard message according to a preset operator protocol, and the standard message may be pushed to the operator so that the operator can send the data to be transmitted to the target terminal device.

[0049] An embodiment of the present invention obtains an initial filter field and data to be transmitted in a message to be transmitted, performs a standardization operation on the initial filter field to obtain a field to be filtered, filters the field to be filtered based on a preset dictionary tree and a preset bitmap, determines a target transmission message, converts the target transmission message into a standard message through a preset operator protocol, and transmits the standard message to the operator so that the operator sends the data to be transmitted to the target terminal device, thereby realizing memory compression and efficient comparison query under multi-dimensional filtering conditions or large data volume filtering conditions. Compared with commonly used external database query, Bloom filter and hash table search methods, it reduces IO interaction delay, enhances query accuracy, and saves memory usage space.

[0050] In one embodiment, the process of constructing the preset dictionary tree and the preset bitmap includes:

[0051] receiving a filter condition modification instruction, extracting a logic symbol field of the filter condition modification instruction, and splitting the filter condition modification instruction into target filter condition modification instructions according to the logic symbol field;

[0052] Extract the instruction type, limited condition field, and unlimited condition field of the target filter condition modification instruction;

[0053] When it is determined that there is an infinite condition field in the target filtering condition, updating the preset bitmap based on the infinite condition field;

[0054] When it is determined that the infinite condition field does not exist in the target filtering condition, the preset dictionary tree is updated based on the infinite condition field.

[0055] The filtering condition modification instruction can be understood as instruction information for updating the filtering condition. Exemplarily, the filtering condition modification instruction can include adding, deleting, or modifying the filtering condition. In actual application, the filtering condition modification instruction is a JSON message, which can include an instruction type, a limited condition field, and an unlimited condition field. The instruction type updates a preset bitmap or a preset dictionary tree type, and the instruction type can include insert (Insert), delete (Delete), and update (Update). The limited condition field and the unlimited condition field can be fields divided according to a value range of the field. If a set of a filtering dimension is limited, it is referred to as a limited condition field, and if a filtering dimension condition has a large number of possibilities or cannot be determined how many possibilities there are, it is referred to as an unlimited condition field. Generally, the limited condition field and the unlimited condition field can be preset according to business requirements. Exemplarily, the limited condition field can include, but is not limited to, province and city, operator, product number, and product type, and the unlimited condition field can include, but is not limited to, a mobile phone number dimension. The logical symbol field can be understood as field information indicating a logical symbol. Exemplarily, the logical symbol can include "and" and "or".

[0056] In an embodiment, a filter condition modification instruction inputted by a front end or an interface can be received, the logical symbol field of the filter condition modification instruction can be extracted, the filter condition modification instruction can be split according to the logical symbol field, and the split filter condition modification instruction can be used as the target filter condition modification instruction. For example, it can be determined whether the input filter condition modification instruction contains the logical symbol "or". If so, the filter condition modification instruction can be split into multiple conditions containing only "and". The type of instruction added, deleted or modified in the target filter condition modification instruction is extracted; limited condition fields such as province, city, operator, product number and product type and unlimited condition fields such as mobile phone number dimension. If there is an unlimited condition field in the target filter condition, the preset bitmap is updated according to the unlimited condition field; if there is no unlimited condition field in the target filter condition, the preset dictionary tree is updated according to the unlimited condition field. In the actual operation process, the instruction type, limited condition field and unlimited condition field of the target filter condition modification instruction all have corresponding field information, and the value corresponding to each field information can be extracted to realize the acquisition of the instruction type, limited condition field and unlimited condition field. In one embodiment, finite conditional filters and infinite conditional filters can be used to filter finite conditional fields and infinite conditional fields, respectively. Dimensional filters are in an "AND" relationship with each other, and multiple values ​​in an "OR" relationship can be selected within each dimension filter value. When there are multiple target filter condition modification instructions, each instruction can be determined to determine whether it contains an infinite conditional field and a finite conditional field, thereby determining the corresponding values ​​of the infinite conditional field and the finite conditional field.

[0057] In one embodiment, updating a preset bitmap based on an infinite condition field includes:

[0058] Determine whether there are limited condition fields in the target filter conditions, as well as the preset position fields of unlimited condition fields;

[0059] If it exists, concatenate the finite condition field and the infinite condition field as the first target field, and transmit the first target field to the single index map builder, so that the single index map builder updates the single index map in the preset bitmap according to the first target field; wherein the finite condition field and the preset position field in the first target field are concatenated as the index, and the remaining fields of the infinite condition field are used as the value;

[0060] If it does not exist, the infinite condition fields are spliced ​​as the second target field, and the second target field is transmitted to the graph builder so that the graph builder updates the multi-index graph in the preset bitmap according to the second target field; wherein the preset position fields in the second target field are spliced ​​as indexes, and the remaining fields of the infinite condition fields are used as values.

[0061] Among them, the preset location field can be understood as the field information corresponding to the preset location in the infinite condition field. The preset location field can be set according to business needs. For example, when the infinite condition field is a mobile phone number, the preset location field can be the first 3 digits of the mobile phone number. The value refers to the value of the index corresponding to the target filtering condition. For example, a mobile phone number is usually 11 digits from 0 to 9, and can contain up to 10 11 So you can use up to 10 11 A binary number represents the presence or absence of all mobile phone numbers in this set. If a mobile phone number exists, the corresponding position is marked as 1, and if it does not exist, it is marked as 0. 0 and 1 are the values.

[0062] In an embodiment, the preset position field of the infinite condition field can be extracted, and whether the target filtering condition contains the finite condition field can be determined one by one, that is, when the finite condition field is not empty, it is determined that there is a finite condition field in the target filtering condition. If there is a finite condition field, the finite condition field and the infinite condition field can be spliced ​​as the first target field. The finite condition field and the preset position field are spliced ​​as the index of the first target field, and the remaining fields of the infinite condition field are used as the value of the first target field. If there is no finite condition field, the infinite condition field can be spliced ​​as the second target field, the preset position field is used as the index of the second target field, and the remaining fields of the infinite condition field are used as the value of the second target field. In one embodiment, Figure 2 It is an example diagram of a first target field provided according to the first embodiment of the present invention, and is a standard format for inputting a diagram builder when both the finite condition field and the infinite condition field are not empty. Figure 2 Take the example of two finite condition fields, and concatenate finite condition 1, finite condition 2 and infinite condition as the first target field. In the actual operation process, the concatenation order of the finite condition fields can be pre-set. In one embodiment, Figure 3 This is an example diagram of a second target field provided according to the first embodiment of the present invention, which is a standard format for inputting a diagram builder when all infinite condition fields are not empty, such as Figure 3 As shown, the infinite condition fields are concatenated as the second target field.

[0063] In an embodiment, the graph constructor can include a single-index graph constructor and a multi-index graph constructor, both of which can construct a segmented roar bitmap. The single-index graph constructor is used to update a single-index graph, and the multi-index graph constructor is used to update a multi-index graph. In actual operation, when the single-index graph constructor is added, the preset position field in the unlimited condition field can be extracted as an index, and the remaining fields can be stored as values in the bitmap table of the single-index graph. For example, in a mobile phone number, the first three digits of the mobile phone number are extracted as an index, and digits 4-11 are stored in the bitmap table. The bitmap table automatically performs segmentation and compression on the data, and automatically selects an appropriate compression method when the number of mobile phone numbers increases or decreases. When the single-index graph constructor is deleted, the index and value can be extracted in the same way as the addition, the corresponding bitmap table is searched, and the corresponding segment and position on the segment are queried in the same way as the addition, and are removed. In the input condition of the multi-index graph constructor, each additional dimension of the limited condition field requires an additional layer of index, and the last layer of index is extracted from the preset position field in the unlimited condition field, and the remaining fields are stored as values in the bitmap table. For example, the product number and mobile phone number are included in the limited condition field. The product number is used as the first-level index, and the first three digits of the mobile phone number are extracted as the second-level index. Digits 4-11 are stored in the bitmap table to achieve accurate division of the index graph.

[0064] In an embodiment, updating a preset dictionary tree based on an unlimited condition field includes:

[0065] Determining whether there is a null value in the limited condition field;

[0066] If there is, a wildcard is supplemented according to the field bit number corresponding to the null value;

[0067] The limited condition fields are spliced into a string according to a preset order, and the instruction type and the string are transmitted to a tree constructor to enable the tree constructor to update the preset dictionary tree according to the instruction type and the string.

[0068] The wildcard is a special character or symbol used to replace one or more unknown characters when searching or matching text or files, such as “*” and “#”. The preset order refers to a pre-set arrangement order.

[0069] In an embodiment, it can be determined whether there is a null value in the limited condition field, and the field bit number corresponding to each limited condition field can be determined. If there is a null value, a wildcard is supplemented according to the field bit number. Then the limited condition fields are spliced into a string according to a preset order, and the instruction type and the string are transmitted to a tree constructor to update the preset dictionary tree through the tree constructor. In an embodiment, Figure 4 is an example of a string provided by the first embodiment of the present application, and is a standard format input by the tree constructor, such as Figure 4As shown, the limited condition 1, the limited condition 2 and the limited condition 3 are taken as an example, the limited condition 1, the limited condition 2 and the limited condition 3 are spliced as a string, all bits of the string need to be filled, if a filter does not exist a filtering condition, then fill an equal wildcard. In the actual operation process, the preset dictionary tree modification process is to delete the original string first, and then add a new string. If the instruction type is addition, the first character can be started from the first layer of the tree to see if there is a corresponding match, if there is no corresponding match, a first layer node of the character is added to the first layer node, and then the node is started from the leaf node to recursively remove the nodes without child nodes layer by layer. If the instruction type is deletion, the last character of the string is used to find the sub-tree to the leaf node, and then the leaf node is used to recursively remove the nodes without child nodes layer by layer.

[0070] In an embodiment, after filtering the to-be-filtered field based on the preset dictionary tree and the preset bitmap to determine the target transmission packet, further comprising:

[0071] Determine the filtering mode of filtering the to-be-filtered field based on the preset dictionary tree and the preset bitmap;

[0072] When the filtering mode is storage, store the filtered field and the to-be-transmitted data based on the preset dictionary tree and the preset bitmap as the to-be-retransmitted packet;

[0073] When the filtering mode is discard, discard the filtered field based on the preset dictionary tree and the preset bitmap.

[0074] The filtering mode is used to indicate whether to persistently save the filtered data, and the filtering mode includes storage and discard. The to-be-retransmitted packet refers to a packet that is filtered under the current conditions, but may exist a retransmission condition, and the to-be-retransmitted packet can be stored in the storage unit.

[0075] In an embodiment, the filtering mode of filtering the to-be-filtered field based on the preset dictionary tree and the preset bitmap can be different, and the filtering mode of filtering the to-be-filtered field based on the preset dictionary tree and the preset bitmap can be determined respectively. If the filtering mode is storage, the filtered field and the to-be-transmitted data based on the preset dictionary tree and the preset bitmap can be stored as the to-be-retransmitted packet; if the filtering mode is discard, the filtered field based on the preset dictionary tree and the preset bitmap can be discarded, reducing the storage space occupation.

[0076] In an embodiment, before filtering the to-be-filtered field based on the preset dictionary tree and the preset bitmap to determine the target transmission packet, further comprising:

[0077] Receive a change instruction for updating the filtering mode, and update the filtering mode according to the change instruction.

[0078] In an embodiment, a change instruction inputted from a front end or an interface may be received, and the filtering mode may be updated according to the change instruction. Generally speaking, the filtering mode of whether to persist the filtered data may be indicated by field information. A Boolean value of true or false may be optional, with the default value being false. If true, the filtering mode is storage, i.e., all filtered data is stored in a storage unit. If false, the filtering mode is discard, i.e., the message is directly discarded.

[0079] In one embodiment, to ensure blacklist filtering efficiency and avoid repeated filtering based on duplicate conditions, the filtering conditions in the condition input unit should also be specified and verified based on actual business conditions. This verification can be performed after obtaining the message to be transmitted. For example, the mobile phone number, province, city, and operator should not appear at the same time, because a specific mobile phone number already has a specific province, city, and operator, and the conditions should not be repeated. If duplicate conditions exist, a fault message will be fed back to the front end.

[0080] Example 2

[0081] Figure 5 This is a flow chart of a data filtering method provided according to the second embodiment of the present invention. This embodiment is based on the above embodiment to further optimize and expand, and can be combined with various optional technical solutions in the above embodiment. Figure 5 As shown, the method includes:

[0082] S201: Acquire an initial filter field and data to be transmitted in a message to be transmitted, and extract a logical symbol field of the initial filter field.

[0083] In an embodiment, the initial filter field and the data to be transmitted in the message to be transmitted can be extracted according to the field position or field information, and the logic symbol field of the initial filter field can be determined. In actual high school, the logic symbol field can include "and" and "or".

[0084] S202: When the first symbol field does not exist in the logical symbol field, the initial filtering field is used as the field to be filtered.

[0085] The first symbol field may be “or” (ie “||”).

[0086] In an embodiment, when it is determined that the first symbol field does not exist in the logical symbol field, the initial filtering field may be directly used as the field to be filtered.

[0087] S203. When the first symbol field exists in the logical symbol field, split the initial filtering field until the first symbol field does not exist in the logical symbol field of the initial filtering field, and use the split initial filtering field as the field to be filtered.

[0088] In an embodiment, when it is determined that the first symbol field exists in the logical symbol field, the initial filter field can be split into multiple equivalent inputs that do not contain the first symbol field. Exemplarily, the initial filter field can be split into multiple conditions containing only "and", and the split initial filter field is used as the field to be filtered.

[0089] S204: Extract the limited condition fields from the fields to be filtered, determine whether there are null values ​​in the limited condition fields, and if so, add wildcards to the field digits corresponding to the null values, and concatenate the limited condition fields in a preset order to form a matching string.

[0090] In an embodiment, it is possible to determine whether an infinite condition is included, that is, whether there is a value in all the finite condition fields, and determine the null value in the finite condition fields. If so, the number of field digits corresponding to the null value is determined, and wildcards are added according to the number of field digits. For example, if the number of field digits corresponding to the null value is 2, two wildcards are added in the null value position. The finite condition fields are spliced ​​in a preset order to form a matching string. In one embodiment, Figure 6 is an example diagram of a character string provided according to the second embodiment of the present invention, such as Figure 6 As shown, the limited condition fields include province, city, operator, product number and product type as an example, and the preset order is a string of province, city, operator, product number and product type.

[0091] S205 , matching the preset dictionary tree according to the matching character string. If the match is successful, determining the successfully matched field to be filtered as the filter field.

[0092] In this embodiment, a preset dictionary tree can be matched according to the matching string. Specifically, each bit of the matching string is obtained one by one and matched with the key value of each layer of the preset dictionary tree map. If the same character or wildcard # is matched, its value is obtained, that is, the subtree continues to match. If there is a match up to the end of the character, the match is successful, and the successfully matched field to be filtered is used as the filter field.

[0093] S206: If the match is not successful, extract the limited condition field and the unlimited condition field in the field to be filtered, and determine the preset position field of the unlimited condition field as the first field.

[0094] In an embodiment, if no match is found, the graph filtering can be continued. The limited condition field and the unlimited condition field are extracted from the field to be filtered, and the preset position field of the unlimited condition field is used as the first field. In an embodiment, the number of the first field can be one or more.

[0095] S207: Concatenate the limited condition field and the first field as a first index, perform matching on a multi-index map in a preset bitmap according to the first index, and determine a successfully matched map as a filter field.

[0096] In an embodiment, the limited condition field and the first field can be spliced ​​together to obtain a first index, a multi-index map in a preset bitmap can be determined, the first index can be matched in the multi-index map, and the successfully matched map can be used as a filter field.

[0097] S208: Splice the first fields as a second index, match the second index in a single index map in a preset bitmap, and determine a successfully matched map as a filter field.

[0098] In an embodiment, the first fields may be concatenated to obtain a second index, a single index map in a preset bitmap may be determined, the second index may be matched in the single index map, and the successfully matched map may be used as a filter field.

[0099] S209: The field to be filtered after removing the filter field is used as the target field, and the target field and the data to be transmitted are concatenated to form a target transmission message.

[0100] In an embodiment, the field to be filtered excluding the filtering field may be used as the target field, and the target field and the data to be transmitted may be concatenated to form a target transmission message.

[0101] S210: Convert the target transmission message into a standard message through a preset operator protocol, and transmit the standard message to the operator, so that the operator sends the data to be transmitted to the target terminal device.

[0102] In an embodiment of the present invention, an initial filtering field and data to be transmitted are obtained from a message to be transmitted, a logical symbol field of the initial filtering field is extracted, and a field to be filtered is determined according to the logical symbol field, thereby facilitating the individual determination of limited condition fields and unlimited condition fields. Furthermore, whether a null value exists in the limited condition field is determined. If so, a wildcard character is added to the field digits corresponding to the null value, and the limited condition fields are concatenated in a preset order to form a matching string. A preset dictionary tree is matched according to the matching string. If a match is successful, the successfully matched field to be filtered is determined to be the filtering field. If a complete match is successful, the limited condition field and the first field are concatenated as a first index, and a match is performed according to a multi-index map of the first index in a preset bitmap. The first field concatenation is used as a second index, and a match is performed according to a single index map of the second index in a preset bitmap. The successfully matched map is determined as the filtering field, thereby achieving full coverage filtering and improving filtering accuracy. Furthermore, no database or middleware access is required during message filtering, significantly reducing filtering time. Furthermore, the preset dictionary tree, single index map, and multi-index map can store information of multiple dimensions without hash collisions, supporting addition, deletion, modification, and query, and preventing false positives. This application compresses the original data through an improved dictionary tree structure and segmented indexed bitmaps, reduces the storage of duplicate data, and greatly reduces memory consumption.

[0103] Example 3

[0104] Figure 7 This is a structural diagram of a data filtering system provided according to the third embodiment of the present invention. In this embodiment, the message to be transmitted is an SMS message as an example, and the data filtering is a blacklist SMS message filtering based on a tree and a bitmap. In this embodiment, the limited condition is the limited condition field, and the unlimited condition is the unlimited condition field. Figure 7 As shown, the system includes: an input unit, a condition parsing unit, a construction unit, a storage unit, and a short message sending unit. The input unit includes a condition inputter and a mode changer; the construction unit includes a graph builder and a tree builder; and the short message sending unit includes a filter, a sender, and a resender.

[0105] Among them, the input unit is used to receive commands for adding, deleting and modifying filter conditions (i.e., filter condition modification instructions) input by the front end or interface, and commands for changing the mode of discarding / storing filtered messages (i.e., change instructions). The condition parsing unit parses the input filter conditions into a standardized format and passes it to the tree builder or the corresponding graph builder. The construction unit modifies the tree or graph according to the filter conditions. If the filter conditions do not contain infinite conditions, the tree is modified, otherwise the graph is modified. When the filter mode is storage, the storage unit can store the filtered messages. After the SMS sending unit receives the SMS message from the upstream system or reads the message from the storage unit, it filters it through the tree or graph generated by the construction unit, and sends the SMS after the filtering is completed.

[0106] In an embodiment, Figure 8 is a workflow diagram of a data filtering system according to an embodiment of the present application. The data filtering system is particularly used for the construction of filters and the filtering and sending of short messages. As shown in Figure 8 , the condition inputter of the input unit, the condition analysis unit and the construction unit are used for the construction of filters; the sender in the filter and the short message sending unit is used for the filtering and sending of short messages.

[0107] Specifically, the condition inputter of the input unit receives the add, delete, modify and query filter condition command (filter condition modification instruction) input by the front end or interface. When adding or modifying, the filter condition can input limited conditions such as province, city, operator, product number and product type, and unlimited conditions such as mobile phone number dimension, and logical symbols can be inputted as AND and OR. The mode changer receives the command (change instruction) input by the front end to discard / persistently store the filtered message, which changes the filtering mode. The condition analysis unit first judges whether the input filter condition (filter condition modification instruction) contains the logical symbol "or". If it contains, it will split this filter condition into multiple conditions containing only "and", and judge whether the newly added, modified or deleted filter condition contains unlimited conditions. If it does not contain unlimited conditions, the filter condition will be decomposed into the standard format of tree construction (second target field) and passed to the tree constructor. Otherwise, it will be directly passed to the corresponding graph constructor. The construction unit receives the add, delete and modify commands of the condition analysis unit and edits the corresponding tree or graph. After receiving the upstream short message or reading the pending message, the short message sending unit filters the message through the tree or graph of the construction unit. The filtered message is persistently stored in the storage unit according to the current filtering mode, or directly discarded. The unfiltered message is directly sent.

[0108] The input unit contains two types of condition inputters and mode changers. The condition inputter is used to receive the standardized format command (filter condition modification instruction) of the add, delete and modify filter condition input by the front end or interface. The command is a JSON message and can contain limited condition filters and unlimited conditions. For example, the example format of the filter condition modification instruction can be as follows:

[0109]

[0110] Among them, the instructionType field is the instruction type, and "I", "D" and "U" are optional, representing insert, delete and update respectively. If left blank or filled in, the remaining instructions will be invalid. limitedFilter represents a limited condition-related filter, and unlimitedFilter represents an unlimited condition-related filter. Both filter conditions need to be represented by integers with a fixed number of digits. The difference is that the integer number of the unlimited condition-related filter is usually longer. For example, the province filter in the limited condition filter can be represented by a 2-digit integer from 0 to 9, and the mobile phone number filter in the unlimited condition filter can be represented by an 11-digit integer from 0 to 9. In one embodiment, limitedFilter may include provinceFilter, spFilter, prodcodeFilter and businessTypeFilter, which are respectively province (2-digit string), operator (2-digit string), product number (9-digit string), and product type (4-digit string). unlimitedFilter may include phoneFilter, i.e., mobile phone number (11-digit string). The dimensional conditions are in an "and" relationship. The dimensional condition values ​​can optionally fill in multiple "or" relationship values. "||" means "or".

[0111] To ensure filtering efficiency and avoid repeated filtering, the filtering conditions in the conditional input unit should be defined and verified based on actual business conditions. For example, the mobile phone number, province, city, and carrier should not appear at the same time. This is because a specific mobile phone number already has a specific province, city, and carrier, so the conditions should not be repeated. If the product number is not empty, the product type must be empty because the product number itself contains the product type attribute and should not be filtered repeatedly.

[0112] Generally speaking, the mode changer only contains one field, ifSave, which indicates whether to persist the filtered data. It can be filled with a Boolean value of true or false. The default value is false. If it is true, all filtered data is saved to the storage unit. Otherwise, the message is discarded directly.

[0113] In one embodiment, Figure 9 This is a workflow diagram of a conditional parsing unit provided according to the third embodiment of the present invention. Figure 9 As shown, the condition parsing unit parses the input filter conditions into a standardized format and passes it to the tree builder or the corresponding graph builder. The execution process is as follows:

[0114] Receive the filter condition (filter condition modification instruction), analyze whether there is a logical symbol "or" (i.e. "||") in each field of the message passed by the condition input unit, and if so, disassemble the input message into multiple equivalent inputs without ||. Determine whether it contains infinite conditions one by one, that is, whether all the fields corresponding to the unlimitedFilter (unlimited condition related filters) are empty. If the infinite fields corresponding to the unlimitedFilter (condition related filters) are all empty, then it does not contain infinite conditions. Concatenate the values ​​of all limited filter conditions one by one to obtain a string. If the value of a filter condition is empty, use the wildcard # with the same number of fixed bits as the field to fill it, and standardize it as follows Figure 4 The string format shown passes the command type (addition, deletion, or modification) and command string to the tree builder one by one. If only one infinite condition is not empty, the value of that condition is filled into the second target field and passed to the graph builder corresponding to that dimension condition, i.e., a single-index graph. If both the finite and infinite conditions are not empty, the finite and infinite conditions are combined as the value of the instruction information field and passed as the second target field to the graph builder corresponding to the finite and infinite conditions, with the remaining finite and infinite conditions as the index and the bitmap dot value as the rest of the infinite condition, i.e., a multi-index graph.

[0115] In the graph builder of this patent, only one infinite condition is used. When there are multiple infinite conditions, the one with the largest change can be regarded as the infinite condition, and the other infinite conditions can be regarded as finite conditions.

[0116] In one embodiment, Figure 10 Schematic diagram of a structure of a building unit provided according to the third embodiment of the present invention. Figure 10 As shown, the construction units are divided into tree builders and graph builders. The tree builder can build a dictionary tree for filtering conditions that do not contain infinite conditions. The graph builder is divided into single-index graph builders and multi-index graph builders. Both graph builders can build segmented roaring bitmaps, as shown in the following figure:

[0117] After receiving the command passed by the conditional parsing unit, the tree builder adds or modifies the dictionary tree according to the Zhang Zhiling type. The dictionary tree is a nested hash structure. The key of each layer in the Map stores the value of the current node, and the value stores the Map of the subtree. The dictionary tree modification process is to first delete the original string and then add a new string. The addition and deletion process is as follows:

[0118] Addition can start from the first character of the string and check whether there is a corresponding match in the first layer of the tree. If there is no corresponding match, a child dictionary tree node for the character is added to the first layer node, and then child nodes are added from this node until the entire string is read and added. If there is a corresponding subtree, the child nodes of the subtree are compared from the second character. Deletion can be done by searching the subtree according to the string to the leaf node, and then recursively removing nodes without child nodes from the leaf node upwards.

[0119] The graph builder is divided into two categories based on whether the filter conditions include only infinite conditions, limited conditions, or infinite conditions. Each type of data received from the condition parsing unit constructs a roaring bitmap. A single-index graph containing only infinite conditions is constructed, while a multi-index graph containing both limited and infinite conditions is constructed. The addition and deletion process is as follows:

[0120] The Single Index Graph Builder now allows you to extract a relatively fixed field from an infinite condition as an index, and store the remaining fields in a bitmap table as values. For example, for a phone number, you can extract the first three digits as the index and store 4-11 in a bitmap table. The bitmap table automatically shards and compresses the data, and automatically selects the appropriate compression method as the number of phone numbers grows or shrinks.

[0121] The single index graph builder deletes the index and value in the same way as adding, searches for the corresponding bitmap chart, and queries the corresponding shard and position on the shard in the same way as adding to remove it.

[0122] In the input conditions of the multi-index graph builder, each additional finite condition in a dimension adds an index layer. Finally, the relatively fixed fields in the infinite conditions are extracted as the last index layer, and the remaining fields are stored as values ​​in the bitmap table. For example, if the filter conditions include product number and phone number, the product number is used as the first-level index, the first three digits of the phone number are extracted as the second-level index, and digits 4-11 are stored in the bitmap table.

[0123] In one embodiment, the SMS sending unit receives SMS messages transmitted by the upstream system or stored by the storage unit, and after filtering the SMS messages through a filter and a blacklist, uses a sender to push the SMS messages to the operator according to the standard protocol. The working process is as follows:

[0124] After receiving the SMS message, the filter extracts the limited condition fields of the message and standardizes them into Figure 3The standard format is used to filter using a dictionary tree constructed by a tree builder. The filtering method is as follows: each bit of the standard string is obtained one by one and matched with the key value of each layer of the dictionary tree map. If the same character or wildcard # is matched, its value is obtained, that is, the subtree continues to match. If there is a match at the end of the character, the match is successful, indicating that the current message needs to be filtered. Otherwise, it means that the current message has no match and the graph filtering continues. When filtering the graph, each single-index graph and multi-index graph is checked one by one. If there is no match, the text message is sent, otherwise the text message is filtered.

[0125] The sender receives SMS messages sent from upstream, filters them, converts them into standard messages according to different operator protocols, and pushes them to the operator.

[0126] The resender reads the SMS message to be resent from the storage unit buffer, filters it, converts it into a standard message according to different operator protocols, and pushes it to the operator.

[0127] In the embodiment of the present invention, there is no need to access the database or middleware when filtering messages, which reduces IO consumption. The dictionary tree structure comparison reduces the repeated comparison of the same value in the same dimension compared to the database one by one, which can greatly shorten the time consumption of filtering. At the same time, information of multiple dimensions can be stored without hash collision, and addition, deletion, modification and query are supported without false alarms. Moreover, in this embodiment, the original filtered data is not directly pulled into the memory, but the original data is compressed through an improved dictionary tree structure (the standard dictionary tree structure does not support the operation of the wildcard #) and a segmented indexed bitmap, thereby reducing the storage of duplicate data and greatly reducing memory consumption.

[0128] Example 4

[0129] Figure 11 FIG. 1 is a structural diagram of a data filtering device provided according to the fourth embodiment of the present invention. Figure 11 As shown, the device includes: a field determination module 111, a data filtering module 112 and a data transmission module 113.

[0130] The field determination module 111 is configured to obtain the initial filtering field and the data to be transmitted in the message to be transmitted, and perform a standardization operation on the initial filtering field to obtain the field to be filtered;

[0131] A data filtering module 112 is configured to filter the to-be-filtered fields based on a preset dictionary tree and a preset bitmap to determine target transmission messages;

[0132] The data transmission module 113 is configured to convert the target transmission message into a standard message through a preset operator protocol, and transmit the standard message to the operator so that the operator can send the data to be transmitted to the target terminal device.

[0133] The technical scheme of the embodiment of the application determines the initial filtering field and the to-be-transmitted data in the to-be-transmitted message through the field obtaining module, performs a standardization operation on the initial filtering field to obtain a to-be-filtered field, filters the to-be-filtered field based on a preset dictionary tree and a preset bitmap by the data filtering module, determines a target transmission message, and converts the target transmission message into a standard message by the data transmission module through a preset operator protocol, and transmits the standard message to an operator, so that the operator sends the to-be-transmitted data to a target terminal device, thereby realizing memory compression and efficient comparison and query under a multi-dimensional filtering condition or a large data volume filtering condition. Compared with the commonly used external database query, Bloom filter and hash table search methods, the method reduces IO interaction delay, enhances query accuracy, and saves memory usage space.

[0134] In an embodiment, the data filtering apparatus further comprises:

[0135] The symbol determining module is configured to receive a filtering condition modification instruction, extract a logical symbol field of the filtering condition modification instruction, and split the filtering condition modification instruction into a target filtering condition modification instruction according to the logical symbol field.

[0136] The field extracting module is configured to extract an instruction type, a limited condition field and an unlimited condition field of the target filtering condition modification instruction.

[0137] The first updating module is configured to update the preset bitmap based on the unlimited condition field when it is determined that the target filtering condition includes the unlimited condition field.

[0138] The second updating module is configured to update the preset dictionary tree based on the unlimited condition field when it is determined that the target filtering condition does not include the unlimited condition field.

[0139] In an embodiment, the first updating module is specifically configured to:

[0140] determine whether the target filtering condition includes a limited condition field and a preset position field of the unlimited condition field.

[0141] If yes, the limited condition field and the unlimited condition field are spliced as a first target field, and the first target field is transmitted to the single index graph constructor, so that the single index graph constructor updates the single index graph in the preset bitmap according to the first target field; wherein the limited condition field and the preset position field in the first target field are spliced as an index, and the remaining field of the unlimited condition field is taken as a value.

[0142] If no, the unlimited condition field is spliced as a second target field, and the second target field is transmitted to the graph constructor, so that the graph constructor updates the multi-index graph in the preset bitmap according to the second target field; wherein the preset position field in the second target field is spliced as an index, and the remaining field of the unlimited condition field is taken as a value.

[0143] In one embodiment, the second update module is specifically configured to:

[0144] Determine if there are null values ​​in the limited condition field;

[0145] If it exists, the number of field digits corresponding to the null value will be supplemented with wildcard characters;

[0146] The finite condition fields are concatenated in a preset order to form a string, and the instruction type and the string are transmitted to the tree builder, so that the tree builder updates the preset dictionary tree according to the instruction type and the string.

[0147] In one embodiment, the field determination module 111 includes:

[0148] A field extraction unit, used for extracting a logical symbol field of an initial filtering field;

[0149] a first extraction unit, configured to use the initial filtering field as a field to be filtered when the first symbol field does not exist in the logical symbol field;

[0150] The second extraction unit is used to split the initial filtering field when the first symbol field exists in the logical symbol field until the first symbol field does not exist in the logical symbol field of the initial filtering field, and use the split initial filtering field as the field to be filtered.

[0151] In one embodiment, the data filtering module 112 includes:

[0152] A string determination unit is used to extract the limited condition fields from the fields to be filtered, determine whether there are null values ​​in the limited condition fields, and if so, add wildcard characters to the field digits corresponding to the null values, and concatenate the limited condition fields in a preset order to form a matching string;

[0153] A field determination unit is used to match a preset dictionary tree according to a matching string, and if the match is successful, determine the successfully matched to-be-filtered field as a filter field;

[0154] A first matching unit is configured to extract the limited condition field and the unlimited condition field from the field to be filtered if no match is successful, and determine a preset position field of the unlimited condition field as the first field;

[0155] A second matching unit is configured to concatenate the limited condition field and the first field as a first index, perform matching according to a multi-index map in a preset bitmap according to the first index, and determine a successfully matched map as a filter field;

[0156] A third matching unit is configured to concatenate the first fields as a second index, perform matching according to a single index graph of the second index in a preset bitmap, and determine a successfully matched graph as a filter field;

[0157] The message determination unit is used to use the field to be filtered after removing the filtering field as the target field, and to splice the target field and the data to be transmitted as the target transmission message.

[0158] In one embodiment, a data filtering device further includes:

[0159] A mode determination module, used for determining a filtering mode for filtering a field to be filtered based on a preset dictionary tree and a preset bitmap;

[0160] A first determining module is configured to store, when the filtering mode is storage, the filtering fields based on the preset dictionary tree and the preset bitmap and the data to be transmitted as the message to be resent;

[0161] The second determining module is configured to discard the filtering fields based on the preset dictionary tree and the preset bitmap when the filtering mode is discard.

[0162] In one embodiment, a data filtering device further includes:

[0163] The mode change module is used to receive a change instruction for updating the filtering mode and update the filtering mode according to the change instruction.

[0164] A data filtering device provided by an embodiment of the present invention can execute a data filtering method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects of the execution method.

[0165] Example 5

[0166] Figure 12 1 is a schematic diagram of the structure of an electronic device that implements a data filtering method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0167] like Figure 12As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0168] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0169] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a data filtering method.

[0170] In some embodiments, a data filtering method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of a data filtering method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform a data filtering method in any other suitable manner (e.g., via firmware).

[0171] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0173] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0174] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0175] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0176] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0177] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0178] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data filtering method, characterized in that: include: Acquire an initial filtering field and data to be transmitted in a message to be transmitted, and perform a standardization operation on the initial filtering field to obtain a field to be filtered; Filter the to-be-filtered fields based on a preset dictionary tree and a preset bitmap to determine a target transmission message; The target transmission message is converted into a standard message through a preset operator protocol, and the standard message is transmitted to the operator, so that the operator sends the data to be transmitted to the target terminal device.

2. The method according to claim 1, characterized in that The construction process of the preset dictionary tree and preset bitmap includes: receiving a filter condition modification instruction, extracting a logic symbol field of the filter condition modification instruction, and splitting the filter condition modification instruction into target filter condition modification instructions according to the logic symbol field; Extracting the instruction type, limited condition field, and unlimited condition field of the target filtering condition modification instruction; When it is determined that the infinite condition field exists in the target filtering condition, updating a preset bitmap based on the infinite condition field; When it is determined that the infinite condition field does not exist in the target filtering condition, a preset dictionary tree is updated based on the infinite condition field.

3. The method according to claim 2, characterized in that The updating of the preset bitmap based on the infinite condition field includes: Determine whether there is a limited condition field in the target filtering condition, and a preset position field of the unlimited condition field; If so, the finite condition field and the infinite condition field are concatenated as a first target field, and the first target field is transmitted to a single index map builder, so that the single index map builder updates a single index map in a preset bitmap according to the first target field; wherein the finite condition field and the preset position field in the first target field are concatenated as an index, and the remaining fields of the infinite condition field are used as values; If it does not exist, the infinite conditional fields are spliced ​​as the second target field, and the second target field is transmitted to the graph builder so that the graph builder updates the multi-index graph in the preset bitmap according to the second target field; wherein the preset position fields in the second target field are spliced ​​as indexes, and the remaining fields of the infinite conditional field are used as values.

4. The method according to claim 2, characterized in that The updating of the preset dictionary tree based on the infinite condition field includes: Determine whether there is a null value in the limited condition field; If it exists, the number of field digits corresponding to the null value will be supplemented with wildcard characters; The finite condition fields are concatenated in a preset order to form a character string, and the instruction type and the character string are transmitted to a tree builder, so that the tree builder updates a preset dictionary tree according to the instruction type and the character string.

5. The method according to claim 1, wherein The performing of a standardization operation on the initial filtering field to obtain a field to be filtered includes: Extracting the logical symbol field of the initial filtering field; When the first symbol field does not exist in the logical symbol field, the initial filtering field is used as the field to be filtered; When the first symbol field exists in the logical symbol field, the initial filtering field is split until the first symbol field does not exist in the logical symbol field of the initial filtering field, and the split initial filtering field is used as the field to be filtered.

6. The method according to claim 1, characterized in that The filtering the to-be-filtered fields based on a preset dictionary tree and a preset bitmap to determine a target transmission message includes: Extract the limited condition fields from the fields to be filtered, determine whether there are null values ​​in the limited condition fields, if so, add wildcards to the field digits corresponding to the null values, and concatenate the limited condition fields in a preset order to form a matching string; Matching the preset dictionary tree according to the matching string, if the match is successful, determining the successfully matched to-be-filtered field as the filter field; If no match is successful, extracting the limited condition field and the unlimited condition field in the field to be filtered, and determining the preset position field of the unlimited condition field as the first field; splicing the limited condition field and the first field as a first index, matching the first index in a multi-index map in a preset bitmap, and determining a successfully matched map as a filter field; The first fields are concatenated as a second index, and a single index map of the second index in a preset bitmap is matched, and a successfully matched map is determined as a filter field; The field to be filtered with the filter field removed is used as the target field, and the target field and the data to be transmitted are spliced ​​together to form a target transmission message.

7. The method according to claim 1, characterized in that After filtering the to-be-filtered fields based on the preset dictionary tree and the preset bitmap to determine the target transmission message, the method further includes: Determine a filtering mode for filtering the field to be filtered based on a preset dictionary tree and a preset bitmap; When the filtering mode is storage, the filtering fields based on the preset dictionary tree and the preset bitmap and the data to be transmitted are stored as the message to be resent; When the filtering mode is discard, the filtering fields based on the preset dictionary tree and the preset bitmap are discarded.

8. The method according to claim 1, characterized in that Before filtering the to-be-filtered fields based on the preset dictionary tree and the preset bitmap to determine the target transmission message, the method further includes: A change instruction for updating a filtering mode is received, and the filtering mode is updated according to the change instruction.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform a data filtering method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a data filtering method according to any one of claims 1 to 8 when executed.

Images