A data filtering method, terminal device and storage medium based on space-time grid

Abstract

The present invention relates to a data filtering method based on space-time grid, terminal equipment and storage medium, the method comprising: S1: filtering data with a rule-based filter to remove abnormal data that does not conform to the rules; S2: performing space-time grid processing on the filtered data to encode the space-time grid and then storing it in a hash table structure; S3: filtering the data in the hash table that does not meet the space-time constraint conditions through the space-time filter. In addition to filtering common abnormal data, the present invention also refines the data by performing space-time grid processing on the data, reduces the storage of a large number of redundant basic collection logs, reduces the storage cost, and retains data with representative value, laying a good foundation for subsequent space-time association mining.

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to a data filtering method, terminal equipment and storage medium based on space-time grid. Background Art

[0002] With the advent of the big data era, the ever-expanding collection of data has caused big data to gradually get lost in the ocean of information. There is often a large amount of invalid and redundant data mixed in, which puts huge pressure on hardware requirements and also greatly affects the difficulty of data mining, resulting in a decrease in the value of mined information. The traditional technologies mainly include the following:

[0003] The first is the method adopted in the Chinese invention patent "A method and device for cleaning massive spatiotemporal data" with application number CN201610974444.4. It does not take into account the convenience of subsequent data retrieval and is not comprehensive, and only determines the legitimacy of data based on time.

[0004] The second is the method adopted in the Chinese invention patent "A spatiotemporal relationship analysis system based on multi-source location perception" with application number CN201710016325.2. It does not perform detailed filtering based on the characteristics of the data itself in combination with actual business, and its data storage is simply described using HDFS for distributed storage, which cannot reflect the ability to achieve fast retrieval in later retrieval.

[0005] In summary, the traditional methods have the following shortcomings: the traditional spatiotemporal data filtering only performs simple integrity and business specification verification filtering, or only considers the time factor, there is still a large amount of redundant data, and the data is not standardized, which is not conducive to retrieval; a large amount of computing power is required to mine and calculate these useless redundant data, and the mined related information is misleading, resulting in the phenomenon of generalizing quality by quantity. Therefore, it cannot meet the needs. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a data filtering method, terminal device and storage medium based on spatiotemporal grid.

[0007] The specific plan is as follows:

[0008] A data filtering method based on a spatiotemporal grid comprises the following steps:

[0009] S1: Use rule-based filters to filter the data and remove abnormal data that does not conform to the rules;

[0010] S2: Processing the filtered data into space-time grids to encode the space-time grids and storing them in a hash table structure;

[0011] S3: Filter the data in the hash table that does not meet the time and space constraints through the time and space filter.

[0012] Furthermore, abnormal data that does not comply with the rules includes: data with incomplete fields, data on the blacklist, and data that does not meet the business rules.

[0013] Furthermore, the content corresponding to the keyword field in the hash table in step S2 is a concatenation of the data type and the spatiotemporal grid code.

[0014] Furthermore, the spatiotemporal grid encoding is specifically: timeSlot|longitudeSlot|latitudeSlot, where: "|" represents a separator, timeSlot represents a time slot, longitudeSlot represents a longitude slot, and latitudeSlot represents a latitude slot. The calculation formulas are:

[0015] timeSlot=(timestamp–timestamp%timeSlotSize) / timeSlotSize

[0016] longitudeSlot=(longitude–longitude%longitudeSlotSize) / longitudeSlotSize

[0017] latitudeSlot=(latitude–latitude%latitudeSlotSize) / latitudeSlotSize

[0018] longitude=int(longitude*100000)

[0019] latitude=int(latitude*100000)

[0020] Among them, timestamp represents the timestamp, timeSlotSize represents the timestamp unit, longitude represents the longitude value, latitude represents the latitude value, longitudeSlotSize represents the longitude slot unit, latitudeSlotSize represents the latitude slot unit, and int(.) represents rounding.

[0021] Furthermore, the specific process of collecting logs in step S3 based on filtering through the spatiotemporal filter includes the following steps:

[0022] S301: extracting the collection location P and collection timestamp t of the received collection log A;

[0023] S302: Obtain the grid corresponding to the collection location grid P from redis, and the collection logs in the first N time slots based on the time slot corresponding to time t;

[0024] S303: Determine whether there is already a collection log in the time slot where the collection log A is located. If yes, proceed to S304; otherwise, proceed to S305;

[0025] S304: Determine whether the time of the collection log A is earlier than the time of the collection log that already exists in the time slot. If yes, proceed to S306. Otherwise, determine that the collection log A is a retained log and end.

[0026] S305: Update the collection log of the time slot on redis to collection log A, and enter S307;

[0027] S306: Update the collection log of the time slot on redis to collection log A, and determine that collection log A is a retained log;

[0028] S307: Determine whether there is a collection log A in the first N time slots. If yes, determine that the collection log A is a retention log; otherwise, determine that the collection log A is a non-retention log;

[0029] S308: Filter all retained logs.

[0030] A data filtering terminal device based on a spatiotemporal grid comprises a processor, a memory and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described above in the embodiment of the present invention are implemented.

[0031] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described above in an embodiment of the present invention are implemented.

[0032] The present invention adopts the above technical solution and has the following beneficial effects:

[0033] (1) A hash table structure storage method based on spatiotemporal grid is proposed, which saves more than N times of storage space, facilitates fast retrieval, solves the reverse order problem of real-time data, and effectively improves the speed of association mining and comparison;

[0034] (2) A spatiotemporal filtering method based on spatiotemporal grid is proposed to effectively reduce redundant collection logs and greatly reduce the computing power of association comparison. The associations mined based on this method are more representative. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Shown is an overall flow chart of Embodiment 1 of the present invention.

[0036] Figure 2 Shown is a schematic diagram of the hash table storage structure in this embodiment.

[0037] Figure 3 Shown is a flow chart of spatiotemporal filter filtering in this embodiment. DETAILED DESCRIPTION

[0038] To further illustrate various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.

[0039] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0040] Embodiment 1:

[0041] The embodiment of the present invention provides a data filtering method based on a spatiotemporal grid, such as Figure 1 As shown, the method comprises the following steps:

[0042] S1: Use rule-based filters to filter the data and remove abnormal data that does not conform to the rules.

[0043] In this embodiment, abnormal data that does not conform to the rules include: data with incomplete fields, data on the blacklist, and data that does not meet the business rules. The corresponding rule filter can be set in an existing common way. In other embodiments, those skilled in the art can set the rules for filtering according to actual application requirements, which is not limited here.

[0044] In this embodiment, the data is the collection log in redis, and the specific process of filtering processing includes the following steps:

[0045] S101: receiving a collection log, determining whether the content of each field in the collection log is complete, if so, proceeding to S102; otherwise, determining that the collection log is an incomplete log, proceeding to S104.

[0046] S102: Determine whether the unique value of the collection log is not on the blacklist, if yes, proceed to S103; otherwise, determine that the collection log is a blacklist log, proceed to S104.

[0047] S103: Determine whether the collected log meets the business specifications. If yes, determine that the log is a normal log and proceed to S104; otherwise, determine that the collected log is an irregular log and proceed to S104.

[0048] S104: Eliminate the collected logs that are abnormal logs.

[0049] S2: The filtered data is processed into a spatiotemporal grid and encoded into a spatiotemporal grid, and then stored in a hash table structure.

[0050] By storing data in a hash table structure, data belonging to the same keyword field can be stored together in the hash table, which is convenient for fast retrieval and processing of spatiotemporal associations and conflicts.

[0051] In this embodiment, the content corresponding to the keyword (KEY) field of the hash table is the concatenation of the data type (dataType) and the spatiotemporal grid code, such as Figure 2 As shown, the spatiotemporal grid code set in this embodiment is specifically: timeSlot|longitudeSlot|latitudeSlot, where: "|" represents a separator, timeSlot represents a time slot, longitudeSlot represents a longitude slot, and latitudeSlot represents a latitude slot. The calculation formulas are:

[0052] timeSlot=(timestamp–timestamp%timeSlotSize) / timeSlotSize

[0053] longitudeSlot=(longitude–longitude%longitudeSlotSize) / longitudeSlotSize

[0054] latitudeSlot=(latitude–latitude%latitudeSlotSize) / latitudeSlotSize

[0055] longitude=int(longitude*100000)

[0056] latitude=int(latitude*100000)

[0057] Among them, timestamp represents the timestamp, timeSlotSize represents the timestamp unit, longitude represents the longitude value, latitude represents the latitude value, longitudeSlotSize represents the longitude slot unit, latitudeSlotSize represents the latitude slot unit, and int(.) represents rounding.

[0058] The value1 and value2 in the value (VALUE) field are specific data information under the spatiotemporal grid.

[0059] S3: Filter the data in the hash table that does not meet the time and space constraints through the time and space filter.

[0060] like Figure 3 As shown, the specific process of filtering the collected logs based on the spatiotemporal filter in this embodiment includes the following steps:

[0061] S301: extracting the collection location P and collection timestamp t of the received collection log A;

[0062] S302: Obtain the grid corresponding to the collection location grid P from redis, and the collection logs in the first N time slots based on the time slot corresponding to time t;

[0063] S303: Determine whether there is already a collection log in the time slot where the collection log A is located. If yes, proceed to S304; otherwise, proceed to S305;

[0064] S304: Determine whether the time of the collection log A is earlier than the time of the collection log that already exists in the time slot. If yes, proceed to S306. Otherwise, determine that the collection log A is a retained log and end.

[0065] S305: Update the collection log of the time slot on redis to collection log A, and enter S307;

[0066] S306: Update the collection log of the time slot on redis to collection log A, and determine that collection log A is a retained log;

[0067] S307: Determine whether there is a collection log A in the first N time slots. If yes, determine that the collection log A is a retention log; otherwise, determine that the collection log A is a non-retention log;

[0068] S308: Filter all retained logs.

[0069] In addition to filtering common abnormal data, the embodiments of the present invention also refine the data by gridding the data in time and space, thereby reducing the storage of a large amount of redundant basic collection logs, reducing storage costs, and retaining data with representative value, laying a good foundation for subsequent time and space correlation mining work.

[0070] Embodiment 2:

[0071] The present invention also provides a data filtering terminal device based on a spatiotemporal grid, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned method embodiment of the first embodiment of the present invention when executing the computer program.

[0072] Further, as an executable solution, the data filtering terminal device based on the space-time grid can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The data filtering terminal device based on the space-time grid can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the composition structure of the data filtering terminal device based on the space-time grid is only an example of a data filtering terminal device based on the space-time grid, and does not constitute a limitation on the data filtering terminal device based on the space-time grid. It can include more or less components than the above, or combine certain components, or different components. For example, the data filtering terminal device based on the space-time grid can also include input and output devices, network access devices, buses, etc., and the embodiment of the present invention does not limit this.

[0073] Further, as an executable solution, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the data filtering terminal device based on the space-time grid, and uses various interfaces and lines to connect various parts of the data filtering terminal device based on the space-time grid.

[0074] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the data filtering terminal device based on the spatiotemporal grid by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0075] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.

[0076] If the module / unit integrated in the data filtering terminal device based on the spatiotemporal grid is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc.

[0077] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A data filtering method based on spatiotemporal grid, characterized in that: The following steps are involved: S1: Use rule-based filters to filter the data and remove abnormal data that does not conform to the rules; S2: The filtered data is processed into space-time grid code and stored in a hash table structure. The space-time grid code is: timeSlot|longitudeSlot|latitudeSlot, where "|" represents a separator, timeSlot represents a time slot, longitudeSlot represents a longitude slot, and latitudeSlot represents a latitude slot. The calculation formulas are: timeSlot=(timestamp–timestamp%timeSlotSize) / timeSlotSize longitudeSlot=(longitude–longitude%longitudeSlotSize) / longitudeSlotSize latitudeSlot=(latitude–latitude%latitudeSlotSize) / latitudeSlotSize longitude=int(longitude*100000) latitude=int(latitude*100000) Among them, timestamp represents the timestamp, timeSlotSize represents the timestamp unit, longitude represents the longitude value, latitude represents the latitude value, longitudeSlotSize represents the longitude slot unit, latitudeSlotSize represents the latitude slot unit, and int(.) represents rounding; S3: Filter the data in the hash table that does not meet the time and space constraints through the time and space filter.

2. The data filtering method based on the spatiotemporal grid according to claim 1, characterized in that: Abnormal data that does not comply with the rules includes: data with incomplete fields, data on the blacklist, and data that does not meet business rules.

3. The data filtering method based on spatiotemporal grid according to claim 1, characterized in that: The content corresponding to the keyword field in the hash table in step S2 is the concatenation of the data type and the spatiotemporal grid code.

4. The data filtering method based on spatiotemporal grid according to claim 1, characterized in that: The specific process of collecting logs in step S3 based on filtering through the spatiotemporal filter includes the following steps: S301: extracting the collection location P and collection timestamp t of the received collection log A; S302: Obtain the grid corresponding to the collection location grid P from redis, and the collection logs in the first N time slots based on the time slot corresponding to time t; S303: Determine whether there is already a collection log in the time slot where the collection log A is located. If yes, proceed to S304; otherwise, proceed to S305; S304: Determine whether the time of the collection log A is earlier than the time of the collection log that already exists in the time slot. If yes, proceed to S306. Otherwise, determine that the collection log A is a retained log and end. S305: Update the collection log of the time slot on redis to collection log A, and enter S307; S306: Update the collection log of the time slot on redis to collection log A, and determine that collection log A is a retained log; S307: Determine whether there is a collection log A in the first N time slots. If yes, determine that the collection log A is a retention log; otherwise, determine that the collection log A is a non-retention log; S308: Filter all retained logs.

5. A data filtering terminal device based on a spatiotemporal grid, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 4 when executing the computer program.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images