Data processing method and device, equipment and storage medium

Abstract

The invention discloses a data processing method and device, equipment and a storage medium, and the method comprises the steps: collecting original housing resource data; performing format standardization processing and abnormal value elimination processing on the original housing resource data to obtain standard housing resource data; comparing the standard housing resource data with building information contained in a preset building information base to establish a data association relationship between the standard housing resource data and the building information; and pushing the standard housing resource data and the corresponding data association relationship to a housing resource information base.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] The management of property information generally relies on manual management. Specifically, property information needs to be manually entered into Excel spreadsheets for secondary processing, which leads to problems such as information delays and high error rates. Summary of the Invention

[0003] To improve information processing efficiency and reduce errors and omissions, this application discloses the following technical solution:

[0004] The first aspect of this application provides a data processing method, including:

[0005] Collect raw housing data;

[0006] The original housing data is subjected to format standardization and outlier removal to obtain standard housing data;

[0007] The standard housing data is compared with the housing information contained in the preset housing information database to establish a data association between the standard housing data and the housing information;

[0008] The standard housing data and corresponding data relationships are pushed to the housing information database.

[0009] Optionally, before pushing the standard housing data and corresponding data associations to the housing information database, the method further includes:

[0010] A compliance check is performed on the standard housing data and the corresponding data relationships.

[0011] Optionally, the collection of raw housing data includes:

[0012] Using a web crawler, raw housing data is retrieved from public data sources at various preset target URLs.

[0013] Optionally, pushing the standard housing data and corresponding data associations to the housing information database includes:

[0014] Based on the distributed transaction lock mechanism, the standard housing data and the corresponding data associations are pushed to the housing information databases of multiple systems.

[0015] A second aspect of this application provides a data processing apparatus, comprising:

[0016] The data acquisition unit is used to collect raw housing data;

[0017] The processing unit is used to perform format standardization processing and outlier removal processing on the original housing data to obtain standard housing data;

[0018] The comparison unit is used to compare the standard housing data with the housing information contained in the preset housing information database to establish a data association between the standard housing data and the housing information.

[0019] The push unit is used to push the standard housing data and the corresponding data association to the housing information database.

[0020] Optionally, the device further includes an inspection unit for:

[0021] A compliance check is performed on the standard housing data and the corresponding data relationships.

[0022] Optionally, when the acquisition unit acquires raw housing data, it is specifically used for:

[0023] Using a web crawler, raw housing data is retrieved from public data sources at various preset target URLs.

[0024] Optionally, when the push unit pushes the standard housing data and corresponding data associations to the housing information database, it is specifically used for:

[0025] Based on the distributed transaction lock mechanism, the standard housing data and the corresponding data associations are pushed to the housing information databases of multiple systems.

[0026] A third aspect of this application provides an electronic device, including a memory and a processor;

[0027] The memory is used to store computer programs;

[0028] The processor is used to execute the computer program to implement the data processing method provided by any of the first aspects of this application.

[0029] The fourth aspect of this application provides a computer-readable storage medium for storing a computer program, which, when executed by a computer device, implements the data processing method provided in any one of the first aspects of this application.

[0030] The beneficial effects of this application are: real-time data collection, automatic cleaning of outliers, and intelligent matching of standard housing data and building information, and automatic entry of the matched data into the housing information database. Thus, this solution can significantly improve the efficiency of data entry. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0032] Figure 1 This is a flowchart of a data processing method provided in an embodiment of this application;

[0033] Figure 2 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0034] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] This embodiment provides a data processing method; please refer to [link / reference]. Figure 1 The method may include the following steps.

[0037] S101, collect raw housing data.

[0038] The method provided in this embodiment can be executed by any computer system (hereinafter referred to as the system).

[0039] Original housing data refers to the housing data published on relevant official websites (such as the official websites of local construction commissions).

[0040] In this embodiment, raw housing data can be collected in a variety of ways.

[0041] In one alternative data collection method, if the data interface provided by the official website is obtained in advance, the corresponding official website can be accessed directly through these data interfaces to request the required raw housing data.

[0042] One possible data collection method is:

[0043] Using a web crawler, raw housing data is retrieved from public data sources at various preset target URLs.

[0044] In this embodiment, one or more dedicated crawler programs can be pre-deployed.

[0045] For example, for each official website from which data needs to be scraped, a dedicated crawler program can be deployed to scrape the public data of that website. Relevant parameters can be configured in the crawler program to enable it to target the public data source specified by the user on the corresponding website.

[0046] The relevant configuration parameters can be any parameters related to the public data source to be crawled. For example, they can include the storage path of the public data source on the corresponding website, the title of the public data source, the keywords contained therein, etc., as long as the crawler program can uniquely identify the public data source to be crawled on the corresponding website based on the relevant parameters.

[0047] Using the methods described above, this embodiment can utilize a web crawler program to collect core data published on relevant websites in real time within seconds through dynamic anti-crawling cracking technology, thereby achieving the capture and landing of core data.

[0048] S102, standardize the format of the original housing data and remove outliers to obtain standard housing data.

[0049] In step S102, the pre-deployed intelligent cleaning engine can be used to process the original housing data to achieve format standardization and outlier removal, thereby obtaining standard housing data with a standard format and outlier removal.

[0050] Standardizing the format of original property listings refers to converting data in different formats within the original listings into a pre-defined, specified format. As examples, standardizing the format of original property listings includes, but is not limited to, standardizing area units and date formats. Standardizing area units means converting all area data appearing in the original listings to a specified area unit (e.g., square meters). Standardizing date formats means converting all date data appearing in the original listings to a standardized date format, such as year.month.day, like 2025.9.23.

[0051] Outlier removal from original property listings refers to identifying and deleting abnormal data, including but not limited to abnormal price data, abnormal character data, and abnormal patterns.

[0052] One way to identify abnormal price data is to generally recognize that property prices can fluctuate within a certain range, but prices should not exceed this range. If a price does exceed this range, it can be determined that the price is abnormal. Therefore, at least one target price range can be identified, and then price data in the original property data that is outside the target price range can be identified as abnormal price data.

[0053] The target price range can be specified by the relevant user, or it can be derived by the intelligent cleaning engine by analyzing the prices of a large number of properties on the market.

[0054] One way to identify abnormal character data is to compare the characters contained in the original property listing data with a preset character library. If the original property listing data contains a character that is not in the character library, then the character can be determined to be illegal. The character library can contain all characters that may be used in property and building information. The characters in the character library can be entered by relevant users, or they can be determined by the intelligent cleaning engine after analyzing various web pages and information related to properties and buildings in the market.

[0055] S103. Compare the standard housing data with the housing information contained in the preset housing information database to establish a data relationship between the standard housing data and the housing information.

[0056] The system's property information database can include property information from multiple pre-entered properties.

[0057] In this embodiment, a fuzzy matching algorithm can be used to achieve intelligent mapping between the construction committee's registered name and the website's property database. Data association is completed by comparing multi-dimensional features such as property coordinates, developer name, and planning permit number.

[0058] The data relationship between standard housing data and property information, also known as the mapping relationship between standard housing data and property information, means that the housing described by the standard housing data belongs to the property corresponding to the property information.

[0059] For example, if all 10 properties corresponding to Standard Property Data 1 belong to Building A, then the aforementioned data association can be established between Standard Property Data 1 and the building information corresponding to Building A.

[0060] The method for establishing data relationships by comparing the coordinates of different buildings is as follows:

[0061] Data representing the geographical location of a property is obtained from standard property data. Based on this data, the geographical location of the property is determined. Then, the geographical location of the property is compared with the geographical location of the property in the property information. If the two are found to overlap, or the distance between them is less than a certain threshold, it can be considered that the property in the standard property data belongs to the property corresponding to the property information. Thus, a data association relationship can be established between the standard property data and the property information.

[0062] The method for establishing data relationships by comparing data based on developer names is as follows:

[0063] Obtain text related to the developer's name from standard housing data, such as "This property was developed by XX company" or "XX company constructed this project," and calculate the similarity between this text and the developer's name set in the property information. If the similarity is greater than a preset threshold, it can be determined that the property corresponding to the standard housing data and the property corresponding to the property information were developed by the same developer, and the property has a high probability of belonging to that property.

[0064] The method for establishing data relationships by comparing data based on planning permit numbers is as follows:

[0065] Obtain the planning permit number from the standard housing data, compare the planning permit number of the standard housing data with the planning permit number of the building information, and if they match, it can be considered that the housing in the standard housing data belongs to the building information corresponding to it. Thus, a data association relationship can be established between the standard housing data and the building information.

[0066] Optionally, when standard housing data and property information lack data in a certain dimension, making direct comparison of that dimension impossible, cross-dimensional comparison can also be performed.

[0067] For example, standard housing data may only contain planning permit numbers without geographical location information, while property information may only contain the property's geographical location. In such cases, a cross-dimensional comparison between the planning permit number and the geographical location can be performed. Specifically, other publicly available data sources can be used to query the permitted land plots corresponding to the planning permit number, and then the geographical location of the permitted land plot can be found. If the geographical location overlaps with the property's geographical location or the distance between the two is less than a certain threshold, then the housing in the standard housing data can be considered to belong to the property corresponding to the property information. Thus, a data association can be established between the standard housing data and the property information.

[0068] S104 pushes standard housing data and corresponding data relationships to the housing information database.

[0069] Optionally, standard housing data and corresponding data associations can be pushed to the housing information database, including:

[0070] Based on a distributed transaction lock mechanism, standard housing data and corresponding data associations are pushed to housing information databases of multiple systems.

[0071] In this embodiment, the above data can be pushed to the housing database in real time via RabbitMQ, and a distributed transaction lock is used to ensure data consistency between multiple systems during the push, with the synchronization delay controlled within 3 seconds.

[0072] Through the above solution, this embodiment can connect to the developer's or construction commission's website. The distributed data crawler automatically captures changes in housing information, and the intelligent cleaning engine processes the data in seconds, synchronously updating it to the official website, APP, mini-program, etc., realizing real-time connection of the entire link from the construction commission's data source to the marketing platform to the customer terminal, and achieving full automation of the data collection process.

[0073] Optionally, before pushing standard housing data and corresponding data associations to the housing information database, the following may also be included:

[0074] Conduct compliance checks on standard housing data and corresponding data relationships.

[0075] If the standard housing data fails the compliance check, the standard housing data and its corresponding data relationships can be deleted and not entered into the housing information database. Alternatively, the non-compliant data can be processed, and the processed data that passes the compliance check can be entered into the housing information database. If the compliance check is passed, the standard housing data and its corresponding data relationships can be directly entered into the housing information database.

[0076] The aforementioned compliance checks can be implemented based on a dual verification mechanism.

[0077] Specifically, when conducting compliance checks, business compliance checks and data security checks can be performed sequentially.

[0078] Business compliance checks may include: checking whether the standard property listing data is duplicated with existing property listing data in the property listing database, and checking whether it is necessary to update the corresponding fields of the standard property listing data and / or the existing property listing data in the property listing database.

[0079] Data security checks may include checking whether standard property data contains sensitive fields, checking the data content of standard property data, and determining access permissions based on the data content to achieve hierarchical access control.

[0080] If the inspection finds that the standard housing data contains sensitive fields, the sensitive fields can be anonymized before the anonymized data is entered into the housing information database.

[0081] De-identification processes may include: encrypting or obfuscating confidential sensitive fields, and replacing potentially non-compliant text with compliant text that has the same semantic meaning.

[0082] The beneficial effects of this embodiment are: real-time data collection, automatic cleaning of outliers, and intelligent matching of standard housing data and building information, and automatic entry of the matched data into the housing information database. Thus, this solution can significantly improve the efficiency of data entry.

[0083] Furthermore, it achieves real-time data collection in seconds through dynamic anti-crawling technology, automatically cleans outliers by combining a thousand-level rule engine and AI algorithms, and improves the accuracy of property mapping to 92% by integrating multi-factor intelligent matching such as coordinate positioning and developer verification. It also incorporates dual protection of business compliance review and sensitive data de-identification, and relies on a distributed architecture to ensure 3-second data synchronization in high-concurrency scenarios, stably processing more than 2 million data entries per day with zero loss.

[0084] This application also provides a data processing apparatus; please refer to [link to relevant documentation]. Figure 2 The device may include the following units.

[0085] Data acquisition unit 201 is used to collect raw housing data;

[0086] Processing unit 202 is used to perform format standardization processing and outlier removal processing on the original housing data to obtain standard housing data;

[0087] The comparison unit 203 is used to compare the standard housing data with the housing information contained in the preset housing information database in order to establish the data association between the standard housing data and the housing information.

[0088] Push unit 204 is used to push standard housing data and corresponding data associations to the housing information database.

[0089] Optionally, the device also includes an inspection unit for:

[0090] Conduct compliance checks on standard housing data and corresponding data relationships.

[0091] Optionally, when the data acquisition unit collects raw housing data, it is specifically used for:

[0092] Using a web crawler, raw housing data is retrieved from public data sources at various preset target URLs.

[0093] Optionally, when the push unit pushes standard housing data and corresponding data associations to the housing information database, it is specifically used for:

[0094] Based on a distributed transaction lock mechanism, standard housing data and corresponding data associations are pushed to housing information databases of multiple systems.

[0095] This embodiment provides an electronic device, see [link / reference] Figure 3 This includes a memory 301 and a processor 302;

[0096] Memory 301 is used to store computer programs;

[0097] The processor 302 is used to execute computer programs to implement the data processing method provided in any embodiment of this application.

[0098] This application provides a computer-readable storage medium for storing a computer program. When the computer program is executed by a computer device, it is used to implement the data processing method provided in any embodiment of this application.

[0099] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0100] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0101] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0102] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0103] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: collecting original house data; performing format standardization processing and outlier rejection processing on the original house data to obtain standard house data; comparing the standard house data with house information contained in a preset house information database to establish a data association relationship between the standard house data and the house information; pushing the standard house data and the corresponding data association relationship to a house information database.

2. The method of claim 1, wherein, Before the step of pushing the standard house data and the corresponding data association relationship to the house information database, the method further comprises the following step: performing compliance checking on the standard house data and the corresponding data association relationship.

3. The method of claim 1, wherein, The step of collecting original house data comprises the following step: using a crawler program to grab original house data from public data sources of preset target websites.

4. The method of claim 1, wherein, The step of pushing the standard house data and the corresponding data association relationship to the house information database comprises the following step: based on a distributed transaction lock mechanism, pushing the standard house data and the corresponding data association relationship to house information databases of multiple systems.

5. A data processing apparatus, characterized by, The method comprises the following steps: a collecting unit configured to collect original house data; a processing unit configured to perform format standardization processing and outlier rejection processing on the original house data to obtain standard house data; a comparison unit configured to compare the standard house data with house information contained in a preset house information database to establish a data association relationship between the standard house data and the house information; a pushing unit configured to push the standard house data and the corresponding data association relationship to a house information database.

6. The apparatus of claim 5, wherein, The device further comprises an inspection unit configured to: perform compliance checking on the standard house data and the corresponding data association relationship.

7. The apparatus of claim 5, wherein, When the collecting unit collects original house data, the collecting unit is specifically configured to: use a crawler program to grab original house data from public data sources of preset target websites.

8. The apparatus of claim 5, wherein, When the pushing unit pushes the standard house data and the corresponding data association relationship to the house information database, the pushing unit is specifically configured to: based on a distributed transaction lock mechanism, push the standard house data and the corresponding data association relationship to house information databases of multiple systems.

9. An electronic device, comprising: The device comprises a memory and a processor; the memory is configured to store a computer program; the processor is configured to execute the computer program to implement the data processing method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer program is stored in the memory and is executed by the processor to implement the data processing method according to any one of claims 1 to 4.

Images