A mass database data cleaning processing system and method

Abstract

The application relates to the technical field of data cleaning, in particular to a large-quantity database data cleaning processing system method, which comprises an analysis module, a definition module and a database. The analysis module is used for analyzing the title and description of a newly-uploaded product of a merchant; the definition module is used for defining a product data model; and the database is used for storing various data. The analysis module is used for analyzing the title and description of the newly-uploaded product of the merchant, and the definition module is used for defining the product data model. The analysis module compares the title and description of the product with the data in the database, obtains a similarity report, and determines whether to store according to the similarity, so as to improve the accuracy of the determination of the merchant user, effectively avoid the influence of too many repeated products in the store of the merchant on the sales of the products, and further comprehensively detect the similarity of the products through the combination of the multi-attribute fields, and effectively avoid the data redundancy in the database.

Description

Technical Field

[0001] This invention relates to the field of data cleaning technology, and in particular to a system and method for cleaning and processing large volumes of database data. Background Technology

[0002] In e-commerce websites, merchants need to frequently introduce new products to ensure timely updates to their store's inventory. When updating with new products, merchants often list a large number of items to increase the quantity of products in their store. During the process of listing new products, a large number of duplicate and highly similar products will appear. This can easily lead to too many duplicate products in the store, which not only affects the store's browsing and sales data and ranking, but also causes a large amount of redundant data in the server's database, which will have a certain impact on the database's operation.

[0003] Therefore, it is necessary to propose a technology to improve the above-mentioned defects. Summary of the Invention

[0004] In view of the above-mentioned shortcomings and defects of data cleaning in the prior art, the present invention provides a large-scale database data cleaning and processing system and method, which comprehensively analyzes the similarity of product information, improves the accuracy of information judgment, and determines the similarity of product data information with the participation of merchants.

[0005] To achieve the above objectives, the present invention provides a large-scale database data cleaning and processing system, comprising: The analysis module analyzes the similarity between product titles and descriptions; A definition module, which is used to define the product data model; A database, which is used to store various types of data.

[0006] As a preferred technical solution, the analysis module uses NLP algorithms to perform timed analysis on the product and provides relevant prompts based on the analysis results.

[0007] As a preferred technical solution, the definition module uses Apache Flink technology to define the product's data model, compares the product content, assigns scores, and stores the data in the database.

[0008] As a preferred technical solution, the database is updated with periodic marker records to ensure database updates.

[0009] A method for cleaning and processing large volumes of database data includes the following steps: Step 1: After the product is analyzed by NLP algorithm, it is identified that the product has duplicate content, and then it is pushed to the user interface; Step 2: After a user publishes a product, the system compares the data with the data in the database using NLP algorithms, and generates attribute codes using Apache Flink technology, which are then compared with historical product codes in the database.

[0010] Step 3: After the product is compared with the database, a verification report is generated and sent to the user for confirmation on whether to publish or save.

[0011] As described above, the large-scale database data cleaning and processing system and method provided by the present invention have the following beneficial effects: (1) This invention analyzes the titles and descriptions of new products launched by merchants by setting up an analysis module, and defines the product data model by defining a definition module. The analysis module compares the titles and descriptions of the products with the data in the database to generate a similarity report. Users can decide whether to store the product based on the similarity, which improves the accuracy of the judgment of merchants and users and effectively avoids the impact of too many duplicate products in the merchant's store on the sales of the products. In addition, the combination of multiple attribute fields can more comprehensively detect the similarity of products and effectively avoid data redundancy in the database.

[0012] (2) The NLP algorithm used in the analysis module can perform more accurate analysis of the title and description of the product, improve the accuracy of similarity check, and make more accurate judgment of products with high similarity, thereby reducing the false alarm rate of product similarity; the Apache Flink technology used in the definition module is used to compare the published products, which helps to discover and block products with extremely high similarity in a timely manner during the product publishing stage, so as to ensure the quality of published products. Attached Figure Description

[0013] Figure 1 The diagram shown is a schematic representation of the overall structural flow of the present invention. Figure 2 The diagram shown is a schematic representation of the operation flow of an embodiment of the present invention.

[0014] 1. Analysis module; 2. Definition module; 3. Database. Detailed Implementation

[0015] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.

[0016] It should be noted that the illustrations provided in this embodiment are only schematic representations of the basic concept of the present invention. Although the illustrations only show components related to the present invention and are not drawn according to the actual number, shape and size of the components, the shape, quantity, positional relationship and proportion of each component can be arbitrarily changed under the premise of realizing the technical solution of this invention, and the layout of the components may also be more complex.

[0017] Example This embodiment provides a large-scale database data cleaning and processing system, as shown in the attached document. Figure 1 The data processing system includes an analysis module 1, a definition module 2, and a database 3. Analysis module 1 analyzes the similarity of product titles and descriptions. Definition module 2 defines the product data model. Analysis module 1 employs NLP algorithms to perform periodic analysis on products and provides relevant prompts for duplicate content found in the analysis results. Definition module 2 uses Apache Flinks technology to define the product data model, compares the listed products with the existing database 3, assigns a score, and stores it in database 3. Additionally, the NLP algorithm in analysis module 1 transforms product titles and descriptions into high-dimensional vectors, which, combined with the Apache Flinks technology in the definition module, process the product data. The data is then combined with the score assigned in database 3 and stored, thus achieving the cleaning of duplicate data.

[0018] See attached document Figure 2 In this data cleaning method, when a merchant user lists a new product, the NLP algorithm in the analysis module 1 analyzes the product title and description. Duplicate content in the analysis results is then identified and a relevant prompt is sent to the user. After the user publishes or edits the product, the Apache Flinks technology in the definition module compares the product title with historical titles in the database. For the product content, Apache Flinks generates an attribute code and compares it with historical products in database 3. Finally, a verification report is generated and sent to the user interface, allowing the user to determine whether the product title and description have been published and store them in database 3.

[0019] In summary, this invention analyzes the titles and descriptions of new products uploaded by merchants through an analysis module and defines the product data model through a definition module. The analysis module compares the product titles and descriptions with data in the database to generate a similarity report. Users can then decide whether to store the product based on the similarity score, improving the accuracy of merchants' judgments and effectively preventing excessive duplicate products in merchants' stores from affecting product sales. Furthermore, the combination of multiple attribute fields allows for a more comprehensive detection of product similarity and effectively avoids data redundancy in the database.

[0020] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A large-scale database data cleaning and processing system, characterized in that, include The analysis module analyzes the similarity between product titles and descriptions; A definition module, which is used to define the product data model; A database, which is used to store various types of data.

2. The large-scale database data cleaning and processing system according to claim 1, characterized in that, The analysis module uses NLP algorithms to perform timed analysis on the product and provides relevant prompts based on the analysis results.

3. The large-scale database data cleaning and processing system according to claim 1, characterized in that, The definition module uses Apache Flink technology to define the data model of the product, compares the product content, assigns a score, and stores it in the database.

4. The large-scale database data cleaning and processing system according to claim 1, characterized in that, The database is updated periodically with tagged records to ensure database uptime.

5. A method for cleaning and processing large-scale database data, characterized in that, Includes the following steps: Step 1: After the product is analyzed by NLP algorithms, it is identified that the product has duplicate content, and then pushed to users; Step 2: After a user publishes a product, the system compares the data with the data in the database using NLP algorithms, and generates attribute codes using Apache Flink technology, which are then compared with historical product codes in the database.

6. Step 3: After the product is compared with the database, a verification report is generated and sent to the user for confirmation on whether to publish or save.

Images