Data duplicate checking method and device and electronic equipment

Abstract

The embodiment of the invention provides a data duplicate checking method and device and electronic equipment. The method is applied to a data information management system, and the data information management system is internally provided with a preset data engine and a database; comprising the following steps: acquiring newly-added user data in a system, segmenting the newly-added user data, and identifying to obtain data contents corresponding to different data fields; synchronizing the data content to a preset data engine, and calling a query grammar preset by the data engine to search historical documents in the data engine; if the same data content is not searched in the historical document of the data engine, calling a structured query language to search a database; if no data identical to the data content exists in the database, newly-added user data corresponding to the data content is stored in the database, duplicate checking before data falling into the database is added after duplicate checking is conducted through a data engine, and whether the data is unique or not can be confirmed by checking the database under the condition that the data engine cannot check the data; the problem of data repetition can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data retrieval, and in particular to a data duplication checking method, device and electronic equipment. Background Art

[0002] Existing customer relationship management systems often use the elasticsearch engine for duplicate checking. The premise of using the elasticsearch engine for duplicate checking is that the data needs to be synchronized to the elasticsearch engine when adding and updating data. However, there may be problems such as failure and delay during the synchronization process, which makes it impossible to synchronize data effectively in some scenarios, which may also lead to inaccurate duplicate checking results. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a data duplication checking method, device and electronic device, which adds a duplication check before the data is stored in the database after using the data engine to check for duplication. If the data engine cannot find the data, it can be confirmed whether the data is unique by checking the database, which can effectively improve the problem of data duplication.

[0004] In the first aspect, an embodiment of the present invention provides a data duplication checking method, which is applied to a data information management system, and the data information management system has a preset data engine and database built in; the method includes: obtaining new user data in the system, and segmenting the new user data to identify data content corresponding to different data fields; synchronizing the data content to a preset data engine, and calling the query syntax preset by the data engine to search historical documents in the data engine; if the same data content is not found in the historical documents of the data engine, calling the structured query language to search the database; if there is no data identical to the data content in the database, saving the new user data corresponding to the data content in the database.

[0005] Furthermore, before obtaining the newly added user data in the system, the method also includes: determining whether the system has a duplicate checking function turned on; if so, obtaining the newly added user data in the system.

[0006] Furthermore, the method also includes: if the system does not enable the duplicate checking function, the newly added user data is saved in the database.

[0007] Furthermore, the data engine is a distributed search and analysis engine; the query syntax preset in the data engine is called to search the historical documents in the data engine, including: obtaining the data content to be searched, and determining the data type of the data content; determining the query syntax corresponding to the data field based on the data type; and calling the query syntax to perform a global search on the historical documents in the data engine.

[0008] Furthermore, the method of calling query syntax to perform a global search for historical documents in a data engine includes: obtaining identifiers of historical documents, wherein the identifiers correspond to different historical fields; comparing the identifiers with the data content one by one based on the query syntax to determine whether there is historical content in the identifiers that is identical to the data content; if so, terminating the search and outputting the search results to the interactive interface of the data information management system to prompt the existence of duplicate data content.

[0009] Furthermore, the method further comprises: if there is no historical content identical to the data field in the identifier, calling a structured query language to search the database.

[0010] Furthermore, the structured query language is called to search the database, including: obtaining the data content to be searched and determining the data relationship of the data content; based on the data relationship, searching all database contents stored under the data relationship in the database; comparing the database content with the data fields one by one to determine whether there is database content identical to the data content in the database.

[0011] In the second aspect, an embodiment of the present invention further provides a data duplication checking device, which is applied to a data information management system and is used to implement any of the above methods, including: an acquisition module, used to acquire new user data in the system, and segment the new user data to identify different data fields; a first search module, used to synchronize the data field to a preset data engine, and call the query syntax preset by the data engine to search the historical documents in the data engine; a second search module, if the same data field is not found in the historical documents of the data engine, then call the structured query language to search the database; a storage module, used to save the new user data corresponding to the data field into the database if there is no field identical to the data field in the database.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements any of the above methods when executing the computer program.

[0013] In a fourth aspect, an embodiment of the present invention further provides a computer-readable medium, in which a program code is stored, and when the program code is executed by a processor, it is used to implement the method of any one of claims 1-7.

[0014] The embodiments of the present invention bring the following beneficial effects:

[0015] An embodiment of the present invention provides a data duplication checking method, which is applied to a data information management system, which has a preset data engine and database built in; the method comprises: acquiring new user data in the system, segmenting the new user data, identifying data content corresponding to different data fields; synchronizing the data content to a preset data engine, calling a query syntax preset by the data engine to search historical documents in the data engine; if the same data content is not found in the historical documents of the data engine, calling a structured query language to search the database; if there is no data identical to the data content in the database, saving the new user data corresponding to the data content into the database, adding a duplicate check before the data is stored in the database after using the data engine to check for duplicates, and confirming whether the data is unique by checking the database when the data engine cannot find the data, which can effectively improve the problem of data duplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A schematic diagram of a flow chart of a data duplication checking method provided by an embodiment of the present invention;

[0018] Figure 2 A schematic diagram of a flow chart of another data duplication checking method provided by an embodiment of the present invention;

[0019] Figure 3 A schematic diagram of the structure of a data duplication checking device provided by an embodiment of the present invention;

[0020] Figure 4 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] In existing data information management systems, such as existing CRM (Customer Relationship Management) systems that use the elasticsearch engine for duplicate checking, the premise of using elasticsearch for duplicate checking is that data needs to be synchronized to elasticsearch when adding and updating data. However, there may be problems such as failure and delay in the synchronization process, which may make the data inconsistent in some scenarios, resulting in inaccurate duplicate checking results.

[0024] In view of this, the purpose of the present invention is to provide a data duplication checking method, device and electronic device, which adds a duplication check before the data is stored in the database after using the data engine to check for duplication. If the data engine cannot find the data, it can be confirmed whether the data is unique by checking the database, which can effectively improve the problem of data duplication.

[0025] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0026] An embodiment of the present invention provides a data duplication checking method. Figure 1 A flowchart of a data duplication checking method provided by an embodiment of the present invention. The method is applied to a data information management system, which has a preset data engine and database built in. The method comprises the following steps:

[0027] Step S101, acquiring newly added user data in the system, segmenting the newly added user data, and identifying data contents corresponding to different data fields;

[0028] Specifically, taking a CRM system as an example, when a user enters a customer information named Zhang San, the information "Name: Zhang San" can be added through the above-mentioned new user data; the data content under the corresponding "Name" data field can be extracted as "Zhang San".

[0029] Step S102, synchronizing the data content to a preset data engine, and calling a query syntax preset by the data engine to search for historical documents in the data engine;

[0030] Specifically, after determining that it is necessary to enter the duplicate checking logic, you can first check for duplicates through the preset Elasticsearch data engine. Through the Elasticsearch query syntax, build query conditions to search for historical documents in the data engine. Each historical document has a unique identifier that records the content after the data is synchronized. Elasticsearch will return document results that match the query conditions. If the same data is found, it will be returned, and the system's external display interface will prompt the existence of the same data content.

[0031] Step S103, if the same data content is not found in the historical documents of the data engine, then the structured query language is called to search the database;

[0032] Specifically, the database provided in the embodiment of the present application can be a relational database, such as a MySQL database. The corresponding data content can be queried according to the above-mentioned customer name, mobile phone number and other data fields. In actual applications, the database can be queried through SQL statements. If one or more data can be found in the database, it is considered that duplicate data exists. If so, the data will not be saved and a prompt will be given that the same data exists, and the original existing data will not be processed. If the same data cannot be found, the customer data is recorded in the database and the processing ends.

[0033] Step S104: if there is no data with the same content as the data in the database, the newly added user data corresponding to the data content is saved in the database.

[0034] An embodiment of the present invention provides a data duplication checking method, which is applied to a data information management system, which has a preset data engine and database built in; the method comprises: acquiring new user data in the system, segmenting the new user data, identifying data content corresponding to different data fields; synchronizing the data content to a preset data engine, calling a query syntax preset by the data engine to search historical documents in the data engine; if the same data content is not found in the historical documents of the data engine, calling a structured query language to search the database; if there is no data identical to the data content in the database, saving the new user data corresponding to the data content into the database, adding a duplicate check before the data is stored in the database after using the data engine to check for duplicates, and confirming whether the data is unique by checking the database when the data engine cannot find the data, which can effectively improve the problem of data duplication.

[0035] For ease of understanding, Figure 1 Based on this, the embodiment of the present invention provides another data duplication checking method, such as Figure 2 The flowchart of another data duplication checking method shown includes the following steps:

[0036] Step S201, determining whether the system has enabled the duplicate checking function;

[0037] Step S202: if yes, then obtain the newly added user data in the system.

[0038] In actual applications, if the system does not have the duplicate checking function enabled, the newly added user data will be saved in the database.

[0039] Step S203, if the same data content is not found in the historical documents of the data engine, then the structured query language is called to search the database;

[0040] Step S204, synchronizing the data content to a preset data engine, and calling a query syntax preset by the data engine to search for historical documents in the data engine;

[0041] In actual applications, the data engine is a distributed search and analysis engine; calling the query syntax preset by the data engine to search for historical documents in the data engine may include the following steps A1-A3:

[0042] Step A1, obtaining the data content to be searched and determining the data type of the data content;

[0043] Step A2, determining a query syntax corresponding to the data field based on the data type;

[0044] Step A3, calling the query syntax to perform a global search on the historical documents in the data engine.

[0045] Specifically, the above method for calling the query syntax to perform a global search on historical documents in the data engine includes the following steps A31-A33:

[0046] Step A31, obtaining an identifier of a historical document, wherein the identifier corresponds to different historical fields;

[0047] Step A32, comparing the identifier with the data content one by one based on the query syntax to determine whether there is historical content in the identifier that is identical to the data content;

[0048] Step A33: if yes, terminate the search and output the search results to the interactive interface of the data information management system to prompt the existence of duplicate data content;

[0049] Step A34: If the identifier does not contain the same historical content as the data field, the structured query language is called to search the database.

[0050] Step S205, if the same data content is not found in the historical documents of the data engine, then the structured query language is called to search the database;

[0051] Specifically, calling the structured query language to search the database may include the following steps C1-C3:

[0052] Step C1, obtaining the data content to be searched and determining the data relationship of the data content;

[0053] Step C2, based on the data relationship, searching all database contents stored under the data relationship in the database;

[0054] Step C3, compare the database content with the data fields one by one to determine whether there is database content identical to the data content in the database.

[0055] Step S206: If there is no data with the same content as the data in the database, the newly added user data corresponding to the data content is saved in the database.

[0056] Based on the above method embodiment, the present invention also provides a data duplication checking device. Figure 3 The structural schematic diagram of a data duplication checking device shown is applied to a data information management system and is used to implement any of the above methods, including:

[0057] The acquisition module 301 is used to acquire the newly added user data in the system, and segment the newly added user data to identify different data fields;

[0058] A first search module 302 is used to synchronize data fields to a preset data engine and call a query syntax preset by the data engine to search historical documents in the data engine;

[0059] The second search module 303 calls the structured query language to search the database if the same data field is not found in the historical documents of the data engine;

[0060] The storage module 304 is used to save the newly added user data corresponding to the data field into the database if there is no field identical to the data field in the database.

[0061] The data duplication checking device provided in the embodiment of the present invention has the same technical features as the data duplication checking method provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0062] The present application also provides an electronic device, such as Figure 4 As shown, it is a schematic diagram of the structure of the electronic device, wherein the electronic device includes a processor 121 and a memory 120, the memory 120 stores computer executable instructions that can be executed by the processor 121, and the processor 121 executes the computer executable instructions to implement the above-mentioned data duplication checking method.

[0063] exist Figure 4 In the illustrated embodiment, the electronic device further includes a bus 122 and a communication interface 123 , wherein the processor 121 , the communication interface 123 and the memory 120 are connected via the bus 122 .

[0064] Among them, the memory 120 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 123 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 122 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 122 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0065] The processor 121 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 121. The above processor 121 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor 121 reads the information in the memory and completes the steps of the data duplication checking method of the aforementioned embodiment in combination with its hardware.

[0066] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned data duplication checking method. The specific implementation can be found in the aforementioned method embodiment, which will not be repeated here.

[0067] The software upgrade method, device and computer program product of the electronic device provided in the embodiments of the present application include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0068] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0069] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0070] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", and "secondary" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0071] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A data duplication checking method, characterized in that: The method is applied to a data information management system, wherein the data information management system has a preset data engine and a database built in; the method comprises: Acquire new user data in the system, segment the new user data, and identify data contents corresponding to different data fields; Synchronize the data content to a preset data engine, and call the query syntax preset by the data engine to search for historical documents in the data engine; If the same data content is not found in the historical documents of the data engine, calling the structured query language to search the database; If there is no data with the same data content in the database, the newly added user data corresponding to the data content is saved in the database.

2. The method according to claim 1, characterized in that Before acquiring the newly added user data in the system, the method further includes: Determine whether the system has enabled a duplicate checking function; If yes, then the newly added user data in the system is obtained.

3. The method according to claim 2, characterized in that The method further comprises: If the system does not have a duplicate checking function enabled, the newly added user data will be saved in the database.

4. The method according to claim 1, characterized in that: The data engine is a distributed search and analysis engine; The calling of the query grammar preset by the data engine to search the historical documents in the data engine includes: Obtaining the data content to be searched, and determining the data type of the data content; Determine a query syntax corresponding to the data field based on the data type; The query syntax is called to perform a global search on historical documents in the data engine.

5. The method according to claim 4, characterized in that The method of calling the query grammar to perform a global search on historical documents in the data engine includes: Obtaining an identifier of the historical document, wherein the identifier corresponds to different historical fields; Compare the identifier with the data content one by one based on the query grammar to determine whether there is historical content in the identifier that is identical to the data content; If yes, the search is terminated and the search results are output to the interactive interface of the data information management system to prompt the existence of duplicate data content.

6. The method according to claim 5, characterized in that The method further comprises: If the identifier does not contain the same historical content as the data field, a structured query language is invoked to search the database.

7. The method according to claim 1, characterized in that The calling of structured query language to search the database includes: Acquire the data content to be searched, and determine the data relationship of the data content; Based on the data relationship, searching all database contents stored under the data relationship in the database; The database content is compared with the data fields one by one to determine whether there is database content identical to the data content in the database.

8. A data duplication checking device, characterized in that: The device is applied to a data information management system, and is used to implement the method described in any one of claims 1 to 7, including: An acquisition module is used to acquire new user data in the system, and segment the new user data to identify different data fields; A first search module, used to synchronize the data field to a preset data engine, and call a query syntax preset by the data engine to search for historical documents in the data engine; A second search module, calling a structured query language to search the database if the same data field is not found in the historical documents of the data engine; The storage module is used to save the newly added user data corresponding to the data field into the database if there is no field identical to the data field in the database.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable medium, characterized in that The computer-readable medium stores program codes, and the program codes are used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images