Data reference intelligent verification method, system and device, medium and program product

Abstract

The invention discloses a data reference intelligent verification method, system and device, a medium and a program product, and the method comprises the steps: obtaining an original reference table having a reference relationship with certain data in a database according to a reference relationship in a metadata reference model, and obtaining an original reference table result set; dynamically calculating the number of threads needing to be started according to the number of reference tables in the result set and reference table verification time consumption; according to the historical verification time consumption of each reference table, intelligent grouping is carried out according to the thread count, so that the sum of the verification time consumption of the reference tables in each group is basically equal; a multi-thread verification execution step is started, in execution, as long as one thread hits the reference table with the reference relation, a result is returned, a thread termination mark is set, other threads terminate execution after receiving the thread termination mark, and rapid intelligent verification of data reference is completed. According to the method, through the multi-thread steps of intelligent grouping and intelligent sorting, rapid and intelligent verification of data reference in a massive business number scene can be realized, and the verification efficiency is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data reference intelligent verification method, system, device, medium and program product. Background Art

[0002] In the business system, when you need to delete a basic file, you need to check whether the file data is referenced by the business document. If it is referenced, the file cannot be deleted. If a file has a large number of file references, when deleting the file data, the relevant reference table will be queried based on the metadata reference relationship, and then the reference verification will be performed one by one according to the original reference table result set composed of the relevant reference tables until the actual reference is verified. Only then will the query be terminated and the reference verification result will be returned.

[0003] However, the current archival data reference verification method based on metadata reference query has the following problems: (1) When there are a large number of archives referenced by business (such as user archives), in the extreme case that the last reference table has reference data, when verifying whether the user archive is referenced, the last reference table will be polled to verify the existence of the reference, which will cause serious efficiency problems; (2) Fault-tolerant mechanism: Under the premise of using metadata reference verification, there may be a situation where the reference exists but the entity table does not exist. When querying, query errors will occur. Frequent errors may cause problems such as database connection interruption. The PG kernel database will experience query connection interruption, resulting in business interruption; (3) The reference table data is too large, and the speed of reference query on these tables is very slow. When analyzing the efficiency after going online, a lot of positioning based on performance analysis tools is required to locate the problems caused by these reference tables, which is very inconvenient and inefficient.

[0004] It can be seen that the above existing reference verification methods are still inconvenient and defective in use, and are in urgent need of further improvement. How to create a new data reference intelligent verification method, system, device, medium and program product that can overcome the above problems of the existing methods and greatly improve the efficiency and high availability of the methods has become a goal that the industry urgently needs to improve. Summary of the invention

[0005] In view of this, the present invention provides a data reference intelligent verification method, which at least partially solves the problems existing in the prior art.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for intelligent verification of data references, which is used in a scenario with massive business data and specifically includes the following steps:

[0007] According to the reference relationship in the metadata reference model, an original reference table in the database having a reference relationship with certain data is obtained to obtain an original reference table result set;

[0008] Calculate the number of threads to be enabled based on the number of reference tables in the original reference table result set and the dynamics of the historical verification time of the reference tables;

[0009] Then, based on the historical verification time of each reference table, the reference tables in the original reference table result set are intelligently grouped according to the calculated number of enabled threads, so that the total verification time of the reference tables in each group is basically equal;

[0010] Distribute each intelligent grouping result to a thread and start the multi-threaded verification execution step;

[0011] During the multi-threaded verification execution process, as long as one thread hits the reference table with a reference relationship, the result is returned and the thread termination mark is set. Other threads terminate execution after receiving the thread termination mark, completing the fast and intelligent verification of the data reference.

[0012] According to a specific implementation method of the embodiment of the present disclosure, after the original reference table result set is intelligently grouped, it also includes: a step of intelligently sorting the reference tables in each intelligent group, wherein the intelligent sorting is performed based on the priority of hot and cold data of each reference table, and the priority of hot and cold data is performed based on the weight of the number of times the reference table is referenced, and the more times it is referenced, the greater the weight and the higher the priority.

[0013] According to a specific implementation method of an embodiment of the present disclosure, before the original reference table result set is intelligently grouped, the step of first completing the reference table in the blacklist is also included; wherein, the method for establishing the blacklist is that during the execution of multi-threaded verification, if the verified reference table does not exist in the database, then the reference table is included in the blacklist.

[0014] According to a specific implementation of the embodiment of the present disclosure, during the multi-threaded verification execution process, the reference time of the reference table with a reference relationship is recorded and output to the dashboard to form a reference table query efficiency dashboard with dynamic sorting of reference time. In addition, the method also includes the step of performing database index processing or excluding exception processing on the reference table with long time consumption in the reference table query efficiency dashboard.

[0015] According to a specific implementation method of an embodiment of the present disclosure, the original reference table result set, dynamic time-consuming data of the reference table, hot and cold data of the reference table, and blacklist data obtained in the method are all persisted using a data source isolation cache mechanism, wherein the data source isolation cache mechanism of a single-machine cluster uses a file cache mechanism for caching, and the data source isolation cache mechanism of a cloud-native environment uses redis third-party middleware for caching.

[0016] In a second aspect, the embodiments of the present disclosure provide a data reference intelligent verification system, which is used for rapid intelligent verification of data references in a scenario with a large number of business data, and specifically includes:

[0017] The reference relationship query module is configured to obtain an original reference table in the database that has a reference relationship with certain data according to the reference relationship in the metadata reference model, and obtain an original reference table result set;

[0018] A thread number calculation module is configured to calculate the number of threads to be enabled according to the number of reference tables in the original reference table result set and the dynamic time consumption of historical verification of the reference tables;

[0019] The intelligent grouping module is configured to intelligently group the reference tables in the original reference table result set according to the historical verification time of each reference table according to the calculated number of enabled threads, so that the total verification time of the reference tables in each group is substantially equal;

[0020] A multi-thread processing module is configured to distribute each intelligent grouping result to a thread and start a multi-thread verification execution step;

[0021] The result processing module is configured to be used in the multi-threaded verification execution process. As long as one thread hits the reference table with a reference relationship, the result is returned and the thread termination mark is set. Other threads terminate execution after receiving the thread termination mark, completing the fast and intelligent verification of the data reference.

[0022] According to a specific implementation of the embodiment of the present disclosure, the system further includes:

[0023] An intelligent sorting module is configured to perform intelligent sorting on the reference tables in each intelligent group. The intelligent sorting is performed based on the priority of hot and cold data in each reference table. The priority of hot and cold data is based on the weight of the number of times the reference table is referenced. The more times the reference table is referenced, the greater the weight and the higher the priority.

[0024] The blacklist establishment module is configured to include the reference table in the blacklist if the reference table to be checked does not exist in the database during the multi-threaded check execution process;

[0025] The reference table query efficiency dashboard module is configured to receive and output the reference time consumption data of the reference table with reference relationship in the multi-threaded verification process, and dynamically sort the reference time consumption data;

[0026] The data persistence processing module is configured to perform persistence processing on the original reference table result set, reference table dynamic time-consuming data, reference table hot and cold data and blacklist data obtained in the method using a data source isolation cache mechanism.

[0027] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0028] at least one processor; and,

[0029] a memory communicatively connected to the at least one processor; wherein,

[0030] The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor implements the data reference intelligent verification method described in any one of the first aspect or any one of the implementations of the first aspect.

[0031] In a fourth aspect, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by at least one processor, the at least one processor executes the data reference intelligent verification method in the aforementioned first aspect or any implementation method of the first aspect.

[0032] In the fifth aspect, the embodiments of the present disclosure further provide a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the data reference intelligent verification method in the aforementioned first aspect or any implementation method of the first aspect.

[0033] The data reference intelligent verification method of the present invention is used in the scenario of massive business data. It can realize rapid verification of the reference table in the original reference table result set through multi-threaded processing steps of intelligent grouping and intelligent sorting, so that the verification efficiency is significantly improved. For example, if the user profile data with references is deleted in a big data environment, the existing verification method takes about 3 to 10 minutes to complete, while the verification method of the present invention only takes 1 second to 2 minutes, which greatly improves the efficiency.

[0034] By adding a blacklist and pre-checking the blacklist, the error messages that appear in the execution log console are significantly reduced, and there are even no related abnormal errors.

[0035] By adding a reference table query efficiency dashboard with dynamic sorting of reference time consumption, you can view the time-consuming reference query table information at any time, which is convenient for further analysis and processing and more flexible and convenient.

[0036] Furthermore, by using a data source isolation cache mechanism to perform persistent processing on the data obtained in the method, high availability of the verification method of the present invention can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0038] Figure 1 A schematic diagram of the verification principle of the intelligent verification method for archival data references according to an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a specific execution flow of the intelligent verification method for archival data references according to an embodiment of the present invention;

[0040] Figure 3 This is a diagram of the architecture of the intelligent verification system for archival data references according to an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0043] In order to improve the efficiency of data reference verification, the present invention provides a data reference intelligent verification method, and specifically takes the intelligent verification of archive data (such as user archive) reference as an example to describe the present invention in detail.

[0044] See attached Figure 1 and 2 As shown, the intelligent verification method for archival data references in this embodiment is to quickly verify the original reference table result set queried under the condition of massive business data scenario, and the specific verification steps include:

[0045] Firstly, according to the reference relationship in the metadata reference model, the original reference table in the database that has a reference relationship with certain archive data is obtained to obtain the original reference table result set.

[0046] Secondly, add hot and cold data priority sorting mechanism and reference table time consumption mechanism. The hot and cold data priority sorting mechanism is that when a reference table is queried and a reference relationship exists, the weight of the reference table is added by one to obtain the weight value of each reference table, that is, the more times it is referenced, the greater the weight value and the higher the priority; the reference table time consumption mechanism is to record the reference table verification time consumption and realize dynamic overwriting to obtain the historical verification time consumption value of each reference table.

[0047] Then, the number of threads to be enabled is calculated for the original reference table result set according to the number of reference tables existing therein and the dynamic time consumption of historical verification of the reference tables.

[0048] Then, based on the historical verification time of each reference table, the reference tables in the original reference table result set are intelligently grouped according to the calculated number of enabled threads, so that the total verification time of the reference tables in each group is basically equal.

[0049] Then, the reference tables in each intelligent group are intelligently sorted. The intelligent sorting is based on the hot and cold data priority sorting mechanism of each reference table, that is, the weight values ​​are sorted in descending order from large to small, and the references with the largest weight are queried first.

[0050] Next, each intelligent grouping result is distributed to a thread, and the multi-threaded verification execution step is started, so that each thread executes the verification step independently.

[0051] During the multi-threaded verification execution process, as long as one thread hits the reference table with a reference relationship, the result is returned and the thread termination mark true is set. Other threads terminate execution after receiving the thread termination mark true, completing the fast and intelligent verification of the archive data reference.

[0052] Preferably, before the original reference table result set is intelligently grouped in the above method, the step of verifying the reference table in the blacklist is also included. When there is no reference in the reference table in the blacklist, the reference table in the blacklist is filtered and deleted through a table filter, such as an oracle database filter, a sqlserver database filter, a PG database filter, etc., and then the above-mentioned intelligent grouping, intelligent sorting, and multi-threaded verification steps are continued. The method for establishing the blacklist is that during the multi-threaded verification execution process, if the verified reference table does not exist in the database query table or view, the reference table is included in the blacklist. In addition, when restarting the service, it is also necessary to start an independent thread to verify whether the reference table in the blacklist still exists to prevent someone from initializing the table in the blacklist into the library. The addition of the blacklist mechanism can effectively avoid the problem of database connection interruption caused by frequent errors in the execution log console.

[0053] The above method also includes a mechanism for quickly viewing the reference table query efficiency dashboard. The mechanism is to record the reference time of the reference table with a reference relationship and output it to the dashboard, forming a reference table query efficiency dashboard with dynamic sorting of reference time. And it includes database index processing or exception exclusion processing for the reference table with long time consumption in the dashboard, that is, it can achieve the effect of targeted and quick viewing and processing of the reference table with long time consumption.

[0054] The original reference table result set, dynamic time-consuming data of the reference table, hot and cold data of the reference table, and blacklist data established in the above method are all persisted using the data source isolation cache mechanism to achieve high availability of the method. Specifically, the data source isolation cache mechanism of the single-machine cluster uses the file cache mechanism for caching, and the data source isolation cache mechanism of the cloud native environment uses the redis third-party middleware for caching.

[0055] Figure 3 The fast intelligent verification system 300 of archive data references provided by the present invention is shown, including a reference relationship query module 310, a thread number calculation module 320, an intelligent grouping module 330, an intelligent sorting module 340, a multi-thread processing module 350 and a result processing module 360.

[0056] The reference relationship query module 310 is used to obtain the original reference table in the database that has a reference relationship with certain data according to the reference relationship in the metadata reference model, and obtain the original reference table result set;

[0057] The thread number calculation module 320 is used to calculate the number of threads to be enabled according to the number of reference tables in the original reference table result set and the dynamic time consumption of historical verification of the reference tables;

[0058] An intelligent grouping module 330 is used to intelligently group the reference tables in the original reference table result set according to the historical verification time of each reference table according to the calculated number of enabled threads, so that the total verification time of the reference tables in each group is substantially equal;

[0059] An intelligent sorting module 340, used to sort the reference tables in each intelligent group in descending order according to the weight value of each reference table;

[0060] The multi-thread processing module 350 is used to distribute each intelligent grouping result to a thread and start the multi-thread verification execution step;

[0061] The result processing module 360 ​​is used in the multi-threaded verification execution process. As long as one thread hits the reference table with a reference relationship, the result is returned and the thread termination mark is set. Other threads terminate execution after receiving the thread termination mark, completing the fast and intelligent verification of the data reference.

[0062] In an embodiment of the present invention, the system further includes:

[0063] The blacklist creation module is used to create a reference table blacklist. If the reference table does not exist in the database during reference verification, it will be included in the blacklist.

[0064] The reference table query efficiency dashboard module is used to receive and output the reference time consumption data of the reference table with a reference relationship during the multi-threaded verification processing, and dynamically sort the reference time consumption.

[0065] The data persistence processing module is used to implement data caching for the established original reference table result set, reference table dynamic time-consuming data, reference table hot and cold data, and blacklist data using a data source isolation cache mechanism.

[0066] See also Figure 4 The embodiment of the present disclosure further provides an electronic device 40, the electronic device 40 comprising:

[0067] at least one processor; and,

[0068] a memory communicatively connected to the at least one processor; wherein,

[0069] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the intelligent verification method for archival data reference in the aforementioned method embodiment.

[0070] The embodiment of the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the intelligent verification method for archival data reference in the aforementioned method embodiment.

[0071] The embodiments of the present disclosure also provide a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the intelligent verification method for archival data references in the aforementioned method embodiment.

[0072] Reference below Figure 4 , which shows a schematic diagram of the structure of an electronic device 40 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0073] like Figure 4As shown, the electronic device 40 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 40 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0074] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 40 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 4 shows an electronic device 40 having various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0075] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0076] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0077] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0078] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains at least two Internet Protocol addresses; sends a node evaluation request including the at least two Internet Protocol addresses to a node evaluation device, wherein the node evaluation device selects an Internet Protocol address from the at least two Internet Protocol addresses and returns it; receives the Internet Protocol address returned by the node evaluation device; wherein the obtained Internet Protocol address indicates an edge node in a content distribution network.

[0079] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: receives a node evaluation request including at least two Internet Protocol addresses; selects an Internet Protocol address from the at least two Internet Protocol addresses; and returns the selected Internet Protocol address; wherein the received Internet Protocol address indicates an edge node in a content distribution network.

[0080] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0081] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0082] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0083] It should be noted that the above embodiments and features therein may be combined with each other if there is no conflict.

[0084] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A data reference intelligent verification method, characterized in that: The method is used in a scenario with a large number of services, and specifically includes the following steps: According to the reference relationship in the metadata reference model, an original reference table in the database having a reference relationship with certain data is obtained to obtain the original reference table result set; Calculate the number of threads to be enabled based on the number of reference tables in the original reference table result set and the dynamics of the historical verification time of the reference tables; Then, based on the historical verification time of each reference table, the reference tables in the original reference table result set are intelligently grouped according to the calculated number of enabled threads, so that the total verification time of the reference tables in each group is basically equal; Distribute each intelligent grouping result to a thread and start the multi-threaded verification execution step; During the multi-threaded verification execution process, as long as one thread hits the reference table with a reference relationship, the result is returned and the thread termination mark is set. Other threads terminate execution after receiving the thread termination mark, completing the fast and intelligent verification of the data reference.

2. The data reference intelligent verification method according to claim 1, characterized in that: After intelligent grouping of the original reference table result set, it also includes: The step of intelligently sorting the reference tables in each intelligent grouping is performed according to the priority of hot and cold data in each reference table. The priority of hot and cold data is based on the weight of the number of times the reference table is referenced. The more times the reference table is referenced, the greater the weight and the higher the priority.

3. The data reference intelligent verification method according to claim 2, characterized in that: Before the original reference table result set is intelligently grouped, the step of first verifying the reference table in the blacklist is also included; The method for establishing the blacklist is as follows: during the execution of the multi-threaded verification, if the reference table to be verified does not exist in the database, the reference table is included in the blacklist.

4. The data reference intelligent verification method according to claim 3 is characterized in that: During the multi-threaded verification execution process, the reference time of the reference table with a reference relationship is recorded and output to the dashboard to form a reference table query efficiency dashboard with dynamic sorting of reference time; The method further comprises the step of performing database index processing or exception exclusion processing on the long time-consuming reference table in the reference table query efficiency dashboard.

5. The data reference intelligent verification method according to claim 4, characterized in that: The original reference table result set, reference table dynamic time-consuming data, reference table hot and cold data, and blacklist data obtained in the method are all persisted using a data source isolation cache mechanism. Among them, the data source isolation cache mechanism of a single-machine cluster uses a file cache mechanism for caching, and the data source isolation cache mechanism of a cloud-native environment uses redis third-party middleware for caching.

6. A data reference intelligent verification system, characterized in that: The system is used for fast and intelligent verification of data references in scenarios with massive business data, specifically including: The reference relationship query module is configured to obtain an original reference table in the database that has a reference relationship with certain data according to the reference relationship in the metadata reference model, and obtain an original reference table result set; A thread number calculation module is configured to calculate the number of threads to be enabled according to the number of reference tables in the original reference table result set and the dynamic time consumption of historical verification of the reference tables; The intelligent grouping module is configured to intelligently group the reference tables in the original reference table result set according to the historical verification time of each reference table according to the calculated number of enabled threads, so that the total verification time of the reference tables in each group is substantially equal; A multi-thread processing module is configured to distribute each intelligent grouping result to a thread and start a multi-thread verification execution step; The result processing module is configured to be used in the multi-threaded verification execution process. As long as one thread hits the reference table with a reference relationship, the result is returned and the thread termination mark is set. Other threads terminate execution after receiving the thread termination mark, completing the fast and intelligent verification of the data reference.

7. The data reference intelligent verification system according to claim 6, characterized in that: The system further comprises: An intelligent sorting module is configured to perform intelligent sorting on the reference tables in each intelligent group. The intelligent sorting is performed based on the priority of hot and cold data in each reference table. The priority of hot and cold data is based on the weight of the number of times the reference table is referenced. The more times the reference table is referenced, the greater the weight and the higher the priority. A blacklist building module is configured to include the reference table in the blacklist if the reference table to be checked does not exist in the database during the execution of the multi-threaded check; The reference table query efficiency dashboard module is configured to receive and output the reference time consumption data of the reference table with reference relationship in the multi-threaded verification process, and dynamically sort the reference time consumption data; The data persistence processing module is configured to perform persistence processing on the original reference table result set, reference table dynamic time-consuming data, reference table hot and cold data and blacklist data obtained in the method using a data source isolation cache mechanism.

8. An electronic device, characterized in that: The electronic device includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor executes the data reference intelligent verification method as described in any one of claims 1 to 5.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which, when executed by at least one processor, cause the at least one processor to perform the data reference intelligent verification method according to any one of claims 1 to 5.

10. A computer program product, characterized in that The computer program product includes a computer program / instruction, which, when executed by a processor, implements the data reference intelligent verification method according to any one of claims 1 to 5.

Images