Data processing method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of data processing, and relates to a data processing method which comprises the following steps: judging whether a data adding request for a target index database triggered by a user is received or not; wherein the data adding request carries original data; if yes, displaying a preset configuration page; receiving a specified compression algorithm name input in the configuration page by the user; searching an implementation class of a specified compression algorithm corresponding to the specified compression algorithm name from a preset target cache, and calling a specified compression method corresponding to the implementation class of the specified compression algorithm; performing compression processing on the original data based on a specified compression method to obtain corresponding target data; and storing the target data in the target index database. The invention further provides a data processing device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the target data can be stored in a block chain. According to the invention, the flexibility and intelligence of data compression processing are effectively improved.

Description

Technical Field

[0001] The present application relates to the fields of data processing technology, financial technology, and digital medicine, and in particular to data processing methods, devices, computer equipment, and storage media. Background Art

[0002] In highly data-intensive fields like finance and healthcare, the ability to process, store, and retrieve text data is crucial for improving business efficiency and supporting decision making. Apache Lucene, an open-source, high-performance full-text search engine, has been widely used in these fields thanks to its powerful indexing and retrieval capabilities. Lucene significantly improves text data retrieval speeds through its efficient indexing mechanisms and optimized data storage strategies. Furthermore, to effectively manage the storage pressures brought on by massive amounts of data, Lucene integrates multiple compression algorithms to reduce storage space usage.

[0003] However, while the application of compression algorithms in the existing Lucene technology framework significantly improves the system's storage efficiency and retrieval performance, its compression strategies have certain limitations. Specifically, Lucene primarily relies on a fixed compression algorithm for data compression. While this design simplifies system implementation and maintenance, it also limits users' ability to flexibly select or change compression algorithms based on actual needs. In real-world applications, different data types, application scenarios, and even user preferences often have different requirements for compression effectiveness. For example, financial data may prioritize the balance between compression ratio and decryption speed, while medical data may prioritize data integrity and security.

[0004] Therefore, the fixed compression algorithm currently used by Lucene limits the intelligence and flexibility of its data compression to a certain extent, making it difficult to meet the personalized needs of different users. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a data processing method, apparatus, computer device, and storage medium to address the technical problem that the fixed compression algorithm mode adopted by the existing Lucene limits the intelligence and flexibility of its data compression to a certain extent, making it difficult to meet the personalized needs of different users.

[0006] In order to solve the above technical problems, the present application provides a data processing method, which adopts the following technical solutions:

[0007] Determining whether a user-triggered data addition request for a target index library is received; wherein the data addition request carries original data;

[0008] If yes, display the preset configuration page;

[0009] Receiving a specified compression algorithm name entered by the user in the configuration page;

[0010] Searching for an implementation class of a specified compression algorithm corresponding to the specified compression algorithm name from a preset target cache, and calling a specified compression method corresponding to the implementation class of the specified compression algorithm;

[0011] Compressing the original data based on the specified compression method to obtain corresponding target data;

[0012] The target data is stored in the target index library.

[0013] Furthermore, before the step of searching the preset target cache for the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name, the method further includes:

[0014] Call the default plugin loader;

[0015] Scan the plug-in jar files in the preset plug-in directory based on the plug-in loader;

[0016] Obtain the implementation class in the plug-in jar file, and obtain the compression algorithm name corresponding to the implementation class in the plug-in jar file;

[0017] The implementation class and the corresponding compression algorithm name in the plug-in jar file are loaded into the target cache.

[0018] Furthermore, before the step of scanning the plug-in jar file in the preset plug-in directory based on the plug-in loader, the method further includes:

[0019] Obtain a predefined compression interface; wherein the compression interface defines compression, decompression, and the name of the compression algorithm;

[0020] Obtaining multiple preset compression algorithms, and creating implementation classes corresponding to the compression interface for each compression algorithm;

[0021] Generate plug-in jar files corresponding to each compression algorithm based on the implementation class;

[0022] Store the plug-in jar file in the plug-in directory.

[0023] Furthermore, after the step of storing the plug-in jar file in the plug-in directory, the method further includes:

[0024] Determine whether a compression algorithm adding request triggered by a specified user is received; wherein the compression algorithm adding request carries the name of the target compression algorithm;

[0025] If so, extracting the target compression algorithm name from the compression algorithm adding request;

[0026] Determine a target compression algorithm corresponding to the target compression algorithm name;

[0027] The plug-in directory is updated based on the target compression algorithm.

[0028] Furthermore, the step of compressing the original data based on the specified compression method to obtain corresponding target data specifically includes:

[0029] obtaining the original data;

[0030] Performing data cleaning on the original data to obtain corresponding designated data;

[0031] Performing verification processing on the specified data based on a preset verification strategy;

[0032] If the designated data passes the verification, the designated data is compressed based on the designated compression method to obtain the corresponding target data.

[0033] Furthermore, the step of verifying the specified data based on a preset verification strategy specifically includes:

[0034] Performing standardized verification on the specified data;

[0035] If the specified data passes the standard verification, sensitive verification is performed on the specified data;

[0036] If the designated data passes the sensitive verification, it is determined that the designated data passes the verification; otherwise, it is determined that the designated data fails the verification.

[0037] Furthermore, the step of performing sensitive verification on the designated data specifically includes:

[0038] Obtain preset sensitive identification rules;

[0039] Performing item-by-item matching processing on the designated data based on the sensitive identification rule, and determining whether there is sensitive data matching the sensitive identification rule in the designated data;

[0040] If there is no sensitive data matching the sensitive identification rule in the specified data, it is determined that the specified data has passed the sensitive verification; otherwise, it is determined that the specified data has not passed the sensitive verification.

[0041] In order to solve the above technical problems, the present application also provides a data processing device, which adopts the following technical solution:

[0042] A first judgment module is configured to judge whether a user-triggered request for adding data to a target index library is received; wherein the data addition request carries original data;

[0043] Display module, used to display the preset configuration page if yes;

[0044] A receiving module, configured to receive a specified compression algorithm name entered by the user in the configuration page;

[0045] A processing module, configured to search a preset target cache for an implementation class of a specified compression algorithm corresponding to the specified compression algorithm name, and call a specified compression method corresponding to the implementation class of the specified compression algorithm;

[0046] A compression module, configured to compress the original data based on the specified compression method to obtain corresponding target data;

[0047] The first storage module is configured to store the target data in the target index library.

[0048] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0049] Determining whether a user-triggered data addition request for a target index library is received; wherein the data addition request carries original data;

[0050] If yes, display the preset configuration page;

[0051] Receiving a specified compression algorithm name entered by the user in the configuration page;

[0052] Searching for an implementation class of a specified compression algorithm corresponding to the specified compression algorithm name from a preset target cache, and calling a specified compression method corresponding to the implementation class of the specified compression algorithm;

[0053] Compressing the original data based on the specified compression method to obtain corresponding target data;

[0054] The target data is stored in the target index library.

[0055] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0056] Determining whether a user-triggered data addition request for a target index library is received; wherein the data addition request carries original data;

[0057] If yes, display the preset configuration page;

[0058] Receiving a specified compression algorithm name entered by the user in the configuration page;

[0059] Searching for an implementation class of a specified compression algorithm corresponding to the specified compression algorithm name from a preset target cache, and calling a specified compression method corresponding to the implementation class of the specified compression algorithm;

[0060] Compressing the original data based on the specified compression method to obtain corresponding target data;

[0061] The target data is stored in the target index library.

[0062] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0063] The present application first determines whether a data addition request for a target index library triggered by a user is received; wherein the data addition request carries original data; if so, a preset configuration page is displayed; then the specified compression algorithm name entered by the user in the configuration page is received; then the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name is found from the preset target cache, and the specified compression method corresponding to the implementation class of the specified compression algorithm is called; subsequently, the original data is compressed based on the specified compression method to obtain the corresponding target data; finally, the target data is stored in the target index library. When the present application receives a data addition request for a target index library triggered by a user, it displays a preset configuration page and receives the specified compression algorithm name entered by the user in the configuration page, then searches the preset target cache for the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name, and calls the specified compression method corresponding to the implementation class of the specified compression algorithm, and then compresses the original data based on the specified compression method to obtain the corresponding target data, and stores the target data in the target index library. This enables the user to select the corresponding specified compression algorithm according to specific needs and application scenarios to compress the original data that needs to be added to the target index library, effectively improving the flexibility and intelligence of data compression processing, and is conducive to improving the processing performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0065] Figure 1is an exemplary system architecture diagram to which the present application may be applied;

[0066] Figure 2 A flowchart of an embodiment of a data processing method according to the present application;

[0067] Figure 3 is a structural diagram of an embodiment of a data processing device according to the present application;

[0068] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0070] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0071] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0072] like Figure 1 As shown, system architecture 100 may include a terminal device 101, a network 102, and a server 103. Terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. Network 102 is a medium for providing a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0073] The user can use the terminal device 101 to interact with the server 103 via the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0074] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0075] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0076] It should be noted that the data processing method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the data processing device is generally set in the server / terminal device.

[0077] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0078] Continue to refer Figure 2 , shows a flow chart of an embodiment of the data processing method according to the present application. According to different needs, the order of the steps in the flow chart can be changed, and some steps can be omitted. The data processing method provided in the embodiment of the present application can be applied to any scenario that requires data processing, and the data processing method can be applied to products in these scenarios, for example, data compression processing of databases in financial insurance platforms and medical platforms. The data processing method includes the following steps:

[0079] Step S201 : determining whether a data addition request for a target index library triggered by a user is received; wherein the data addition request carries original data.

[0080] In this embodiment, the data processing method is executed on the electronic device (eg Figure 1The server / terminal device shown in the figure) can obtain the data addition request through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future. The designated subject of this application may be a data processing system integrated into an application, or simply referred to as a system. In the business scenarios of the financial field, the above-mentioned applications may include transaction applications, payment applications, claims applications, etc. In the business scenarios of the medical field, the above-mentioned applications may include medical applications, health consultation applications, etc. The above-mentioned target index library may specifically be an Apache Lucene index library. The above-mentioned data addition request is a request triggered by the user in the system interface to add the above-mentioned original data to the target index library.

[0081] Step S202: If yes, display the preset configuration page.

[0082] In this embodiment, the configuration page is a pre-built page containing the settings configuration of the target index library. The configuration interface pre-displays multiple compression algorithms that can be used by the target index library, such as LZ4, Zstandard, Snappy, XZ, etc.

[0083] Step S203: receiving the specified compression algorithm name entered by the user in the configuration page.

[0084] In this embodiment, the user can select a specified compression algorithm name corresponding to a specific specified compression algorithm from all compression algorithms displayed on the configuration page according to specific needs and application scenarios, thereby completing the input process of the specified compression algorithm name.

[0085] Step S204: searching the preset target cache for the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name, and calling the specified compression method corresponding to the implementation class of the specified compression algorithm.

[0086] In this embodiment, after obtaining the specified compression algorithm name, the target cache can be searched and retrieved by searching for and obtaining the implementation class of the specified compression algorithm that matches the specified compression algorithm name. The specified compression method corresponding to the implementation class of the specified compression algorithm can then be called. The target cache data construction process will be described in further detail in subsequent specific embodiments of this application and will not be elaborated on here.

[0087] Step S205 : compressing the original data based on the specified compression method to obtain corresponding target data.

[0088] In this embodiment, the specific implementation process of compressing the original data based on the specified compression method to obtain the corresponding target data will be further described in detail in subsequent specific embodiments of this application and will not be elaborated on here.

[0089] Step S206: storing the target data in the target index library.

[0090] In this embodiment, the compressed target data is stored in a target index library for subsequent rapid retrieval and use.

[0091] The present application first determines whether a data addition request for a target index library triggered by a user is received; wherein the data addition request carries original data; if so, a preset configuration page is displayed; then the specified compression algorithm name entered by the user in the configuration page is received; then the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name is found from the preset target cache, and the specified compression method corresponding to the implementation class of the specified compression algorithm is called; subsequently, the original data is compressed based on the specified compression method to obtain the corresponding target data; finally, the target data is stored in the target index library. When the present application receives a data addition request for a target index library triggered by a user, it displays a preset configuration page and receives the specified compression algorithm name entered by the user in the configuration page, then searches the preset target cache for the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name, and calls the specified compression method corresponding to the implementation class of the specified compression algorithm, and then compresses the original data based on the specified compression method to obtain the corresponding target data, and stores the target data in the target index library. This enables the user to select the corresponding specified compression algorithm according to specific needs and application scenarios to compress the original data that needs to be added to the target index library, effectively improving the flexibility and intelligence of data compression processing, and is conducive to improving the processing performance of the system.

[0092] In some optional implementations, before step S204, the electronic device may further perform the following steps:

[0093] Calls the default plugin loader.

[0094] In this embodiment, a plug-in loader is pre-written, and the plug-in loader automatically scans the jar files in the plug-in directory when the application of the integrated system is started.

[0095] The plug-in jar file in the preset plug-in directory is scanned based on the plug-in loader.

[0096] In this embodiment, the plug-in directory is a designated directory for placing the plug-in jar file. The construction process of the plug-in jar file will be described in further detail in subsequent specific embodiments of the present application, and will not be elaborated on here.

[0097] Obtain the implementation class in the plug-in jar file, and obtain the compression algorithm name corresponding to the implementation class in the plug-in jar file.

[0098] In this embodiment, the implementation class in the jar file can be loaded using Java's reflection mechanism, and the compression algorithm name corresponding to the implementation class in the plug-in jar file can be obtained by following the pre-set correspondence between these implementation classes and their respective compression algorithm names.

[0099] The implementation class and the corresponding compression algorithm name in the plug-in jar file are loaded into the target cache.

[0100] In this embodiment, the target cache can specifically utilize a memory or Redis database. After obtaining the plug-in JAR file, the implementation class and corresponding compression algorithm name in the plug-in JAR file are further loaded into the target cache. This allows the target cache to quickly and accurately locate the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name entered by the user on the configuration page.

[0101] The present application calls a preset plug-in loader; then scans the plug-in jar file in the preset plug-in directory based on the plug-in loader; then obtains the implementation class in the plug-in jar file and obtains the compression algorithm name corresponding to the implementation class in the plug-in jar file; and subsequently loads the implementation class in the plug-in jar file and the corresponding compression algorithm name into the target cache. The present application scans the plug-in jar file in the preset plug-in directory based on the use of the plug-in loader, then obtains the implementation class in the plug-in jar file and obtains the compression algorithm name corresponding to the implementation class in the plug-in jar file, and then loads the implementation class in the plug-in jar file and the corresponding compression algorithm name into the target cache, so that when a user-triggered data addition request for the target index library is subsequently received, the target cache can quickly and accurately find the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name, thereby improving the search efficiency and search accuracy of the implementation class of the specified compression algorithm that matches the user's needs.

[0102] In some optional implementations of this embodiment, before the step of scanning the plug-in jar file in the preset plug-in directory based on the plug-in loader, the electronic device may further perform the following steps:

[0103] Obtain a predefined compression interface; wherein the compression interface defines compression, decompression, and a compression algorithm name.

[0104] In this embodiment, the compression interface refers to the decompression interface named CompressionCodes, i.e., the compression interface mentioned above. This compression interface contains three methods: compress(byte[] input), decompress(byte[] compressed), and getAlgorithmName(). These methods are used to obtain the compression name, decompression name, and compression algorithm name, respectively.

[0105] Specifically, compress(byte[]input): receives a byte array as input and returns a compressed byte array. decompress(byte[]compressed): receives a compressed byte array as input and returns the decompressed original byte array. getAlgorithmName(): returns the name of the compression algorithm.

[0106] Acquire multiple preset compression algorithms, and create implementation classes corresponding to the compression interface for each compression algorithm.

[0107] In this embodiment, the compression algorithms mentioned above may include at least LZ4, Zstandard, Snappy, and XZ. By implementing the specific compression interface CompressionCodes based on the compression algorithm, and implementing the compression, decompression, and compression algorithm name of the interface based on the specific decompression algorithm, an implementation class corresponding to the compression interface is obtained. The implementation class implements the three methods corresponding to the compression interface.

[0108] A plug-in jar file corresponding to each compression algorithm is generated based on the implementation class.

[0109] In this embodiment, the three methods described above are implemented using the aforementioned implementation classes and packaged into independent plug-in JAR files. Specifically, a class is created for each compression algorithm, such as LZ4Compression and ZstandardCompression. These classes implement the CompressionCodes interface. Within each implementation class, the implementations of the compress, decompress, and getAlgorithmName methods are written. Furthermore, each implementation class and its dependencies are packaged into a plug-in JAR file using a Java packaging tool (such as the jar command).

[0110] Store the plug-in jar file in the plug-in directory.

[0111] In this embodiment, the plug-in jar file is stored in a plug-in directory so that the plug-in jar file can be quickly queried by subsequently accessing the plug-in directory.

[0112] By customizing the plug-in loading method, the plug-in jar file is placed in a specific directory. When the application starts, the plug-in jar file in the specific directory (plug-in) is scanned, and the implementation class is loaded using the reflection mechanism. The correspondence between the cache compression algorithm name and the plug-in implementation can be obtained by configuring the compression algorithm name.

[0113] The present application obtains a predefined compression interface; wherein the compression interface defines compression, decompression and the name of the compression algorithm; then obtains a plurality of preset compression algorithms, and creates an implementation class corresponding to the compression interface for each of the compression algorithms; then generates a plug-in jar file corresponding to each of the compression algorithms based on the implementation class; and subsequently stores the plug-in jar file in the plug-in directory. The present application obtains a plurality of preset compression algorithms, and creates an implementation class corresponding to the predefined compression interface for each of the compression algorithms, and then generates a plug-in jar file corresponding to each of the compression algorithms based on the implementation class and stores the plug-in jar file in the plug-in directory. This is advantageous in that the plug-in jar file can be quickly queried by accessing the plug-in directory, thereby improving the efficiency and intelligence of obtaining the implementation classes corresponding to the various compression algorithms contained in the plug-in jar file.

[0114] In some optional implementations, after the step of storing the plug-in jar file in the plug-in directory, the electronic device may further perform the following steps:

[0115] Determine whether a compression algorithm adding request triggered by a specified user is received; wherein the compression algorithm adding request carries a target compression algorithm name.

[0116] In this embodiment, the compression algorithm addition request is a user-triggered request for adding permission to a new target compression algorithm during the data addition process of the target index library. The target compression algorithm name refers to the algorithm name corresponding to the new target compression algorithm to be added.

[0117] If so, extract the target compression algorithm name from the compression algorithm adding request.

[0118] In this embodiment, the compression algorithm adding request may be parsed to extract the target compression algorithm name carried in the compression algorithm adding request.

[0119] A target compression algorithm corresponding to the target compression algorithm name is determined.

[0120] In this embodiment, the target compression algorithm name may be used to perform algorithm retrieval to determine the target compression algorithm corresponding to the target compression algorithm name.

[0121] The plug-in directory is updated based on the target compression algorithm.

[0122] In this embodiment, a target implementation class corresponding to the above-mentioned compression interface can be created for the target compression algorithm, and then the target implementation class and its dependencies can be packaged into a target plug-in jar file using Java packaging tools (such as the jar command), and the target plug-in jar file can be added to the above-mentioned plug-in directory to complete the update processing of the plug-in directory, so as to facilitate the addition of usage permissions for the new target compression algorithm during the data addition processing of the target index library.

[0123] The present application determines whether a compression algorithm adding request triggered by a specified user is received; wherein, the compression algorithm adding request carries the target compression algorithm name; if so, extracts the target compression algorithm name from the compression algorithm adding request; then determines the target compression algorithm corresponding to the target compression algorithm name; and subsequently updates the plug-in directory based on the target compression algorithm. After receiving a compression algorithm adding request triggered by a specified user, the present application extracts the target compression algorithm name from the compression algorithm adding request, determines the target compression algorithm corresponding to the target compression algorithm name, and then automatically updates the plug-in directory based on the target compression algorithm, thereby quickly and intelligently implementing the addition of new target compression algorithm usage permissions during the data addition processing of the target index library without modifying the existing code, effectively improving the intelligence of adding compression algorithms and increasing the scalability of the system.

[0124] In some optional implementations, step S205 includes the following steps:

[0125] The original data is obtained.

[0126] In this embodiment, the data adding request may be parsed to extract the original data from the data adding request.

[0127] Perform data cleaning on the original data to obtain corresponding specified data.

[0128] In this embodiment, the above-mentioned data cleaning includes removing irrelevant characters, spaces, line breaks, etc. in the original data, so as to ensure the neatness of the data and obtain the corresponding specified data.

[0129] The specified data is verified based on a preset verification strategy.

[0130] In this embodiment, the specific implementation process of the verification processing of the specified data based on the preset verification strategy will be further described in detail in subsequent specific embodiments of this application and will not be elaborated on here.

[0131] If the designated data passes the verification, the designated data is compressed based on the designated compression method to obtain the corresponding target data.

[0132] In this embodiment, by using the verified specified data as the input parameter of the above-mentioned specified compression method (compress method), the specified compression method will execute the compression logic of the above-mentioned specified compression algorithm to compress the specified data into a smaller byte array to obtain the corresponding target data.

[0133] The present application obtains the original data; then performs data cleaning on the original data to obtain the corresponding designated data; subsequently verifies the designated data based on a preset verification strategy; if the designated data passes the verification, the designated data is compressed based on the designated compression method to obtain the corresponding target data. After finding the implementation class of the designated compression algorithm corresponding to the name of the designated compression algorithm from the target cache and calling the designated compression method corresponding to the implementation class of the designated compression algorithm, the present application first intelligently obtains the original data and performs data cleaning on the original data to obtain the corresponding designated data, thereby ensuring the neatness of the designated data. The designated data is then verified based on the use of the verification strategy, and only when it is detected that the designated data passes the verification, the designated data will be compressed based on the designated compression method to obtain the corresponding target data, effectively improving the processing standardization and processing accuracy of the data compression processing.

[0134] In some optional implementations of this embodiment, the verification process on the specified data based on a preset verification strategy includes the following steps:

[0135] Performing specification verification on the specified data.

[0136] In this embodiment, the aforementioned standard verification refers to checking whether the data conforms to the expected format and range, such as the character encoding and length limit of text data. If the specified data conforms to the expected format and range, the specified data is determined to have passed the standard verification; otherwise, the specified data is determined to have failed the standard verification. Alternatively, if the specified data fails the standard verification, the specified data is directly determined to have failed the verification.

[0137] If the specified data passes the standard verification, sensitive verification is performed on the specified data.

[0138] In this embodiment, the specific implementation process of the sensitive verification of the designated data will be described in further detail in subsequent specific embodiments of the present application and will not be elaborated on here.

[0139] If the designated data passes the sensitive verification, it is determined that the designated data passes the verification; otherwise, it is determined that the designated data fails the verification.

[0140] In this embodiment, only when it is detected that the specified data has passed both the standard verification and the sensitive verification, will it be determined that the specified data has passed the verification; otherwise, it will be determined that the specified data has not passed the verification.

[0141] This application performs normative verification on the designated data; if the designated data passes the normative verification, the designated data is subjected to sensitive verification; if the designated data passes the sensitive verification, the designated data is determined to have passed the verification, otherwise the designated data is determined to have failed the verification. This application performs normative verification and sensitive verification on the designated data based on the use of verification strategies. Only when it is detected that the designated data has passed both normative verification and sensitive verification, the designated data will be subsequently determined to have passed the verification, which effectively improves the accuracy of the verification process for the designated data and helps to ensure that sensitive data is not involved in the data compression process, thereby reducing the risk of data leakage and ensuring the privacy and security of sensitive data.

[0142] In some optional implementations of this embodiment, performing sensitive verification on the designated data includes the following steps:

[0143] Get the preset sensitive identification rules.

[0144] In this embodiment, the above-mentioned sensitive identification rules are pre-built rules for sensitive identification of data. The procedural content of the sensitive identification rules may include: regular expression matching, dictionary lookup, and context analysis processing. Regular expression matching includes: using regular expressions to match specific data patterns, such as credit card numbers usually follow a specific numeric format. Dictionary lookup includes: checking whether the data contains predefined sensitive words or phrases. Context analysis includes: considering the contextual information of the data to determine whether it is sensitive. For example, a single phone number may not be sensitive, but if it appears together with personal information such as name and address, it may constitute sensitive data.

[0145] The designated data is matched item by item based on the sensitive identification rule to determine whether there is sensitive data matching the sensitive identification rule in the designated data.

[0146] In this embodiment, the designated data may be matched item by item according to the rule content of the sensitive identification rule, so as to detect whether there is sensitive data matching the sensitive identification rule in the designated data.

[0147] If there is no sensitive data matching the sensitive identification rule in the specified data, it is determined that the specified data has passed the sensitive verification; otherwise, it is determined that the specified data has not passed the sensitive verification.

[0148] In this embodiment, if it is detected that no sensitive data matching the sensitive identification rule exists in the specified data, the specified data is determined to have passed the sensitive verification. If it is detected that sensitive data matching the sensitive identification rule exists in the specified data, the specified data is determined to have failed the sensitive verification.

[0149] The present application obtains preset sensitive identification rules; then performs item-by-item matching processing on the designated data based on the sensitive identification rules, and determines whether there is sensitive data matching the sensitive identification rules in the designated data; if there is no sensitive data matching the sensitive identification rules in the designated data, then the designated data is determined to have passed sensitive verification, otherwise the designated data is determined to have failed sensitive verification. The present application obtains preset sensitive identification rules, and then performs item-by-item matching processing on the designated data based on the use of the sensitive identification rules, and obtains a judgment result on whether there is sensitive data matching the sensitive identification rules in the designated data, and then can quickly and accurately generate sensitive verification results corresponding to the designated data based on the obtained judgment results, effectively ensuring the accuracy of the obtained sensitive verification results.

[0150] In some optional implementations, the user information obtained is obtained with the user's consent and complies with relevant laws and policies.

[0151] In addition, any software tools or components not provided by our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0152] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0153] It should be emphasized that in order to further ensure the privacy and security of the above target data, the above target data can also be stored in a node of a blockchain.

[0154] The blockchain referred to in this application is a new application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0155] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0156] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0157] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0158] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0159] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in FIG, the present application provides an embodiment of a data processing device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0160] like Figure 3 As shown, the data processing device 300 of this embodiment includes: a first judgment module 301, a display module 302, a receiving module 303, a processing module 304, a compression module 305 and a first storage module 306.

[0161] The first judgment module 301 is used to judge whether a data addition request for a target index library triggered by a user is received; wherein the data addition request carries original data;

[0162] Display module 302, for displaying a preset configuration page if yes;

[0163] The receiving module 303 is configured to receive the specified compression algorithm name entered by the user in the configuration page;

[0164] The processing module 304 is configured to search the preset target cache for an implementation class of the specified compression algorithm corresponding to the specified compression algorithm name, and call the specified compression method corresponding to the implementation class of the specified compression algorithm;

[0165] The compression module 305 is configured to compress the original data based on the specified compression method to obtain corresponding target data;

[0166] The first storage module 306 is configured to store the target data in the target index library.

[0167] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data processing method in the aforementioned embodiment, and are not described in detail here.

[0168] In some optional implementations of this embodiment, the data processing device 300 further includes:

[0169] Calling module, used to call the preset plug-in loader;

[0170] A scanning module, configured to scan the plug-in jar files in a preset plug-in directory based on the plug-in loader;

[0171] A first acquisition module is used to obtain the implementation class in the plug-in jar file and obtain the compression algorithm name corresponding to the implementation class in the plug-in jar file;

[0172] The loading module is used to load the implementation class and the corresponding compression algorithm name in the plug-in jar file into the target cache.

[0173] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data processing method in the aforementioned embodiment, and are not described in detail here.

[0174] In some optional implementations of this embodiment, the data processing device 300 further includes:

[0175] A second acquisition module is used to acquire a predefined compression interface; wherein the compression interface defines compression, decompression and compression algorithm name;

[0176] A creation module is used to obtain a plurality of preset compression algorithms and create implementation classes corresponding to the compression interface for each compression algorithm;

[0177] A generation module, configured to generate a plug-in jar file corresponding to each compression algorithm based on the implementation class;

[0178] The second storage module is used to store the plug-in jar file in the plug-in directory.

[0179] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data processing method in the aforementioned embodiment, and are not described in detail here.

[0180] In some optional implementations of this embodiment, the data processing device 300 further includes:

[0181] The second judgment module is used to judge whether a compression algorithm adding request triggered by a specified user is received; wherein the compression algorithm adding request carries the name of the target compression algorithm;

[0182] An extraction module, configured to extract the target compression algorithm name from the compression algorithm adding request;

[0183] A determination module, configured to determine a target compression algorithm corresponding to the target compression algorithm name;

[0184] An update module is used to update the plug-in directory based on the target compression algorithm.

[0185] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data processing method in the aforementioned embodiment, and are not described in detail here.

[0186] In some optional implementations of this embodiment, the compression module 305 includes:

[0187] An acquisition submodule, used for acquiring the original data;

[0188] A cleaning submodule, configured to perform data cleaning on the original data to obtain corresponding specified data;

[0189] A verification submodule, configured to verify the specified data based on a preset verification strategy;

[0190] The compression submodule is used to compress the designated data based on the designated compression method to obtain the corresponding target data if the designated data passes the verification.

[0191] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data processing method in the aforementioned embodiment, and are not described in detail here.

[0192] In some optional implementations of this embodiment, the verification submodule includes:

[0193] A first verification unit, configured to perform standard verification on the specified data;

[0194] a second verification unit, configured to perform sensitive verification on the designated data if the designated data passes the standard verification;

[0195] The determination unit is configured to determine that the specified data has passed the verification if the specified data has passed the sensitive verification, and otherwise determine that the specified data has failed the verification.

[0196] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data processing method in the aforementioned embodiment, and are not described in detail here.

[0197] In some optional implementations of this embodiment, the second verification unit includes:

[0198] An acquisition subunit, used to acquire preset sensitive identification rules;

[0199] a judgment subunit, configured to perform item-by-item matching processing on the designated data based on the sensitive identification rule, and judge whether there is any sensitive data matching the sensitive identification rule in the designated data;

[0200] The determination subunit is configured to determine that the specified data has passed the sensitive verification if no sensitive data matching the sensitive identification rule exists in the specified data, and otherwise determine that the specified data has failed the sensitive verification.

[0201] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data processing method in the aforementioned embodiment, and are not described in detail here.

[0202] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0203] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0204] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0205] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of data processing methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0206] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for executing the data processing method.

[0207] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0208] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0209] In an embodiment of the present application, the present application first determines whether a data addition request for a target index library triggered by a user is received; wherein the data addition request carries the original data; if so, a preset configuration page is displayed; then the specified compression algorithm name entered by the user in the configuration page is received; thereafter, the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name is found from the preset target cache, and the specified compression method corresponding to the implementation class of the specified compression algorithm is called; subsequently, the original data is compressed based on the specified compression method to obtain the corresponding target data; finally, the target data is stored in the target index library. When the present application receives a data addition request for a target index library triggered by a user, it displays a preset configuration page and receives the specified compression algorithm name entered by the user in the configuration page, then searches the preset target cache for the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name, and calls the specified compression method corresponding to the implementation class of the specified compression algorithm, and then compresses the original data based on the specified compression method to obtain the corresponding target data, and stores the target data in the target index library. This enables the user to select the corresponding specified compression algorithm according to specific needs and application scenarios to compress the original data that needs to be added to the target index library, effectively improving the flexibility and intelligence of data compression processing, and is conducive to improving the processing performance of the system.

[0210] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data processing method as described above.

[0211] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0212] In an embodiment of the present application, the present application first determines whether a data addition request for a target index library triggered by a user is received; wherein the data addition request carries the original data; if so, a preset configuration page is displayed; then the specified compression algorithm name entered by the user in the configuration page is received; thereafter, the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name is found from the preset target cache, and the specified compression method corresponding to the implementation class of the specified compression algorithm is called; subsequently, the original data is compressed based on the specified compression method to obtain the corresponding target data; finally, the target data is stored in the target index library. When the present application receives a data addition request for a target index library triggered by a user, it displays a preset configuration page and receives the specified compression algorithm name entered by the user in the configuration page, then searches the preset target cache for the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name, and calls the specified compression method corresponding to the implementation class of the specified compression algorithm, and then compresses the original data based on the specified compression method to obtain the corresponding target data, and stores the target data in the target index library. This enables the user to select the corresponding specified compression algorithm according to specific needs and application scenarios to compress the original data that needs to be added to the target index library, effectively improving the flexibility and intelligence of data compression processing, and is conducive to improving the processing performance of the system.

[0213] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0214] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A data processing method, characterized in that: The steps include: Determine whether a data addition request for a target index library triggered by a user is received; wherein the data addition request carries original data; If yes, display the preset configuration page; Receiving a specified compression algorithm name entered by the user in the configuration page; Finding an implementation class of a specified compression algorithm corresponding to the specified compression algorithm name from a preset target cache, and calling a specified compression method corresponding to the implementation class of the specified compression algorithm; Compressing the original data based on the specified compression method to obtain corresponding target data; The target data is stored in the target index library.

2. The data processing method according to claim 1, characterized in that: Before the step of searching the preset target cache for the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name, the method further includes: Call the preset plugin loader; Scanning the plug-in jar file in the preset plug-in directory based on the plug-in loader; Obtaining the implementation class in the plug-in jar file, and obtaining the compression algorithm name corresponding to the implementation class in the plug-in jar file; The implementation class and the corresponding compression algorithm name in the plug-in jar file are loaded into the target cache.

3. The data processing method according to claim 2, characterized in that: Before the step of scanning the plug-in jar file in the preset plug-in directory based on the plug-in loader, the method further includes: Obtain a predefined compression interface; wherein the compression interface defines compression, decompression and a compression algorithm name; Acquire multiple preset compression algorithms, and create implementation classes corresponding to the compression interface for each compression algorithm; Generate a plug-in jar file corresponding to each compression algorithm based on the implementation class; The plug-in jar file is stored in the plug-in directory.

4. The data processing method according to claim 3, characterized in that: After the step of storing the plug-in jar file in the plug-in directory, the method further includes: Determine whether a compression algorithm adding request triggered by a specified user is received; wherein the compression algorithm adding request carries a target compression algorithm name; If so, extracting the target compression algorithm name from the compression algorithm adding request; Determine a target compression algorithm corresponding to the target compression algorithm name; The plug-in directory is updated based on the target compression algorithm.

5. The data processing method according to claim 1, characterized in that: The step of compressing the original data based on the specified compression method to obtain corresponding target data specifically includes: Acquiring the original data; Performing data cleaning processing on the original data to obtain corresponding designated data; Performing verification processing on the specified data based on a preset verification strategy; If the designated data passes the verification, the designated data is compressed based on the designated compression method to obtain the corresponding target data.

6. The data processing method according to claim 5, characterized in that: The step of performing verification processing on the specified data based on the preset verification strategy specifically includes: Performing standardized verification on the specified data; If the specified data passes the specification verification, sensitive verification is performed on the specified data; If the designated data passes the sensitive verification, it is determined that the designated data passes the verification; otherwise, it is determined that the designated data fails the verification.

7. The data processing method according to claim 6, characterized in that: The step of performing sensitive verification on the designated data specifically includes: Obtain preset sensitive identification rules; Performing item-by-item matching processing on the designated data based on the sensitive identification rule, and determining whether there is any sensitive data matching the sensitive identification rule in the designated data; If there is no sensitive data matching the sensitive identification rule in the specified data, it is determined that the specified data has passed the sensitive verification; otherwise, it is determined that the specified data has not passed the sensitive verification.

8. A data processing device, characterized in that: include: A first judgment module is used to judge whether a data addition request for a target index library triggered by a user is received; wherein the data addition request carries original data; Display module, used to display the preset configuration page if yes; A receiving module, used for receiving the specified compression algorithm name entered by the user in the configuration page; A processing module, used for searching the implementation class of the specified compression algorithm corresponding to the specified compression algorithm name from a preset target cache, and calling the specified compression method corresponding to the implementation class of the specified compression algorithm; A compression module, used for compressing the original data based on the specified compression method to obtain corresponding target data; The first storage module is used to store the target data in the target index library.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data processing method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images