Concurrent safe self-adaptive data exporting method and device
Through the combination of policy mode and thread-local storage, a concurrent and secure adaptive data export method is realized, which solves the efficiency and performance problems of traditional data export methods, supports multiple file formats and stream writing, and improves through style parsers The readability and visual effects of data files.
Patent Information
- Application Number
- CN202410842455.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-27
- Publication Date
- 2024-11-19
AI Technical Summary
Traditional data export methods have low development efficiency, single functions, performance bottlenecks and lack of flexibility, making it difficult to meet the high performance and scalability requirements of modern software development.
Adopting policy pattern design, through unified export policy abstract classes and thread-local storage, a concurrent and secure adaptive data export method is realized, supporting the export of multiple file formats, and introducing a style parser to provide dynamic style support.
Improve development efficiency and scalability, supports streaming writing and multiple file formats, reduces memory overhead, improves write performance, and simplifies style settings through style parsers, improving readability and visualization of data files Showcase in a variety of ways.
Smart Images

Figure CN118981333A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a concurrent and secure adaptive data export method and device. Background Art
[0002] In the field of data processing, it is often necessary to export data into files of specified formats, such as Excel tables, Word documents, PDF files, CSV files, and HTML web pages. The traditional data export method usually implements the export logic for each file format separately, which has the following main problems:
[0003] 1. Low development efficiency: The codes between export modules of different formats are highly coupled, and there is a lack of effective decoupling design, resulting in poor maintainability and scalability. The code reusability of each independent export module is low. Even if different files of the same file format are exported, it is difficult to reuse the previous code.
[0004] 2. Single function: Traditional solutions can usually only complete the most basic file generation functions and lack support for advanced functions such as styles and data partitions.
[0005] 3. Performance bottleneck: Most existing solutions read all data at once and then write them to files in batches. When processing large amounts of data or streaming data, full reading will result in higher memory overhead and affect write performance.
[0006] 4. Lack of flexibility: When dealing with new file format export requirements, new export modules need to be redeveloped, and existing codes cannot be effectively reused, resulting in high development costs.
[0007] In general, traditional file export implementation solutions have been unable to meet the flexibility, scalability, high performance and maintainability requirements of modern software development. Summary of the invention
[0008] In order to solve the above problems existing in the prior art, the present invention provides a concurrent and secure adaptive data export method and device, which can efficiently and dynamically support new formats, and at the same time have enhanced functions such as streaming writing and style support to meet the high performance requirements of modern data processing.
[0009] To achieve the above object, the present invention adopts the following technical solutions:
[0010] In one embodiment of the present invention, a concurrent and secure adaptive data export method is proposed, the method comprising:
[0011] Define a unified export strategy abstract class to provide a unified encapsulated export data interface to the outside world; use thread local storage in this class to save the writer, shard and file name, and shard name status information of each data writing thread; define the abstract method of file export strategy in this class for the export strategy abstract subclass to implement specific file suffix name, writer creation, shard creation, data writing, and writer closing logic for each file format;
[0012] For each file format, the corresponding export strategy abstract subclass inherits from the unified export strategy abstract class and implements the abstract method of the file export strategy defined in the class;
[0013] The style parser calculates the specific style attributes that should be applied to each cell based on predefined style rules, providing style support for the data writing process.
[0014] Furthermore, for each file format, an instance of the corresponding export strategy abstract class is constructed, and the instance of the corresponding export strategy abstract class is obtained according to the specified file format through the strategy factory.
[0015] Furthermore, based on the obtained instance of the export strategy abstract class, an instance of the writer is first created, and then instances of shards can be repeatedly created in the same writer; if the passed-in shard name is the same as the existing shard name, the instance of the original shard is returned; otherwise, a new shard instance is created; thereafter, the data write operation can be called multiple times, each time a batch of data sets is passed in, and the corresponding shards write the data into them.
[0016] Furthermore, when calculating the cell style, the style parser dynamically builds the conditional judgment logic according to the following steps:
[0017] The style parser parses the style rule parameters in the JSON array format and constructs a condition tree data structure. Each style rule parameter in the JSON array format corresponds to a node in the condition tree. The node contains an operator, a value, and a linkOperator to the next node.
[0018] The style parser constructs a conditional judgment function expression containing variable placeholders based on the data structure of the conditional tree. The conditional judgment function expression receives the value of the cell as input and returns a Boolean value to indicate whether the condition is met. The construction of the conditional judgment function expression uses a recursive method to deeply traverse the conditional tree. At the leaf node, it determines whether the cell value meets the operator and value conditions of the node. At the non-leaf node, the results of the leaf node conditions are merged according to the linkOperator operator.
[0019] When the style parser calculates the style of a cell, it traverses all style rules and executes the conditional judgment function expression in the condition attribute. If it returns True, the style of the style rule is applied to the cell.
[0020] In one embodiment of the present invention, a concurrent and secure adaptive data export device is also proposed, the device comprising:
[0021] The strategy factory module is used to create and return an instance of the abstract class of a specific file format export strategy;
[0022] A unified abstract strategy module is used to define a unified export strategy abstract class, which is used to provide a unified encapsulated export data interface to the outside world. In this class, thread local storage is used to save the writer, shard and file name, and shard name status information of each data writing thread. In this class, the abstract method of file export strategy is defined for the export strategy abstract subclass to implement specific file suffix, writer creation, shard creation, data writing and writer closing logic for each file format. For each file format, the corresponding export strategy abstract subclass inherits from the unified export strategy abstract class and implements the abstract method of file export strategy defined in this class. The style parser calculates the specific style attributes that should be applied to each cell based on predefined style rules to provide style support for the data writing process.
[0023] Furthermore, for each file format, an instance of the corresponding export strategy abstract class is constructed, and the instance of the corresponding export strategy abstract class is obtained according to the specified file format through the strategy factory.
[0024] Furthermore, based on the obtained instance of the export strategy abstract class, an instance of the writer is first created, and then instances of shards can be repeatedly created in the same writer; if the passed-in shard name is the same as the existing shard name, the instance of the original shard is returned; otherwise, a new shard instance is created; thereafter, the data write operation can be called multiple times, each time a batch of data sets is passed in, and the corresponding shards write the data into them.
[0025] Furthermore, when calculating the cell style, the style parser dynamically builds the conditional judgment logic according to the following steps:
[0026] The style parser parses the style rule parameters in the JSON array format and constructs a condition tree data structure. Each style rule parameter in the JSON array format corresponds to a node in the condition tree. The node contains an operator, a value, and a linkOperator to the next node.
[0027] The style parser constructs a conditional judgment function expression containing variable placeholders based on the data structure of the conditional tree. The conditional judgment function expression receives the value of the cell as input and returns a Boolean value to indicate whether the condition is met. The construction of the conditional judgment function expression uses a recursive method to deeply traverse the conditional tree. At the leaf node, it determines whether the cell value meets the operator and value conditions of the node. At the non-leaf node, the results of the leaf node conditions are merged according to the linkOperator operator.
[0028] When the style parser calculates the style of a cell, it traverses all style rules and executes the conditional judgment function expression in the condition attribute. If it returns True, the style of the style rule is applied to the cell.
[0029] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned concurrent and secure adaptive data export method is implemented.
[0030] In one embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program for executing a concurrent and secure adaptive data export method.
[0031] Beneficial effects:
[0032] 1. The present invention adopts the strategy pattern design, encapsulating the export logic of different file formats in independent strategy objects to achieve functional decoupling. To add a new file format, you only need to implement a new strategy object and register it to the strategy factory class, without modifying the existing code, which has strong scalability.
[0033] 2. The present invention is based on a unified writer and sharding framework. Developers only need to focus on implementing the logic of creating files and writing data in a specific file format. Other common functions can be directly reused to reduce duplication of work and improve development efficiency.
[0034] 3. The present invention supports cyclic and multiple writing in the same connection to avoid performance loss caused by connection interruption; adopts batch cyclic writing mode, which eliminates the need to load all data at once and reduces memory overhead; the batch writing has a small granularity and efficient IO scheduling, which is suitable for large data volumes and streaming data scenarios, and significantly improves the overall writing performance.
[0035] 4. In the present invention, users only need to call the policy factory to obtain a writer instance of any file format without paying attention to the complex internal details; sharding provides a flexible way to write shard data, allowing users to freely control shard creation and data writing.
[0036] 5. The present invention introduces a style parser that can automatically calculate the style attributes of each cell according to predefined rules, greatly reducing the workload of manual settings, ensuring style consistency, and improving the readability and visual presentation of data files; style rules can be customized to achieve data-driven dynamic style settings, facilitate highlighting of data such as outliers, and improve data analysis and usage value. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a flowchart of a concurrent and secure adaptive data export method according to an embodiment of the present invention;
[0038] Figure 2 It is a structural schematic diagram of a concurrent and secure adaptive data export device of the present invention;
[0039] Figure 3 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION
[0040] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.
[0041] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a device, apparatus, equipment, method or computer program product. Therefore, the present disclosure may be implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.
[0042] According to an embodiment of the present invention, a concurrent and secure adaptive data export method is proposed, which adopts the strategy pattern to define an abstract class for exporting data strategies. The functions provided by this class include:
[0043] (1) It provides a general data export interface, including methods such as buildWriter (build writer), buildSplitter (build splitter), writeData (write data), and finish (finish writing), shielding the underlying implementation details of different file formats.
[0044] (2) An abstract framework for file export strategy is defined, including abstract methods such as fileSuffix() (used to obtain the file suffix), createWriter() (used to create a writer instance), createSplitter() (used to create a splitter instance), and closeWriter() (used to close a writer). Specific export strategy abstract subclasses implement specific logic for each file format.
[0045] (3) Use ThreadLocal (thread local storage) to save the status information of each data writing thread, such as the writer, splitter, and file name, to ensure the isolation and thread safety of the status during multi-threaded concurrent export, and avoid data competition and synchronization overhead between threads.
[0046] (4) In buildWriter, if the file name changes, the old writer is automatically closed and a new writer is created, supporting enhanced features such as file switching and streaming writing.
[0047] (5) In buildSplitter, if the shard name changes, the shard will be automatically recreated, supporting the distinction of multiple logical partitions within the same file, improving flexibility.
[0048] The present invention uses thread local storage (ThreadLocal) in a unified export strategy abstract class (AbstractExportStrategy) to save state information such as writers, fragments and file names to ensure that each thread has its own state. In addition, abstract methods fileSuffix, createWriter, createSplitter and closeWriter are provided for the export strategy abstract subclass to implement specific file suffix names, writer creation, fragment creation and writer closing logic. Through the use of ThreadLocal, AbstractExportStrategy realizes the binding of state and thread, and each thread exclusively enjoys its own export contexts such as Writer and Splitter without interfering with each other. Even in a high-concurrency multi-threaded environment, accurate export of data can be ensured, greatly improving scalability and stability.
[0049] For each file format, the corresponding export strategy abstract subclass must inherit from the unified export strategy abstract class (AbstractExportStrategy) and implement a series of abstract methods defined in this class. For each file format, an instance of the corresponding export strategy abstract class is constructed. Each instance encapsulates the export logic specific to the file format, implements the interface and abstract methods defined by the unified export strategy abstract class, and obtains the corresponding export strategy abstract class instance through the strategy factory according to the specified export file format. The strategy factory (ExportStrategyFactory) is responsible for creating and returning an instance of the export strategy abstract class for a specific file format. It dynamically determines and instantiates the corresponding export strategy abstract class based on the passed-in file format parameters, thereby realizing the strategy pattern and improving flexibility and scalability.
[0050] Based on the instance of the export strategy abstract class obtained, a writer instance is first created, and then splitter instances can be repeatedly created in the same writer. If the name of the passed-in splitter is the same as the name of the existing splitter, the original splitter instance is returned; if the name is different, a new splitter instance is created. After that, the write operation can be called multiple times, each time a batch of data sets are passed in, and the corresponding splitter writes the data into it. After all the data is written, the write operation is called to complete, and the writer will perform a series of operations such as closing the file stream, closing itself, and clearing the thread local storage.
[0051] When creating a shard instance, you need to specify the shard name, column header information, and style rule information as parameter input. The meaning of shards varies in different file formats, such as Sheet pages in xlsx files, tables in doc, pdf, and csv, and tab pages in html.
[0052] In addition, a style parser (StyleParser) is also provided. The style parser calculates and determines the style attributes that should be applied to each cell based on the input style rule parameter information, including font color, font type, font size, background color, border color, etc. When performing a data write operation, the style parser will be called to obtain the style data of the corresponding cell and write it into the cell style. A cell is a basic element for storing data or data formats at the intersection of rows and columns in a spreadsheet (such as Excel). During the data export process, each data item is usually mapped to a cell, and style rules can be applied to define its display effect, such as font, color, etc.
[0053] For the newly added file formats, you only need to implement a new policy object and register it to the policy factory without modifying other parts of the framework. It also supports enhanced features such as cyclic writing and persistent connection, which greatly improves data archiving performance, thereby greatly improving scalability and stability.
[0054] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.
[0055] The concurrent and secure adaptive data export method proposed by the present invention includes:
[0056] 1. Define the export strategy abstract class
[0057] First, a unified export strategy abstract class is defined: the export strategy abstract class AbstractExportStrategy, which is used to provide a unified encapsulated export data interface to the outside world. AbstractExportStrategy is the core abstract class of the present invention, which defines a unified interface and extension for exporting data. This class provides the following open functions:
[0058] (1) buildWriter(String fileName): Builds a writer, creates a file based on the given file name, and initializes or recreates a writer. If the file name changes, the old writer is closed and a new one is created.
[0059] (2)buildSplitter(StringsplitterName, List <List <string>>header,List <stylerule>rules): Build a sheet, create a new sheet based on the given table name, table header, and style rules. If the table name changes, the sheet will be recreated.
[0060] (3)writeData(List <List <object>>data): Writes data to the shard. This method is implemented by the export strategy abstract subclass.
[0061] (4) finish(): Completes the write operation, closes the writer and clears the thread local storage.
[0062] In addition, internal abstract methods fileSuffix, createWriter, createSplitter, and closeWriter are provided. These methods must be implemented by their subclasses, including specific file suffix names, writer creation, slicing creation, and writer closing logic. These methods are protected and cannot be called externally, which can ensure the uniformity of the interface provided to the outside world while providing functional diversity.
[0063] The present invention uses thread local storage (ThreadLocal) in this class to save the state information such as writer, slice and file name, slice name, etc., to ensure that each thread has its own state. Thread local storage (ThreadLocal) is a thread binding mechanism provided in Java, which creates a separate variable copy for each thread, and each thread can only modify its own copy, thereby avoiding data sharing and competition problems between threads, greatly reducing synchronization overhead, and improving the performance of concurrent processing. Through the use of ThreadLocal, AbstractExportStrategy realizes the binding of state and thread, and each thread has its own export context such as Writer, Splitter, etc., without interfering with each other. Even in a multi-threaded environment, the data export tasks processed by each thread are independent, ensuring the isolation and thread safety of the data.
[0064] 2. Implement the export strategy abstract subclass
[0065] At present, the present invention provides subclass implementation types such as xlsx, doc, pdf, csv, html, md, etc. For each specific file format, it is necessary to inherit the AbstractExportStrategy abstract class and implement some of its abstract methods to create a writer and slicing object in a specific format and complete the actual data writing logic.
[0066] (1) The fileSuffix() method needs to return the file suffix corresponding to the file format, such as .xlsx, .doc, .pdf, etc., which is used to determine the corresponding export strategy based on the export target.
[0067] (2) The createWriter(String fileName) method needs to create a writer object for the file format as the underlying carrier for data writing. Different formats may require the use of different third-party libraries, such as ApachePOI for xlsx and iText for pdf, as shown in Table 1 below.
[0068] Table 1
[0069]
[0070] (3)createSplitter(Stringname, List <List <string>>header,List <stylerule>rules) method needs to create a Splitter object for the file format as a carrier of logical data partitions. Splits have different meanings in different formats, such as xlsx for Sheet pages, doc for tables, html for tab pages, etc. When creating, you need to pass in metadata such as splitter name, header, and style rules.
[0071] (4)writeData(List <List <object>>data) method is the core data writing logic, which needs to traverse the incoming data set and translate each row of data into content that can be recognized by the file format. For example, in xlsx, to create rows and cells and write data, the style parser StylePaser will be called to generate the style before writing data into the cell. The style parser will calculate and apply the cell style according to StyleRule, including font, color, background color, etc. The logic of data writing is highly coupled to the specific file format.
[0072] (5) The closeWriter(Writerwriter) method needs to implement the logic of closing the format writer, completing data writing and releasing related resources, such as closing the file stream, refreshing the buffer, etc.
[0073] Through this design, the export strategies of various file formats are completely decoupled. You only need to implement the specific logic of the above core methods. The rest of the process is automatically handled by the abstract class of the framework, which greatly improves flexibility and scalability. If you need to add export support for a new file format, you only need to create a corresponding export strategy implementation subclass, implement the relevant methods and register it to the strategy factory class, without changing other parts of the framework.
[0074] 3. Style parser implementation
[0075] StyleParser is a style parser in the present invention, and its function is to calculate the specific style attributes that should be applied to each cell according to the predefined StyleRule style rules, and provide style support for the data writing process.
[0076] StyleRule defines a style rule, which usually contains the following parts:
[0077] (1) Column Name: Specifies the column to which the style rule applies.
[0078] (2) Condition: A function condition that determines whether the style rule is applied based on the value of the cell.
[0079] (3) Style: If the conditions are met, the style object is applied to the cells that meet the conditions.
[0080] Style includes a series of style attributes such as cell font, font size, color, background color and border.
[0081] The condition includes:
[0082] 1) linkOperator: When there are multiple conditions, it is used to connect the logical operators of these conditions, such as AND, OR, etc.
[0083] 2) operator: specifies the operator to judge the field value, such as =, !=, >, <, LIKE, in, etc.
[0084] 3)value: specifies the value to be compared with the field value.
[0085] 4)children: sub-conditions, with the same properties as above.
[0086] For example:[
[0087] {"linkOperator": "AND", "operator": "<", "value": "100"},
[0088] {"linkOperator":"AND","operator":">=","value":"60"} ]
[0090] When calculating the cell style, the style parser StyleParser needs to dynamically build the conditional judgment logic based on this structure. The steps are as follows:
[0091] (1) Parse the JSON array and build a conditional tree. The style parser needs to parse the JSON array into a conditional tree data structure. The JSON array is the style rule parameter passed in by the user. There may be multiple style rules, so the JSON array format is used. Each JSON object corresponds to a node in the tree. The node contains an operator, a value, and a linkOperator to the next node.
[0092] (2) Construct a conditional judgment function expression. CCTV parser needs to construct a conditional judgment function expression containing variable placeholders based on this condition tree. This conditional judgment function expression receives the value of the cell as input and returns a Boolean value to indicate whether the condition is met. The construction of the conditional judgment function expression uses a recursive method to deeply traverse the condition tree. At the leaf node, determine whether the value of the cell meets the operator and value conditions of the node. At non-leaf nodes, according to the linkOperator operator, merge the results of the leaf node conditions (AND or OR). The above final expression is, for example: ${value}<100and${value}>=60.
[0093] (3) Conditional judgment is performed when calculating the style. When StyleParser calculates the style of a cell, it traverses all StyleRules and executes the judgment function in the condition attribute. If it returns True, the style of the rule is applied to the cell.
[0094] Through this design, the style parser can support very flexible and complex conditional logic, and the definition and parsing of the condition are separated, which improves scalability. If you need to support new conditional operators or logic, you only need to modify the logic of conditional parsing and building judgment functions without changing the core process of style application.
[0095] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0096] In order to explain the above-mentioned concurrent and safe adaptive data export method more clearly, a specific embodiment is used for illustration below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.
[0097] Example:
[0098] Here is a typical implementation example using export xlsx, such as Figure 1 As shown:
[0099] 1. Get the export strategy instance through ExportStrategyFactory (strategy factory) and pass in the parameter xlsx format.
[0100] AbstractExportStrategystrategy=ExportStrategyFactory.ge tStrategy("xlsx");
[0101] 2. Call buildWriter to create a writer and create the file report.xlsx.
[0102] strategy.buildWriter("report.xlsx");
[0103] 3. Build the data set to be exported and simulate querying data from the database.
[0104] List <List <object>>dataRows=getDatabaseData();
[0105] 4. Prepare metadata such as table headers and style rules.
[0106] / / Header
[0107] List <List <string>>header = Arrays.asList(Arrays.asList("Device Name", "IPv4 Inbound Traffic", "IPv4 Outbound Traffic"));
[0108] / / Define style rules
[0109] StyleRulerule=newStyleRule("device name", "[{"linkOperator": "AND", "operator": "like", "value": "huawei"}]", "{"bgColor": "#FFE500", "fontColor": "#000000"}");
[0110] List <stylerule>rules=Arrays.asList(rule);
[0111] 5. Call buildSplitter to build the shards, passing in the name, header, and style rules.
[0112] strategy.buildSplitter("device traffic", header, rules);
[0113] 6. Call writeData in a loop to write data in batches.
[0114] for(List <List <object>>batchData:splitBatches(dataRows)){
[0115] strategy.writeData(batchData);
[0116] }
[0117] 7. After all data is written, call finish to complete the operation.
[0118] strategy.finish();
[0119] Through the above process, the data is exported to the Excel file. The output file report.xlsx contains a Sheet page named "Device Traffic". The header contains three columns: "Device Name", "IPv4 Inbound Traffic", and "IPv4 Outbound Traffic". According to the style rules, the cell text containing the value "huawei" in the device name column is black and the background is yellow.
[0120] Based on the same inventive concept, the present invention also proposes a concurrent and secure adaptive data export device. The implementation of the device can refer to the implementation of the above method, and the repeated parts will not be repeated. The term "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
[0121] Figure 2 Schematic diagram of the structure of the concurrent and secure adaptive data export device of the present invention. Figure 2 As shown, the device comprises:
[0122] The strategy factory module 101 is used to create and return an instance of a specific file format export strategy abstract class;
[0123] The unified abstract strategy module 102 is used to define a unified export strategy abstract class for providing a unified encapsulated export data interface to the outside world; thread local storage is used in this class to save the writer, shard and file name, and status information of the shard name of each data writing thread; the abstract method of the file export strategy is defined in this class for the export strategy abstract subclass to implement specific file suffix name, writer creation, shard creation, data writing and writer closing logic for each file format; for each file format, its corresponding export strategy abstract subclass inherits from the unified export strategy abstract class and implements the abstract method of the file export strategy defined in this class; the style parser calculates the specific style attributes that should be applied to each cell based on predefined style rules to provide style support for the data writing process.
[0124] For each file format, construct an instance of the corresponding export strategy abstract class, and obtain the corresponding instance of the export strategy abstract class according to the specified file format through the strategy factory.
[0125] Based on the obtained instance of the export strategy abstract class, first create an instance of the writer, and then repeatedly create instances of shards in the same writer; if the passed in shard name is the same as the existing shard name, return the instance of the original shard; otherwise, create a new shard instance; after that, you can call the data write operation multiple times, each time passing in a batch of data sets, and the corresponding shards will write the data into them.
[0126] When calculating the cell style, the style parser dynamically builds the conditional judgment logic according to the following steps:
[0127] The style parser parses the style rule parameters in the JSON array format and constructs a condition tree data structure. Each style rule parameter in the JSON array format corresponds to a node in the condition tree. The node contains an operator, a value, and a linkOperator to the next node.
[0128] The style parser constructs a conditional judgment function expression containing variable placeholders based on the data structure of the conditional tree. The conditional judgment function expression receives the value of the cell as input and returns a Boolean value to indicate whether the condition is met. The construction of the conditional judgment function expression uses a recursive method to deeply traverse the conditional tree. At the leaf node, it determines whether the cell value meets the operator and value conditions of the node. At the non-leaf node, the results of the leaf node conditions are merged according to the linkOperator operator.
[0129] When the style parser calculates the style of a cell, it traverses all style rules and executes the conditional judgment function expression in the condition attribute. If it returns True, the style of the style rule is applied to the cell.
[0130] It should be noted that although several modules of the concurrent and secure adaptive data export device are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be concretized.
[0131] Based on the above invention concept, Figure 3 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220, and when the processor 220 executes the computer program 230, the aforementioned concurrent and secure adaptive data export method is implemented.
[0132] Based on the aforementioned inventive concept, the present invention further proposes a computer-readable storage medium, which stores a computer program for executing the aforementioned concurrent and secure adaptive data export method.
[0133] The concurrent and secure adaptive data export method and device proposed in the present invention is a general and scalable multi-format file data export solution based on the policy mode. It not only solves the pain points of scalability and development efficiency of traditional practices, but also introduces enhanced mechanisms such as cyclic writing and batch export, so that it can efficiently cope with the challenges of modern big data, streaming data and other scenarios, and its performance is significantly better than that of traditional solutions. At the same time, the usage method is also simpler and more flexible, with strong practical value.
[0134] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.
[0135] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.< / object> < / stylerule> < / string> < / object> < / object> < / stylerule> < / string> < / object> < / stylerule> < / string>
Claims
1. A concurrent and secure adaptive data export method, characterized in that: The method includes: Define a unified export strategy abstract class to provide a unified encapsulated export data interface to the outside world; use thread local storage in this class to save the writer, shard and file name, and shard name status information of each data writing thread; define the abstract method of file export strategy in this class for the export strategy abstract subclass to implement specific file suffix name, writer creation, shard creation, data writing, and writer closing logic for each file format; For each file format, the corresponding export strategy abstract subclass inherits from the unified export strategy abstract class and implements the abstract method of the file export strategy defined in the class; The style parser calculates the specific style attributes that should be applied to each cell based on predefined style rules, providing style support for the data writing process.
2. The concurrent and secure adaptive data export method according to claim 1, characterized in that: For each file format, construct an instance of the corresponding export strategy abstract class, and obtain the corresponding instance of the export strategy abstract class according to the specified file format through the strategy factory.
3. The concurrent and secure adaptive data export method according to claim 2, characterized in that: Based on the obtained instance of the export strategy abstract class, first create an instance of the writer, and then repeatedly create instances of shards in the same writer; If the passed shard name is the same as the existing shard name, the instance of the original shard is returned; otherwise, a new shard instance is created; after that, the data write operation can be called multiple times, each time a batch of data sets are passed in, and the corresponding shard writes the data into it.
4. The concurrent and secure adaptive data export method according to claim 1, characterized in that: When calculating the cell style, the style parser dynamically builds the conditional judgment logic according to the following steps: The style parser parses the style rule parameters in the JSON array format and constructs a condition tree data structure. Each style rule parameter in the JSON array format corresponds to a node in the condition tree. The node contains an operator, a value, and a linkOperator to the next node. The style parser constructs a conditional judgment function expression containing variable placeholders according to the data structure of the conditional tree; the conditional judgment function expression receives the value of the cell as input and returns a Boolean value indicating whether the condition is met; the construction of the conditional judgment function expression uses a recursive method to deeply traverse the conditional tree; At the leaf node, determine whether the cell value satisfies the operator and value conditions of the node; at the non-leaf node, merge the results of the leaf node conditions according to the linkOperator operator; When the style parser calculates the style of a cell, it traverses all style rules and executes the conditional judgment function expression in the condition attribute. If it returns True, the style of the style rule is applied to the cell.
5. A concurrent and secure adaptive data export device, characterized in that: The device includes: The strategy factory module is used to create and return an instance of the abstract class of a specific file format export strategy; A unified abstract strategy module is used to define a unified export strategy abstract class, which is used to provide a unified encapsulated export data interface to the outside world. In this class, thread local storage is used to save the writer, shard and file name, and shard name status information of each data writing thread. In this class, the abstract method of file export strategy is defined for the export strategy abstract subclass to implement specific file suffix, writer creation, shard creation, data writing and writer closing logic for each file format. For each file format, the corresponding export strategy abstract subclass inherits from the unified export strategy abstract class and implements the abstract method of file export strategy defined in this class. The style parser calculates the specific style attributes that should be applied to each cell based on predefined style rules to provide style support for the data writing process.
6. The concurrent and secure adaptive data export device according to claim 5, characterized in that: For each file format, construct an instance of the corresponding export strategy abstract class, and obtain the corresponding instance of the export strategy abstract class according to the specified file format through the strategy factory.
7. The concurrent and secure adaptive data export device according to claim 6, characterized in that: Based on the obtained instance of the export strategy abstract class, first create an instance of the writer, and then repeatedly create instances of shards in the same writer; If the passed shard name is the same as the existing shard name, the instance of the original shard is returned; otherwise, a new shard instance is created; after that, the data write operation can be called multiple times, each time a batch of data sets are passed in, and the corresponding shard writes the data into it.
8. The concurrent and secure adaptive data export device according to claim 5, characterized in that: When calculating the cell style, the style parser dynamically builds the conditional judgment logic according to the following steps: The style parser parses the style rule parameters in the JSON array format and constructs a condition tree data structure. Each style rule parameter in the JSON array format corresponds to a node in the condition tree. The node contains an operator, a value, and a linkOperator to the next node. The style parser constructs a conditional judgment function expression containing variable placeholders according to the data structure of the conditional tree; the conditional judgment function expression receives the value of the cell as input and returns a Boolean value indicating whether the condition is met; the construction of the conditional judgment function expression uses a recursive method to deeply traverse the conditional tree; At the leaf node, determine whether the cell value satisfies the operator and value conditions of the node; at the non-leaf node, merge the results of the leaf node conditions according to the linkOperator operator; When the style parser calculates the style of a cell, it traverses all style rules and executes the conditional judgment function expression in the condition attribute. If it returns True, the style of the style rule is applied to the cell.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 4.
