Object-oriented storage file deep analysis and preview method and system

Abstract

The invention relates to a deep analysis and preview method and system for an object storage file, and the method comprises the following steps: obtaining the path information of a target file in S3 object storage, and analyzing the format of the target file; in response to the fact that the format of the target file is JSON or JSONL, performing deep analysis and rendering on the file, and outputting a visual signal to an interactive interface; and in response to the field in the target file selected by the user, outputting preview information of the corresponding field. Compared with the prior art, the method has the advantages of improving data viewing efficiency, being simple and convenient to operate, rich in function and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of file visualization, and in particular to a method and system for deep analysis and preview of object-oriented storage files. Background Art

[0002] With the advent of the big data era, data visualization tools play an increasingly important role in data analysis and decision support. There are many data visualization tools on the market. Most of these tools support previews of common data formats such as CSV and TXT, but the preview support for structured data such as Json and Jsonl is relatively limited. The common practice is that users need to download files to the local computer and then use special software or online tools to preview them, which is cumbersome and inefficient. In addition, existing tools often lack flexible preview mode switching functions and the ability to compare multiple files at the same time. Therefore, existing visualization tools have disadvantages such as single functions and complex operations.

[0003] In summary, existing data visualization tools have technical problems such as low preview efficiency, inconvenient operation, lack of flexible switching of preview modes and lack of ability to compare multiple files when previewing various data formats (especially Json and Jsonl formats) on object storage S3. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a deep analysis and preview method and system for object-oriented storage files, so as to solve or partially solve the technical problems such as low preview efficiency, inconvenient operation, lack of flexible switching of preview modes and ability to compare multiple files when previewing multiple data formats on object storage S3.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] One aspect of the present invention provides a method for deep parsing and previewing an object-oriented storage file, comprising the following steps:

[0007] Get the path information of the target file in the S3 object storage and parse the format of the target file;

[0008] In response to the target file being in a JSON or JSONL format, deeply parsing and rendering the file, and outputting a visualization signal to an interactive interface;

[0009] In response to the user selecting a field in the target file, preview information of the corresponding field is output.

[0010] As a preferred technical solution, the following steps are also included:

[0011] In response to the preview mode selected by the user, the preview information is displayed in the corresponding preview mode.

[0012] As a preferred technical solution, the preview mode includes one or more of web page, Markdown, picture, JSON, and text.

[0013] As a preferred technical solution, the interactive interface includes multiple sub-interfaces, and each sub-interface corresponds to the visualization of a target file.

[0014] As a preferred technical solution, the process of parsing the format of the target file includes:

[0015] Determine whether there is a target file corresponding to the path information in the cache, if so, obtain the target file from the cache, if not, read the target file from the S3 object storage;

[0016] Based on the HTTP header information of the target file, the target file format is obtained by parsing the HTTP header information.

[0017] As a preferred technical solution, it also includes:

[0018] In response to the target file not including file format information, the target file format is obtained by reading the content of the target file and parsing it.

[0019] As a preferred technical solution, the process of deep parsing and rendering the file includes the following steps:

[0020] For a target file of text type, determine whether a range is pre-specified, if so, read the content matching the specified range, if not, read the content of a preset size in sequence;

[0021] For the target file in JSONL format, read the content of the first line and truncate it if the first line size exceeds the preset value;

[0022] Based on the format of the target file, calling a preset corresponding renderer to implement rendering of the target file;

[0023] For the target file in JSON format, traverse the structured JSON syntax tree and parse the field structure and field value of the object;

[0024] For links in http, https, or S3 formats, add interactive HTML a tags and JS codes.

[0025] As a preferred technical solution, the following steps are also included:

[0026] For the target file in JSON format, record the fields selected by the user;

[0027] For the target file in JSONL format, in response to switching the next line or the previous line, the block previewed last time is updated based on the recorded field path;

[0028] In response to the user selecting a field in the target file as a field including a link, the content of the link is opened in the interactive interface.

[0029] As a preferred technical solution, it also includes:

[0030] For the target file in SONL format, during the parsing process, the current line and the next line are parsed at the same time, and cached asynchronously based on the shard URL of the current line as the key;

[0031] When obtaining the file segment content, determine whether the target file segment content is in the cache. If not, obtain the target file segment content from the S3 object storage. If so, obtain the target file segment content from the cache, create an asynchronous task to check whether the next line of content is cached. If not, obtain the next line of content from the S3 object storage and cache it.

[0032] Another aspect of the present invention provides a deep parsing and previewing system for object-oriented storage files, which is used to implement the aforementioned deep parsing and previewing method for object-oriented storage files. The system includes:

[0033] S3 object storage module;

[0034] The parsing module is used to obtain the path information of the target file in the S3 object storage and parse the format of the target file;

[0035] Cache module, used to cache target files;

[0036] An authorization module, used to grant access to the S3 object storage module through identity verification;

[0037] The renderer module includes multiple renderers for various file formats. It deeply parses and renders the files and outputs visualization signals to the interactive interface.

[0038] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0039] (1) Improve data viewing efficiency: By deeply parsing and rendering JSON or JSONL files and outputting visualization signals to the interactive interface, the preview efficiency of Json and JsonL files is improved. When the user selects a field in the target file, the preview information of the corresponding field is output, and the key click expansion function is realized. By making the interactive interface include multiple sub-interfaces, each sub-interface corresponds to the visualization of a target file, and the multi-file comparison function is realized. Users can view and compare data more quickly and intuitively, which significantly improves data viewing efficiency.

[0040] (2) Easy operation: Users do not need to frequently switch files and interfaces. The operation is simple and the complexity of use is reduced.

[0041] (3) Rich in functions: It supports preview of multiple data formats and multiple preview modes, and is rich in functions to meet the diverse needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a method for deep parsing and previewing object-oriented storage files in an embodiment;

[0043] Figure 2 Schematic diagram of a deep parsing and preview system for object-oriented storage files in an embodiment;

[0044] Figure 3 This is a schematic diagram of the file preview process in the embodiment;

[0045] Figure 4 This is a visual interaction diagram of the JSON / JSONL deep parsing previewer in the embodiment;

[0046] Figure 5 It is a cache strategy diagram in the embodiment;

[0047] Figure 6 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0049] The definitions of the relevant terms involved in the embodiments are as follows:

[0050] S3 Object Storage: An object-based storage system that can store and retrieve any type and amount of data

[0051] File renderer: refers to a software tool or component that can convert the content of a file in a specific format into a visual or understandable form. It can convert it into a visual or understandable form according to the type of input file and the application scenario.

[0052] AK&SK: AK (Access key id) is a public identifier, similar to a username; SK (Secret keyid) is a string that needs to be kept confidential, similar to a password

[0053] Example 1

[0054] This embodiment provides a deep parsing and preview method for object storage files, aiming to solve the technical problems of low preview efficiency, inconvenient operation, lack of flexible switching of preview modes and ability to compare multiple files when previewing multiple data formats (especially Json and Jsonl formats) on object storage S3 in existing data visualization tools. This method can not only quickly preview multiple data formats on object storage, but also specially optimize the preview experience of Json and Jsonl formats, so that users can quickly preview, flexibly switch preview modes and compare multiple files, thereby greatly improving the efficiency of data viewing and analysis.

[0055] See also Figure 1 , the method comprises the following steps:

[0056] Step S1, obtaining the path information of the target file in the S3 object storage and parsing the format of the target file.

[0057] For details, see Figure 3 , step S1 may include:

[0058] Step 1. The user enters the S3 path that needs to be accessed (the file is stored in S3).

[0059] Step 2. Interact with the S3 service and obtain access to the S3 service through authentication. After verification, parse the object key to obtain the file name, path, and other information.

[0060] Step 3. After parsing the objects in the S3 bucket, identify the file formats of these objects. Then automatically call the corresponding renderer according to the file format to render and convert the file content into a visual or understandable form. Preferably, the file formats that can be rendered include but are not limited to Json, Jsonl, TXT, PDF, CSV, XML, and YAML.

[0061] Step S2, in response to the target file being in JSON or JSONL format, deeply parse and render the file, and output a visualization signal to the interactive interface.

[0062] For details, see Figure 3 , provides in-depth analysis preview function for JSON and JSONL format files. When previewing JSON / JSONL files in depth, users can click each key in the file to preview it, and the content of each key supports free switching of multiple preview modes.

[0063] Step S3: in response to the user selecting a field in the target file, outputting preview information of the corresponding field.

[0064] See also Figure 4 This method provides a simple and easy-to-use Json / Jsonl visualization page, allowing users to click each key in the file to preview it, and supports flexible switching of multiple preview modes according to needs.

[0065] A typical implementation process includes the following steps:

[0066] Step 1: File acquisition and format identification.

[0067] 1. Enter a file address on the user side (if the same file address is requested within 10 seconds, it will be obtained from the browser cache). After receiving the request, the server uses boto3 to read the HTTP header information of the object from the S3 object storage and determine the file type based on the Content-Type field in the header information.

[0068] 2. If the file is a raw file (i.e., without extension and without file format information), read the first 1kb of the file to determine the file type (using libmagic).

[0069] Step 2: Shard reading and parsing.

[0070] 1. When reading text files, if no range is specified, 1Mb of content is read in sequence (the range parameter is set in advance by the front-end through the UI); if it is a jsonl file, the range of the first line is read and returned to the client (a line can read a maximum of 5Mb, and if it exceeds, it will be truncated). The returned content includes the fragment URL, file type, content size, etc. of the current content.

[0071] 2. The front end obtains the file content and identifies the file type icon and renderer match based on the obtained file type.

[0072] 3. The json content is rendered by the json renderer. Through the json parsing plug-in we developed, the structured json syntax tree is traversed to parse the field key and field value of the object. If it is an http / https / S3 file link, an interactive HTML a tag and related JS code are added.

[0073] Step 3: Record the field path and render field.

[0074] 1. The JSON renderer records the field path of each click, such as abce. In theory, you can click on the field infinitely and open a new block for rendering (depending on hardware limitations).

[0075] 2. For the JSONL content, switch between the next line and the previous line for preview. The JSON renderer records all selected field paths and updates the content according to the last previewed block when switching.

[0076] 3. Click the field value (clickable link type) to open the file content on the current page, and reuse the file renderer module to open the media file or text file.

[0077] See also Figure 5 Considering the problem of unsatisfactory cache efficiency during file segment browsing, this embodiment also provides an efficient cache strategy, which includes the following steps:

[0078] 1. For repeated requests for the same link file content within 10 seconds, read directly from the first cache content (cached in the browser memory)

[0079] 2. When the parsing service obtains a row of JSONL content, in addition to returning the content of the current row, the parsing service also obtains the content of the next row in advance and caches it (Redis) asynchronously according to the shard URL of this row as the key, and caches it for 2 minutes.

[0080] 3. When obtaining the content of the file shard, it is first obtained from the Redis cache, and an asynchronous task is created at the same time. This asynchronous task will check whether the next line of content has been cached. If the cache does not exist, the file content is obtained from S3 and cached.

[0081] After preliminary experiments and simulation tests, the results show that this method can stably support previews of multiple data formats on S3, especially previews of Json and Jsonl files. Users can preview by clicking on the key in the file and flexibly switch preview modes. At the same time, the multi-file comparison function also shows good performance and user experience.

[0082] This method has the following characteristics:

[0083] (1) Expand and preview the key values ​​of fields in JSON files: When a user clicks on any key value in a JSON file (or a row in JSONL), the system will open a new column on the current page to display the preview effect. It supports multiple preview modes (such as web pages, Markdown, images, JSON, text, etc.), allowing users to intuitively view and analyze data structures. By optimizing the preview method of Json and JsonL files, users are allowed to click on each key in the file to preview.

[0084] (2) Multi-file comparison: Users can open multiple JSON files in the same interface. The system will automatically compare the data of different files. By listing the same structural parts of different files, it is easy to compare and analyze, allowing users to quickly discover data differences and improve the efficiency of data analysis.

[0085] (3) View mode switching: Provides a variety of view modes, users can choose to view file content in formats such as web page, Markdown, image, Json, text, etc. The content of each key supports the switching function of multiple preview modes.

[0086] Example 2

[0087] Based on Example 1, see Figure 2 , this embodiment provides a deep parsing and preview system for object-oriented storage files, which mainly includes two parts: the front end and the back end.

[0088] The backend part includes:

[0089] S3 object storage module;

[0090] The parsing module is used to obtain the path information of the target file in the S3 object storage and parse the format of the target file.

[0091] Cache module, used to cache target files.

[0092] The authorization module is used to grant access to the S3 object storage module through identity verification.

[0093] Specifically, in this embodiment, the backend uses Python FastAPI as the Web server framework and Boto3 as the S3 object storage SDK.

[0094] The front-end part includes:

[0095] The renderer module includes multiple renderers for various file formats. It deeply parses and renders the files and outputs visualization signals to the interactive interface.

[0096] Specifically, in this embodiment, the front end uses the React library to abstractly construct functional modules.

[0097] In summary, the present invention has the following beneficial effects:

[0098] (1) Improve data viewing efficiency: The preview efficiency of Json and Jsonl files has been improved. Through the key click expansion function and multi-file comparison function, users can view and compare data more quickly and intuitively, significantly improving data viewing efficiency.

[0099] (2) Easy operation: Users do not need to frequently switch files and interfaces. The operation is simple and the complexity of use is reduced.

[0100] (3) Rich in functions: It supports preview of multiple data formats and multiple preview modes, and is rich in functions to meet the diverse needs of users.

[0101] Example 3

[0102] Based on the above-mentioned embodiments, this embodiment provides a data visualization platform, including the object-oriented storage file deep analysis and preview system in Embodiment 2.

[0103] This platform can be used as part of a data analysis tool to help users analyze and understand data more quickly.

[0104] In addition, this platform can be used as a data monitoring platform to display and compare monitoring data in real time, helping users to discover and solve problems in a timely manner.

[0105] Example 4

[0106] See also Figure 6 This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the deep parsing and preview method of object-oriented storage files as described in Example 1.

[0107] At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The in-depth analysis and preview method described above. Of course, in addition to the software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0108] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0109] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0110] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for deep parsing and previewing object-oriented storage files, characterized in that: The steps include: Get the path information of the target file in the S3 object storage and parse the format of the target file; In response to the target file being in a JSON or JSONL format, deeply parsing and rendering the file, and outputting a visualization signal to an interactive interface; In response to the user selecting a field in the target file, preview information of the corresponding field is output.

2. The method for deep parsing and previewing object-oriented storage files according to claim 1, characterized in that: The following steps are also included: In response to the preview mode selected by the user, the preview information is displayed in the corresponding preview mode.

3. The method for deep parsing and previewing object-oriented storage files according to claim 2, characterized in that: The preview mode includes one or more of web page, Markdown, image, JSON, and text.

4. The object-oriented storage file deep analysis and preview method according to claim 1, characterized in that: The interactive interface includes a plurality of sub-interfaces, and each sub-interface realizes the visualization of a corresponding target file.

5. The method for deep parsing and previewing object-oriented storage files according to claim 1, characterized in that: The process of parsing the format of the target file includes: Determine whether there is a target file corresponding to the path information in the cache, if so, obtain the target file from the cache, if not, read the target file from the S3 object storage; Based on the HTTP header information of the target file, the target file format is obtained by parsing the HTTP header information.

6. The method for deep parsing and previewing object-oriented storage files according to claim 5, characterized in that: Also includes: In response to the target file not including file format information, the target file format is obtained by reading the content of the target file and parsing it.

7. The object-oriented storage file deep analysis and preview method according to claim 1, characterized in that: The process of deep parsing and rendering the file includes the following steps: For a target file of text type, determine whether a range is pre-specified, if so, read the content matching the specified range, if not, read the content of a preset size in sequence; For the target file in JSONL format, read the content of the first line and truncate it if the first line size exceeds the preset value; Based on the format of the target file, calling a preset corresponding renderer to implement rendering of the target file; For the target file in JSON format, traverse the structured JSON syntax tree and parse the field structure and field value of the object; For links in http, https or S3 format, add interactive HTML tags and JS codes.

8. The method for deep parsing and previewing object-oriented storage files according to claim 1, characterized in that: The following steps are also included: For the target file in JSON format, record the fields selected by the user; For the target file in JSONL format, in response to switching the next line or the previous line, the block previewed last time is updated based on the recorded field path; In response to the user selecting a field in the target file as a field including a link, the content of the link is opened in the interactive interface.

9. The object-oriented storage file deep analysis and preview method according to claim 1, characterized in that: Also includes: For the target file in SONL format, during the parsing process, the current line and the next line are parsed at the same time, and cached asynchronously based on the shard URL of the current line as the key; When obtaining the file segment content, determine whether the target file segment content is in the cache. If not, obtain the target file segment content from the S3 object storage. If so, obtain the target file segment content from the cache, create an asynchronous task to check whether the next line of content is cached. If not, obtain the next line of content from the S3 object storage and cache it.

10. A deep parsing and preview system for object-oriented storage files, characterized in that: The system is used to implement the deep parsing and preview method of object-oriented storage files as described in any one of claims 1 to 9, and comprises: S3 object storage module; The parsing module is used to obtain the path information of the target file in the S3 object storage and parse the format of the target file; Cache module, used to cache target files; An authorization module, used to grant access to the S3 object storage module through identity verification; The renderer module includes multiple renderers for various file formats. It deeply parses and renders the files and outputs visualization signals to the interactive interface.

Images