Network data saving method and apparatus, device, and medium

CN117389963BActive Publication Date: 2026-09-22BUSINESS LINE COMMERCIAL PTE LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311317931.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-11
Publication Date
2026-09-22
Estimated Expiration
2043-10-11

AI Technical Summary

Technical Problem

[0002]现有的互联网平台通常具有其为平台用户提供各类型网络服务的平台网站,平台用户所处的用户端可打开平台网站,在平台网站对应的网页中使用平台所提供的各类型的网络服务,平台网站中各类型的网络服务一般由各类型的网络数据作为网页元素所构成,以形成平台网站所对应的网页,例如,对于互联网电商平台而言,其平台网站一般为供平台用户进行电商购物与电商销售的电商平台网站,电商平台网站的网页中具有由商品标题、商品图片或商品详细信息等电商网络数据作为网页元素所构成的电商网络服务,如电商商品浏览网络服务、电商商品详细页网络服务或电商订单网络服务等,平台通常会搜索下载自身的平台网站中的网络数据,以分析各类网络数据在平台网站的网页中所对应的页面元素进行页面优化,但在网络数据搜索下载系统在存储方面存在无法集中分类存储的问题,搜索下载的网络数据通常仅能以网站为其所定义的网站页面元素进行粗分类,平台的开发人员若想以自身的开发需求进行分类保存,仅能由开发人员自行在巨量的网络数据中进行人工筛选,这将消耗开发人员大量的时间

Benefits of technology

[0057]本申请可将从互联网平台的平台网站中搜索下载的网络数据进行自动化数据分类保存处理,无需互联网平台的开发人员自行对网络数据进行数据分类存储,将搜索下载的网络数据以分隔符分隔的数据保存方式,分类保存至对应的数据保存文件中,开发人员可在网络数据搜索下载系统中配置多个用于保存不同数据类型至对应数据保存文件的分隔符分隔数据保存模板,使网络数据搜索下载系统生成各分隔符分隔数据保存模板对应的数据导出器,进而当网络数据搜索下载系统从平台网站中搜索下载网络数据实体后,使用数据导出器,将网络数据实体中分隔符分隔数据保存模板中由开发人员设置的目标保存数据类型对应的网络数据,导出保存至分隔符分隔数据保存模板中由开发人员设置的数据保存文件名对应的数据保存文件中,且数据保存文件中以分隔符分隔的数据保存方式保存导出的各网络数据,使平台的开发人员可通过配置分隔符分隔数据保存模板,驱动网络数据搜索下载系统将开发人员所需导出的网络数据分隔符在对应的数据保存文件中以分隔符分隔的数据保存方式进行数据保存,进而为开发人员提供一种网络数据分类保存处理服务,且借助以分隔符分隔进行数据保存的数据保存文件具有的纯文本文件格式的特性,使保存的网络数据易于被数据交换,便于对网络数据进行数据入表存储处理或数据入库存储处理,提升开发人员的网络数据分析效率。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117389963B_ABST
    Figure CN117389963B_ABST
Patent Text Reader

Abstract

The application discloses a network data saving method and device, equipment and medium, and the method comprises the following steps: acquiring one or more separators to separate the target saving data type of a data saving template, detecting the dynamic self-defined file name of the separator-separated data saving template as a data saving file name; acquiring the preconfigured file name of the separator-separated data saving template without the dynamic self-defined file name as the data saving file name; generating a data exporter for each separator-separated data saving template, and storing each data exporter into an exporter pool; acquiring a network data entity searched and downloaded from a target website, using the data exporter in the exporter pool to export the network data in the network data entity to the corresponding separator-separated data saving file. The application automatically saves the network data searched and downloaded from the Internet in the separator-separated data saving format into the corresponding file, and optimizes the saving and management process of the network data of the platform.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of network data extraction, and in particular to a method for network data storage, as well as the corresponding apparatus, equipment, and non-volatile storage medium. Background Technology

[0002] Existing internet platforms typically have websites that provide various types of online services to their users. Users can access these websites and utilize the services offered. These services are generally composed of various types of online data as webpage elements. For example, e-commerce platforms typically have websites for users to shop and sell online. These websites offer e-commerce services composed of e-commerce data such as product titles, images, and details, including services like product browsing, product detail pages, and order processing. Platforms usually search and download data from their websites to analyze the corresponding page elements and optimize their pages. However, these data search and download systems often suffer from a lack of centralized categorization. Downloaded data is usually only roughly categorized by the website's defined page elements. If developers want to categorize and save data according to their specific needs, they must manually sift through massive amounts of data, consuming a significant amount of their time.

[0003] In view of the existing problems with the storage of data downloaded from online data searches, the applicant has made corresponding explorations in order to solve these problems. Summary of the Invention

[0004] The purpose of this application is to provide a method for storing network data to meet user needs, and it also relates to the corresponding apparatus, equipment, non-volatile storage medium and computer program products.

[0005] To achieve the objectives of this application, the following technical solution is adopted:

[0006] Obtain the target data type of one or more delimiter-separated data storage templates, detect delimiter-separated data storage templates with dynamically customized filenames, and use the dynamically customized filenames as the data storage filenames of the delimiter-separated data storage templates;

[0007] Obtain the pre-configuration file name of the delimiter-separated data saving template that does not have the dynamically customized file name, and use the pre-configuration file name as the data saving file name of the delimiter-separated data saving template;

[0008] For each of the delimiters, a data exporter is generated with the target data type and the data file name as the exported data type and data save path, respectively, and each of the data exporters and its corresponding data file name are mapped and stored in the exporter pool;

[0009] Obtain one or more network data entities downloaded from the target website, and use each of the data exporters in the exporter pool to export the network data corresponding to the target data type of each delimiter-separated data storage template in each network data entity to the delimiter-separated data storage file of the corresponding data storage file name.

[0010] In a further embodiment, the step of obtaining the target data type of one or more delimiter-separated data storage templates, detecting delimiter-separated data storage templates with dynamically customized filenames, and using the dynamically customized filenames as the data storage filenames for the delimiter-separated data storage templates includes:

[0011] In response to a dynamic custom filename edit event, determine the target delimiter for saving data in the dynamic custom filename edit event;

[0012] Obtain the dynamically customized filename from the dynamically customized filename editing event, and store the dynamically customized filename in the target delimiter-separated data saving template.

[0013] In a further embodiment, the step of generating a data exporter for each of the delimiters-separated data storage templates, using the target data type and the data storage file name as the exported data type and data storage path respectively, and mapping and storing each of the data exporters and their corresponding data storage file names in the exporter pool, includes:

[0014] Retrieves one or more target data types and filenames for saving data in a delimited data saving template;

[0015] Call the delimiter-separated data exporter generation interface, take each of the target save data types as the export data type, take the data save file name as the data save path, and generate a data exporter that acts on the delimiter-separated data save template;

[0016] Generate mapping data consisting of the data exporter and the data save file name, and store the mapping data in the exporter pool.

[0017] In a further embodiment, the step of obtaining one or more network data entities downloaded from the target website includes:

[0018] Download network data from the target website corresponding to the preset target website link;

[0019] Based on one or more preset network data entity templates, target network data corresponding to one or more preset target data types in each network data is extracted from each network data.

[0020] Generate one or more network data entities composed of the target network data for each of the network data entity templates.

[0021] In a further embodiment, the step of obtaining one or more network data entities downloaded from a target website, and using each of the data exporters in the exporter pool to export the network data corresponding to the target data type of each delimiter-separated data storage template in each network data entity to a delimiter-separated data storage file of the corresponding data storage file name, includes:

[0022] Obtain the network data entity and determine the delimiter corresponding to the network data entity to save the data template.

[0023] Obtain the dynamic custom file name or pre-configuration file name of the delimiter-separated data storage template, and query the data exporter mapped to the dynamic custom file name or the pre-configuration file name in the exporter pool;

[0024] Determine one or more export data types and data storage paths corresponding to the data exporter, obtain network data corresponding to each export data type from the network data entity, and save each network data to a delimiter-separated data storage file corresponding to the data storage path.

[0025] In a further embodiment, after the step of exporting the network data corresponding to the target data type of each delimiter-separated data storage template in each of the network data entities to the delimiter-separated data storage file with the corresponding data storage file name, the method includes:

[0026] In response to a data entry command applied to a delimiter-separated data storage file, obtain one or more data entry network data type matching rules corresponding to the data entry command;

[0027] According to the matching rules for each type of network data type entering the table, the network data corresponding to the target data type entering the table according to the matching rules for each type of network data type entering the table is matched from the data storage file separated by the delimiter.

[0028] Determine the network data storage table corresponding to the data entry instruction, and store each network data in the storage column position corresponding to the target storage column name in the network data storage table according to the target storage column name matching rule of each network data type.

[0029] In a further embodiment, after the step of exporting the network data corresponding to the target data type of each delimiter-separated data storage template in each of the network data entities to the delimiter-separated data storage file with the corresponding data storage file name, the method includes:

[0030] In response to a data save file retrieval command pushed by the management terminal, retrieve one or more data save file names contained in the data save file retrieval command;

[0031] Retrieve the target delimited data file corresponding to the data file name from the delimiter-separated data file library;

[0032] The data files separated by the delimiters are encapsulated into a data file set, and the data file set is pushed to the management terminal.

[0033] A network data storage device proposed for the purposes of this application includes:

[0034] The dynamic filename acquisition module is used to acquire the target data type of one or more delimiter-separated data storage templates, detect delimiter-separated data storage templates with dynamic custom filenames, and use the dynamic custom filenames as the data storage filenames of the delimiter-separated data storage templates.

[0035] The configuration file name acquisition module is used to acquire the pre-configuration file name of the delimiter-separated data saving template that does not have the dynamically customized file name, and use the pre-configuration file name as the data saving file name of the delimiter-separated data saving template.

[0036] The data exporter generation module generates a data exporter for each of the delimiter-separated data saving templates, using the target data type and the data saving file name as the export data type and data saving path respectively, and maps and stores each of the data exporters and their corresponding data saving file names in the exporter pool;

[0037] The data export and save module is used to obtain one or more network data entities downloaded from the target website, and use the data exporters in the exporter pool to export the network data corresponding to the target save data type of each delimiter-separated data save template in each network data entity to the delimiter-separated data save file of the corresponding data save file name.

[0038] In a further embodiment, the dynamic filename acquisition module includes:

[0039] The filename editing submodule is used to respond to dynamic custom filename editing events and determine the target delimiter for saving data in the dynamic custom filename editing event.

[0040] The filename saving submodule is used to obtain the dynamically customized filename of the dynamically customized filename editing event and store the dynamically customized filename in the target delimiter separated data saving template.

[0041] In a further embodiment, the data exporter generation module includes:

[0042] The template data retrieval submodule is used to retrieve one or more target data types and filenames of the delimiter-separated data template.

[0043] The data exporter generation submodule is used to call the delimiter-separated data exporter generation interface, take each of the target save data types as the export data type, take the data save file name as the data save path, and generate a data exporter that acts on the delimiter-separated data save template.

[0044] The data exporter storage submodule is used to generate mapping relationship data consisting of the data exporter and the data save file name, and store the mapping relationship data in the exporter pool.

[0045] In a further embodiment, the data export and saving module includes:

[0046] The network data download submodule is used to download network data from the target website linked from the preset target website;

[0047] The network data extraction submodule is used to extract target network data corresponding to one or more preset target data types from each network data according to one or more preset network data entity templates.

[0048] The data entity generation submodule is used to generate one or more network data entities composed of the target network data for each of the network data entity templates.

[0049] In a preferred embodiment, the data export and saving module further includes:

[0050] The template determination submodule is used to obtain network data entities and determine the delimiter corresponding to the network data entity to separate the data and save the template.

[0051] The data exporter query submodule is used to obtain the dynamically customized file name or pre-configuration file name of the delimiter-separated data storage template, and query the data exporter mapped and stored by the dynamically customized file name or the pre-configuration file name in the exporter pool.

[0052] The network data storage submodule is used to determine one or more export data types and data storage paths corresponding to the data exporter, obtain network data corresponding to each export data type from the network data entity, and save each network data to a delimiter-separated data storage file corresponding to the data storage path.

[0053] To address the aforementioned technical problems, this application also provides a network data storage device, including a memory and a processor. The memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform the steps of the network data storage method described above.

[0054] To address the aforementioned technical problems, this application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the network data storage method described above.

[0055] To address the aforementioned technical problems, this application also provides a computer program product, including a computer program and computer instructions. When the computer program and computer instructions are executed by a processor, the processor performs the steps of the aforementioned network data storage method.

[0056] Compared with existing technologies, the advantages of this application are as follows:

[0057] This application automates the data classification and saving process for web data downloaded from internet platform websites, eliminating the need for internet platform developers to manually classify and store the data. The downloaded web data is saved to corresponding data files using a delimiter-based data saving method. Developers can configure multiple delimiter-based data saving templates in the web data search and download system to save different data types to their respective data saving files. The system then generates a data exporter for each delimiter-based data saving template. When the web data search and download system searches and downloads web data entities from the platform website, it uses the data exporter to extract the web data corresponding to the target data type set by the developer from the delimiter-based data saving templates. The exported data is saved to a data file corresponding to the filename set by the developer in the delimited data saving template. The exported network data is saved in the data saving file using a delimiter-separated data saving method. This allows the platform's developers to configure the delimiter-separated data saving template and drive the network data search and download system to save the network data to be exported by the developers in the corresponding data saving file using a delimiter-separated data saving method. This provides developers with a network data classification and saving service. Furthermore, the plain text file format of the data saved by delimiters makes the saved network data easy to exchange and facilitates data entry into tables or databases, thereby improving the efficiency of network data analysis for developers.

[0058] Secondly, in this application, developers can configure new data save file names for the delimiter-separated data save template at any time. That is, the delimiter-separated data save template in this application has a dynamic custom file name, which makes it easy for developers to dynamically modify the data save file corresponding to the target data type of the delimiter-separated data save template after exporting network data. This ensures that the data save location of the network data meets the needs of the developers' network data analysis work and guarantees the efficiency of the developers in optimizing the platform website.

[0059] In addition, this application stores the data exporters generated by the data saving template separated by the delimiter in the export pool, so that the generated data exporters can be repeatedly used for the export and saving processing of the corresponding network data entities, saving the computing resources wasted by repeatedly generating data exporters, and improving the execution speed of exporting and saving network data. Attached Figure Description

[0060] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0061] Figure 1 A typical network deployment architecture diagram related to the implementation of the technical solution of this application;

[0062] Figure 2 This is a flowchart illustrating a typical embodiment of the network data storage method of this application;

[0063] Figure 3 This is a flowchart illustrating the specific implementation of the method described in this application for configuring dynamically customized filenames for a data saving template that is separated by a corresponding delimiter in response to a dynamically customized filename editing event.

[0064] Figure 4 This is a flowchart illustrating a specific implementation of the data exporter for generating delimiter-separated data storage templates and storing the data exporter in an exporter pool in this application.

[0065] Figure 5 This is a flowchart illustrating a specific implementation method in this application for extracting corresponding network data from network data based on a network data entity template to form a network data entity.

[0066] Figure 6 This is a flowchart illustrating a specific implementation method in this application for exporting corresponding network data from a network data entity using a data exporter and saving it to a corresponding data storage file;

[0067] Figure 7 This is a flowchart illustrating the specific implementation method of matching the delimiter to separate the corresponding network data in the data file and storing the network data in the corresponding storage column position of the network data storage table in this application;

[0068] Figure 8 This is a flowchart illustrating a specific implementation method for generating a set of data save files, consisting of the data save files specified by the instruction, and pushing it to the management terminal in response to a data save file retrieval instruction pushed by the management terminal.

[0069] Figure 9 This is a schematic block diagram of a typical embodiment of the network data storage device of this application;

[0070] Figure 10 This is a basic structural block diagram of a network data storage device according to an embodiment of this application. Detailed Implementation

[0071] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0072] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0073] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0074] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant) that may include a radio frequency receiver, pager, internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; and traditional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0075] The hardware referred to by the names "server," "client," and "work node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0076] It should be noted that the concept of "server" used in this application can also be extended to the case of server clusters. Based on the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a cluster of computers. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method in this application.

[0077] Please see Figure 1 The hardware infrastructure required for implementing the technical solutions of this application can be deployed according to the architecture shown in the figure. The server 80 mentioned in this application is deployed in the cloud and acts as an online server. It can further connect to relevant data servers and other servers providing related support, thereby forming a logically related service cluster to provide services to relevant terminal devices such as the smartphone 81 and personal computer 82 shown in the figure, or third-party servers (not shown). Both the smartphone and personal computer can access the Internet through known network access methods and establish a data communication link with the cloud server 80 to run terminal applications related to the services provided by the server.

[0078] For servers, the application is usually built as a service process, with corresponding program interfaces exposed for remote calls by applications running on various terminal devices. The relevant technical solutions in this application that are suitable for running on servers can be implemented in servers in this way.

[0079] The application mentioned refers to an application running on a server or terminal device. This application implements the relevant technical solutions of this application in a programmed manner. Its program code can be stored in a non-volatile storage medium that can be recognized by a computer in the form of computer-executable instructions, and is loaded into memory by the central processing unit for execution. The relevant device of this application is constructed by the operation of the application on the computer.

[0080] For servers, the application is usually built as a service process, with corresponding program interfaces exposed for remote calls by applications running on various terminal devices. The relevant technical solutions in this application that are suitable for running on servers can be implemented in servers in this way.

[0081] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0082] Please see Figure 2 A network data storage method according to this application, in a typical embodiment, includes the following steps:

[0083] Step S11: Obtain the target data type of one or more delimiter-separated data storage templates, detect delimiter-separated data storage templates with dynamically customized filenames, and use the dynamically customized filenames as the data storage filenames for the delimiter-separated data storage templates.

[0084] The delimiter-separated data saving template is used to save corresponding network data in a delimiter-separated data saving method to a file in the same data storage format that stores network data in a delimiter-separated manner. Therefore, the delimiter-separated data saving template generally has a preset target data type to set the network data type to be saved by the delimiter-separated data saving template. In addition, the delimiter-separated data saving template generally also has a preset configuration file name or a dynamically customized file name to set the name of the data saving file for the network data of the target data type, that is, the data saving file for the network data of the specified target data type; wherein, the data storage format for storing network data in a delimiter-separated manner is generally called CSV (Comma Separated Values).

[0085] The delimiter-separated data storage template typically uses delimiters to separate network data of the target data type and stores it in a data storage file. For example, when the delimiter is a comma, and the network data is "A", "B", and "C", then these network data will be stored in the corresponding data storage file in the data storage format of "A,B,C". In addition to commas, the delimiter can also be a tab, space, colon, or vertical bar, etc., to separate different network data. Designers in the art can flexibly design the delimiters, which will not be elaborated here.

[0086] The delimiter-separated data storage template is generally deployed in a web data search and download system built on Scrapy. Specifically, the delimiter-separated data storage template is generally deployed in the data filtering pipeline of the web data search and download system, that is, in the Scrapy Pipeline.

[0087] The dynamically customized filenames are typically configured by developers. Users can edit the dynamically customized filenames for the delimiter-separated data storage template at any time to dynamically modify the data storage file for network data corresponding to the target data type in the delimiter-separated data storage template. This allows developers to dynamically modify the storage file for network data according to actual development scenarios. Specifically, in response to the dynamically customized filename edit event triggered when the user edits the dynamically customized filename of the delimiter-separated data storage template, the dynamically customized filename edited by the user is obtained, and then stored in the delimiter-separated data storage template. This modifies the data storage file for the network data in the delimiter-separated data storage template to the data storage file corresponding to the dynamically customized filename, and then uses the data storage file corresponding to the dynamically customized filename to store the network data corresponding to the delimiter-separated data storage template.

[0088] Step S12: Obtain the pre-configuration file name of the delimiter-separated data saving template that does not have the dynamically customized file name, and use the pre-configuration file name as the data saving file name of the delimiter-separated data saving template.

[0089] For delimited data saving templates that do not have dynamically customizable filenames edited by the user, the default configuration file name of such delimited data saving template will be obtained, and then the default configuration file name will be used as the data saving filename of such delimited data saving model.

[0090] The preset configuration file name is generally the data save file name configured for the delimiter-separated data save template when configuring the delimiter-separated data save template. When the delimiter-separated data save template does not have a dynamically customizable file name, the preset configuration file name of the delimiter-separated data save template will be obtained as the data save file name, so that the network data to be saved by the delimiter-separated data save template will be saved to the data save file corresponding to the preset configuration file name. When the user edits a dynamically customizable file name for the delimiter-separated data save template, the dynamically customizable file name edited by the user will be used as the data save file name, so that the data save path of the network data of the delimiter-separated data save template will be changed from the original data save file corresponding to the preset configuration file name to the data save file corresponding to the dynamically customizable file name.

[0091] Step S13: Generate a data exporter for each delimiter-separated data storage template, using the target data type and the data storage file name as the exported data type and data storage path respectively, and map and store each data exporter and its corresponding data storage file name in the exporter pool:

[0092] After obtaining the target data type and corresponding data file name of each delimiter-separated data saving template deployed in the network data search and download system, a data exporter corresponding to each delimiter-separated data saving template will be generated. The data exporter will then be used to export and save the network data corresponding to the target data type in each delimiter-separated data saving template to the data saving file corresponding to the data saving file name.

[0093] The data exporter is used to export and store network data to a corresponding data save file. The network data to be exported and saved by the data exporter is generally determined according to the target save data type in the delimiter-separated data save template. By using one or more target save data types in the delimiter-separated data save template as the export data type of the data exporter, the data data type to be exported and saved by the data exporter is set. Correspondingly, the data save file for which the data exporter saves its exported network data is generally determined according to the data save file name of the delimiter-separated data save template. By setting the data save file name of the delimiter-separated data save template as the data save path of the data exporter, the data exporter saves its exported network data to the data save file of the data save file name.

[0094] In a web data search and download system built on Scrapy, the data exporter generally refers to the Exporter in Scrapy, and is used to export web data in a delimiter-separated data format. In a web data search and download system based on Scrapy, it is generally called CsvItemExporter.

[0095] After generating a data exporter corresponding to an arbitrary delimiter-separated data storage template, the data exporter and the data storage filename of the delimiter-separated data storage template are stored as mapping data in the exporter pool. The exporter pool stores mapping data composed of data storage filenames and data exporters. Generally, the data storage filename is used as the key data and the data exporter is used as the value data. This facilitates searching for the data exporter mapped to the data storage filename in the exporter pool for network data export and storage based on the delimiter-separated data storage template's data storage filename for network data export and storage. It can be seen that by storing the data exporters generated by the delimiter-separated data storage template in the exporter pool, the generated data exporters can be reused for data export and storage processing of network data of the target storage data type of the delimiter-separated data storage template without regenerating the data exporter, thus saving computing resources.

[0096] Step S14: Obtain one or more network data entities downloaded from the target website, and use each of the data exporters in the exporter pool to export the network data corresponding to the target data type of each delimiter-separated data storage template in each network data entity to the delimiter-separated data storage file of the corresponding data storage file name:

[0097] Once the network data search and download system searches and downloads network data from the target website and extracts and generates the corresponding network data entities from the network data, it will use the data exporter in the exporter pool to export the network data corresponding to the template data type of the delimiter-separated data storage template from each network data entity and save it to the corresponding data storage file.

[0098] The network data entities mentioned generally consist of different network data from the same website. The network data types contained in different network data entities are generally specified by pre-configured network data entity templates. These network data entity templates are generally deployed in the network data search and download system to extract the network data types configured in the network data entity templates from various types of network data searched and downloaded from the target website by the network data search and download system, and generate network data entities corresponding to the network data entity templates. Generally, multiple network data entity templates of different entity types can be pre-configured to extract and generate network data entities of different entity types from the network data searched and downloaded from the target website. For example, in e-commerce development scenarios, developers can pre-configure web data entity templates for extracting image tags (ALT tags), original image URLs, and local image storage addresses from product images. Then, from the web data searched and downloaded from the e-commerce shopping website on their own platform, they can obtain image tags, download links, and local image storage addresses for different product images, generating web data entities for different product images. Of course, they can also pre-configure web data entity templates with entity types such as product text, product price, or product object feature information to categorize and extract different types of web data entities from the web data searched and downloaded from the e-commerce shopping website.

[0099] Specifically, after the network data search and download system searches and downloads network data from the target website, it will extract target network data corresponding to one or more target data types in each network data entity template according to one or more preset network data entity templates, and then generate one or more network data entities composed of the target network data for each network data entity template.

[0100] After obtaining any network data entity, the delimiter-separated data storage template corresponding to that network data entity is determined. This delimiter-separated data storage template generally inherits from the same data base class as the network data entity's network data entity template. The delimiter-separated data storage template containing the network data corresponding to the target data type is identified within the network data entity. The dynamically customized filename or pre-configured filename of this delimiter-separated data storage template is then obtained. The data exporter mapped to the dynamically customized filename or pre-configured filename is queried from the exporter pool. One or more export data types and data storage paths corresponding to the data exporter are then determined. Network data corresponding to each of the exported data types is then obtained from the network data entity, and each of the network data types is saved to the delimiter-separated data storage file corresponding to the data storage path. In other words, the data exporter exports the network data of the target data type of the delimiter-separated data storage template from the network data entity and saves this network data to the data storage file corresponding to the dynamically customized filename or pre-configured filename that serves as the data storage filename of the delimiter-separated data storage template.

[0101] After exporting the network data corresponding to the network data entity to the corresponding data save file through the data exporters in the export pool, the network data saved in these data save files can be stored in the corresponding data table of the database, or a data save file acquisition service can be provided to the management end to generate a data save file set consisting of the data save files required by the management end for push. For specific implementation methods, please refer to the relevant embodiments in the following sections, which will not be elaborated here.

[0102] The above typical embodiments and their variations fully disclose the implementation scheme of the network data storage method of this application. However, various variations of the method can still be derived by changing and expanding some technical means. Other embodiments are briefly described below:

[0103] In one embodiment, please refer to Figure 3 The steps of obtaining the target data type of one or more delimiter-separated data storage templates, detecting delimiter-separated data storage templates with dynamically customized filenames, and using the dynamically customized filenames as the data storage filenames for the delimiter-separated data storage templates include:

[0104] Step S111: In response to the dynamic custom filename editing event, determine the target delimiter-separated data saving template for the dynamic custom filename editing event.

[0105] The aforementioned dynamic custom filename editing event refers to the event triggered when a user edits a delimiter-separated data saving template with a dynamically defined filename to modify the data saving file of the network data in the delimiter-separated data saving template. The user's client or server responds to the dynamic custom filename editing event and determines the delimiter-separated data saving template for which the dynamic custom filename editing is performed.

[0106] Step S112: Obtain the dynamically customized filename of the dynamically customized filename editing event, and store the dynamically customized filename in the target delimiter-separated data saving template.

[0107] The system retrieves the dynamically customized filename edited by the user for the delimiter-separated data saving template during the dynamically customized filename edit event, stores the dynamically customized filename in the delimiter-separated data saving template, and then uses the dynamically customized filename as the data saving filename of the delimiter-separated data saving template before generating the data exporter of the delimiter-separated data saving template.

[0108] In this embodiment, developers can customize the filename of the data storage template that is separated by delimiters, so as to dynamically modify the file of the network data corresponding to the target data type of the data storage template that is separated by delimiters, and modify the data storage path of the network data, so that the storage location of the network data can meet the development needs of the developers and improve the development efficiency of the developers.

[0109] In one embodiment, please refer to Figure 4 The step of generating a data exporter for each delimiter-separated data storage template, using the target data type and the data storage file name as the exported data type and data storage path respectively, and mapping and storing each data exporter and its corresponding data storage file name in the exporter pool includes:

[0110] Step S131: Obtain one or more target data types and data file names for the delimiter-separated data saving template.

[0111] After obtaining one or more target data types and filenames for the delimited data saving template, a corresponding data exporter will be generated for the delimited data saving template.

[0112] Step S132: Call the delimiter-separated data exporter generation interface, take each of the target save data types as the export data type, take the data save file name as the data save path, and generate a data exporter that operates on the delimiter-separated data save template.

[0113] The aforementioned delimiter-separated data exporter generation interface generally refers to the method call interface used to generate delimiter-separated data exporter methods in a network data search and download system built based on Scrapy, such as the call interface provided by the CsvItemExporter method in Scrapy.

[0114] The delimiter-separated data exporter generation interface is called to take one or more target data types of the delimiter-separated data saving template as the exported data type and the data saving file name of the delimiter-separated data saving template as the data saving path, thereby generating a data exporter whose exported data type is each of the target data types and whose data saving path is the data saving file name.

[0115] Step S133: Generate mapping relationship data consisting of the data exporter and the data save file name, and store the mapping relationship data in the exporter pool:

[0116] After generating the data exporter for the delimited data storage template, the data exporter and the data storage file name of the delimited data storage template are stored in the exporter pool as a mapping relationship. The data storage file name is used as the key data, and the data exporter is used as the value data. That is, the data storage file name and the data exporter are key-value pairs.

[0117] In this embodiment, a data exporter is generated that uses the target data type of the data storage template separated by delimiters as the exported data type and the data storage file name of the data storage target separated by delimiters as the data storage path. The network data corresponding to the target data type is exported and saved to the file with the data storage file name. The data exporter and the data storage file name are mapped and stored in the exporter pool, which facilitates the reuse of the generated data exporter and saves the computing resources consumed when generating the data exporter.

[0118] In one embodiment, please refer to Figure 5 The step of obtaining one or more web data entities downloaded from a target website includes:

[0119] Step S141: Download network data from the target website corresponding to the preset target website link:

[0120] The target web link generally refers to a web address used to access a target website via the Internet, such as the URL (Uniform Resource Locator) of the target website. After accessing the target website through the target web link, web data from the target website will be downloaded. For example, when the target website is an e-commerce shopping website, web data such as product images, product titles, product prices, or product types will be downloaded from the e-commerce shopping website.

[0121] Step S142: Based on one or more preset network data entity templates, extract the target network data corresponding to one or more preset target data types from each of the network data:

[0122] The network data entity template sets one or more target data types to extract target network data corresponding to each target data type pre-set in the network data entity template from network data downloaded from the target website, and generate a network data entity composed of each target network data.

[0123] Step S143: Generate one or more network data entities composed of the target network data for each of the network data entity templates:

[0124] After extracting the target network data corresponding to each target data type from the preset target data entity template in the network data downloaded from the target website, the target network data are combined into a network data entity corresponding to its network data entity template.

[0125] In this embodiment, the network data downloaded from the target website is classified into network data entities to group the network data downloaded from the target website, thereby generating network data entities composed of network data corresponding to the preset target data type in the network data entity template. This facilitates the export of the corresponding network data from the network data entities for delimiter-separated data storage.

[0126] In one embodiment, please refer to Figure 6 The steps of obtaining one or more network data entities downloaded from a target website, and using each of the data exporters in the exporter pool to export the network data corresponding to the target data type of each delimiter-separated data storage template in each network data entity to a delimiter-separated data storage file with the corresponding data storage file name, include:

[0127] Step S141': Obtain the network data entity and determine the delimiter-separated data saving template corresponding to the network data entity:

[0128] After obtaining any network data entity, a delimiter-separated data saving template that inherits from the same data base class as the network data entity will be determined, or a delimiter-separated data saving template that corresponds to the network data of the target data type will be determined in the network data entity. The data exporter using the delimiter-separated data saving template will export the corresponding network data in the network data entity and save it to the corresponding data saving file in a delimiter-separated data saving mode.

[0129] Step S142': Obtain the dynamically customized filename or pre-configured filename of the delimiter-separated data storage template, and query the data exporter mapped to the dynamically customized filename or the pre-configured filename in the exporter pool:

[0130] After determining the delimiter-separated data saving template corresponding to the network data entity, the dynamic custom file name or pre-configured file name of the delimiter-separated data saving template will be obtained. Then, a data exporter mapped and stored by the dynamic custom file name or pre-configured file name will be re-generated in the exporter pool, so as to use the data exporter to export and save the network data corresponding to the network data entity.

[0131] Step S143': Determine one or more export data types and data storage paths corresponding to the data exporter; obtain network data corresponding to each export data type from the network data entity; and save each piece of network data to a delimiter-separated data storage file corresponding to the data storage path.

[0132] After obtaining the data exporter corresponding to the delimiter-separated data saving template of the network data entity from the exporter pool, the data exporter will be used to determine the network data corresponding to the exported data type of the data exporter from the network data entity, that is, to determine the network data corresponding to the target saving data type of the delimiter-separated data saving template. Then, the network data will be saved to the data saving file corresponding to the data saving path, that is, the network data will be saved to the data saving file corresponding to the dynamically customized file name or the pre-configured file name of the delimiter-separated data saving target.

[0133] In this embodiment, after obtaining the network data entity, the corresponding data exporter in the exporter pool will be used to determine the corresponding network data from the network data entity. The determined network data will be exported and saved to the corresponding data storage file, and the data will be separated by a delimiter. In this way, the network data will be classified and stored, and the network data will be saved in a data storage method that separates the data by a delimiter.

[0134] In one embodiment, please refer to Figure 7 After the step of exporting the network data corresponding to the target data type of each delimiter-separated data storage template in each of the network data entities to the delimiter-separated data storage file with the corresponding data storage file name, the following is included:

[0135] Step S15: In response to a data entry command applied to a delimiter-separated data storage file, obtain one or more data entry network data type matching rules corresponding to the data entry command:

[0136] The data entry instruction is used to classify and store the delimited network data saved in the delimited data file into the corresponding network data storage table. After responding to the data entry instruction, the delimited data file to be stored in the network data will be determined, and the network data storage table to be used to classify and store the network data saved in the delimited data file will be determined, and one or more corresponding network data type matching rules for entry into the table will be obtained.

[0137] Step S16: Based on the matching rules for each type of network data type entering the table, match the network data corresponding to the target data type of each type of network data type entering the table from the delimiter-separated data storage file.

[0138] The inbound network data type matching rule has a character matching regular expression used to match the data saved by the delimiter to the corresponding storage column position in the network data storage table, to match the network data corresponding to the target inbound data type in the character matching regular expression from the data saved by the delimiter, and then to store the matched network data in the storage column position of the corresponding storage column name in the network data storage table.

[0139] Based on the character matching regular expression of the network data type matching rules in the input table, the network data corresponding to the target input data type is matched from the delimiter-separated data storage file. For example, when the character regular expression is used to match the target input data type with the characters ".jpg" or ".png", the network data with the characters ".jpg" or ".png" will be matched from the network data separated by the delimiter in the delimiter-separated data storage file, and the network data will be stored in the storage column position named "product image" in the network data storage table.

[0140] Step S17: Determine the network data storage table corresponding to the data entry instruction, and according to the target storage column name corresponding to the matching rule of each network data type, store each network data in the storage column position corresponding to the target storage column name in the network data storage table:

[0141] After matching the network data corresponding to the network data type matching rules in the delimiter-separated data file, the storage column position corresponding to the target storage column name in the network data storage table will be queried according to the target storage column name corresponding to the network data type matching rules in the input table, so that the matched network data can be stored in that storage column position.

[0142] In this embodiment, network data stored in a delimiter-separated data file is categorized and stored in a network data storage table. This allows developers to query the corresponding network data in the data table for analysis, improving their network data query efficiency.

[0143] In one embodiment, please refer to Figure 8 After the step of exporting the network data corresponding to the target data type of each delimiter-separated data storage template in each of the network data entities to the delimiter-separated data storage file with the corresponding data storage file name, the following is included:

[0144] Step S15': In response to the data save file retrieval command pushed by the management terminal, retrieve one or more data save file names contained in the data save file retrieval command:

[0145] Developers at the management end can set the data storage file that needs to be obtained to store network data in a delimiter-separated data storage mode. This will trigger the generation of a data storage file retrieval instruction containing the data storage file name, and push it to the server or terminal device that stores the data storage file, so as to retrieve the data storage file from the server or terminal device.

[0146] Step S16': Query the delimiter-separated data save file corresponding to the data save file name from the delimiter-separated data save file library:

[0147] The delimiter-separated data save file library stores delimiter-separated data save files that save network data by delimiters. It can be understood that the delimiter-separated data save files refer to data save files that save network data exported by the data exporter in the exporter pool by delimiters.

[0148] After obtaining one or more data save file names contained in the data save file retrieval instruction, the system will query the delimiter-separated data save file library to find the target delimiter-separated data save file corresponding to each data save file name contained in the data save file retrieval instruction.

[0149] Step S17': Encapsulate the data storage files separated by the delimiters into a data storage file set, and push the data storage file set to the management terminal:

[0150] After retrieving the data files separated by each target delimiter from the database, these data files are packaged into a data file set, which is then pushed to the management terminal.

[0151] In this embodiment, developers can obtain the corresponding delimited data files from the network data storage center that centrally manages delimited data files, perform network data analysis, and optimize the corresponding page elements to assist developers in the website development process and improve their development efficiency.

[0152] Furthermore, by functionalizing the various steps in the methods disclosed in the above embodiments, a network data storage device of this application can be constructed. Following this approach, please refer to... Figure 9 In one typical embodiment, the device includes: a dynamic filename acquisition module 11, used to acquire the target data type of one or more delimiter-separated data storage templates, detect delimiter-separated data storage templates with dynamically customized filenames, and use the dynamically customized filenames as the data storage filenames of the delimiter-separated data storage templates; a configuration file name acquisition module 12, used to acquire the pre-configuration filenames of delimiter-separated data storage templates that do not have the dynamically customized filenames, and use the pre-configuration filenames as the data storage filenames of the delimiter-separated data storage templates; a data exporter generation module 13, used to generate data exporters for each delimiter-separated data storage template, using the target data type and the data storage filename as the export data type and data storage path respectively, and to map and store each data exporter and its corresponding data storage filename in an exporter pool; and a data export and storage module 14, used to acquire one or more network data entities downloaded from a target website, and use each of the data exporters in the exporter pool to export the network data corresponding to the target data type of each delimiter-separated data storage template in each network data entity to the delimiter-separated data storage file with the corresponding data storage filename.

[0153] In one embodiment, the dynamic filename acquisition module 11 includes: a filename editing submodule, used to respond to a dynamic custom filename editing event and determine the target delimiter-separated data storage template for the dynamic custom filename editing event; and a filename saving submodule, used to acquire the dynamic custom filename of the dynamic custom filename editing event and store the dynamic custom filename in the target delimiter-separated data storage template.

[0154] In one embodiment, the data exporter generation module 12 includes: a template data acquisition submodule, used to acquire one or more target data types and data file names of a delimiter-separated data storage template; a data exporter generation submodule, used to call a delimiter-separated data exporter generation interface, using each of the target data types as the export data type and the data file name as the data storage path, to generate a data exporter that acts on the delimiter-separated data storage template; and a data exporter storage submodule, used to generate mapping relationship data composed of the data exporter and the data file name, and store the mapping relationship data in an exporter pool.

[0155] In one embodiment, the data export and storage module 14 includes: a network data download submodule, used to download network data from a target website corresponding to a preset target website link; a network data extraction submodule, used to extract target network data corresponding to one or more preset target data types from each of the network data according to one or more preset network data entity templates; and a data entity generation submodule, used to generate one or more network data entities composed of the target network data for each of the network data entity templates.

[0156] In another embodiment, the data export and save module 14 further includes: a save template determination submodule, used to obtain a network data entity and determine the delimiter-separated data save template corresponding to the network data entity; a data exporter query submodule, used to obtain the dynamically customized file name or pre-configured file name of the delimiter-separated data save template, and query the data exporter mapped and stored by the dynamically customized file name or the pre-configured file name in the exporter pool; and a network data save submodule, used to determine one or more export data types and data save paths corresponding to the data exporter, obtain the network data corresponding to each export data type from the network data entity, and save each network data to the delimiter-separated data save file corresponding to the data save path.

[0157] To address the aforementioned technical problems, this application also provides a network data storage device for running a computer program implemented according to the network data storage method. Please refer to the following for details. Figure 10 , Figure 10 This is a basic structural block diagram of the network data storage device in this embodiment.

[0158] like Figure 10The diagram shows the internal structure of a network data storage device. This device includes a processor, a non-volatile storage medium, a memory, and a network interface connected via a system bus. The non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database stores control information sequences. When the computer-readable instructions are executed by the processor, they enable the processor to implement a network data storage method. The processor provides computing and control capabilities, supporting the operation of the entire network data storage device. The memory stores computer-readable instructions, which, when executed by the processor, enable the processor to implement a network data storage method. The network interface is used for communication with a terminal. Those skilled in the art will understand that… Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the network data storage device to which the present application is applied. A specific network data storage device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0159] In this embodiment, the processor executes the specific functions of each module / submodule in the network data storage device of this application, and the memory stores the program code and various types of data required to execute the above modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules / submodules in the network data storage device, and the server can call the server's program code and data to execute the functions of all submodules.

[0160] This application also provides a non-volatile storage medium in which the network data saving method is written as a computer program and stored in the storage medium in the form of computer-readable instructions. When the computer-readable instructions are executed by one or more processors, it means that the program is running in the computer, thereby causing one or more processors to perform the steps of the network data saving method of any of the above embodiments.

[0161] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0162] In summary, this application automatically saves network data downloaded from the Internet in a delimited data saving format to corresponding files, thereby optimizing the network data saving and management process for the platform.

[0163] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0164] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in the prior art that are similar to those disclosed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0165] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for storing network data, characterized in that, Includes the following steps: Obtain the target data type of one or more delimiter-separated data saving templates, detect delimiter-separated data saving templates with dynamically customizable filenames, and use the dynamically customizable filenames as the data saving filenames of the delimiter-separated data saving templates, wherein the dynamically customizable filenames are data saving filenames that can be edited at any time; Obtain the pre-configuration file name of the delimiter-separated data saving template that does not have the dynamically customized file name, and use the pre-configuration file name as the data saving file name of the delimiter-separated data saving template; A data exporter is generated for each of the delimiters that separates the data storage templates. The target data type and the data storage file name are used as the exported data type and the data storage path, respectively. Each of the data exporters and its corresponding data storage file name are mapped and stored in the exporter pool so that the data exporter can be reused for the network data export and storage processing of the corresponding network data entity. Obtain one or more network data entities downloaded from the target website, and use each of the data exporters in the exporter pool to export the network data corresponding to the target data type of each delimiter-separated data storage template in each network data entity to the delimiter-separated data storage file of the corresponding data storage file name.

2. The network data storage method according to claim 1, characterized in that, The steps of obtaining the target data type of one or more delimiter-separated data storage templates, detecting delimiter-separated data storage templates with dynamically customized filenames, and using the dynamically customized filenames as the data storage filenames for the delimiter-separated data storage templates include: In response to a dynamic custom filename edit event, determine the target delimiter for saving data in the dynamic custom filename edit event; Obtain the dynamically customized filename from the dynamically customized filename editing event, and store the dynamically customized filename in the target delimiter-separated data saving template.

3. The network data storage method according to claim 1, characterized in that, The step of generating a data exporter for each of the delimiters that separates the data storage templates, using the target data storage data type and the data storage file name as the exported data type and data storage path respectively, and mapping and storing each of the data exporters and their corresponding data storage file names in the exporter pool, includes: Retrieves one or more target data types and filenames for saving data in a delimited data saving template; Call the delimiter-separated data exporter generation interface, take each of the target save data types as the export data type, take the data save file name as the data save path, and generate a data exporter that acts on the delimiter-separated data save template; Generate mapping data consisting of the data exporter and the data save file name, and store the mapping data in the exporter pool.

4. The network data storage method according to claim 1, characterized in that, The steps of obtaining one or more web data entities downloaded from a target website include: Download network data from the target website corresponding to the preset target website link; Based on one or more preset network data entity templates, target network data corresponding to one or more preset target data types in each network data is extracted from each network data. Generate one or more network data entities composed of the target network data for each of the network data entity templates.

5. The network data storage method according to claim 1, characterized in that, The steps of obtaining one or more network data entities downloaded from a target website, and using each of the data exporters in the exporter pool to export the network data corresponding to the target data type of each delimiter-separated data storage template in each network data entity to a delimiter-separated data storage file with the corresponding data storage file name, include: Obtain the network data entity and determine the delimiter corresponding to the network data entity to save the data template. Obtain the dynamic custom file name or pre-configuration file name of the delimiter-separated data storage template, and query the data exporter mapped to the dynamic custom file name or the pre-configuration file name in the exporter pool; Determine one or more export data types and data storage paths corresponding to the data exporter, obtain network data corresponding to each export data type from the network data entity, and save each network data to a delimiter-separated data storage file corresponding to the data storage path.

6. The network data storage method according to claim 1, characterized in that, After the step of exporting the network data corresponding to the target data type of each delimiter-separated data storage template in each of the network data entities to the delimiter-separated data storage file with the corresponding data storage file name, the following is included: In response to a data entry command applied to a delimiter-separated data storage file, obtain one or more data entry network data type matching rules corresponding to the data entry command; According to the matching rules for each type of network data type entering the table, the network data corresponding to the target data type entering the table according to the matching rules for each type of network data type entering the table is matched from the data storage file separated by the delimiter. Determine the network data storage table corresponding to the data entry instruction, and store each network data in the storage column position corresponding to the target storage column name in the network data storage table according to the target storage column name matching rule of each network data type.

7. The network data storage method according to claim 1, characterized in that, After the step of exporting the network data corresponding to the target data type of each delimiter-separated data storage template in each of the network data entities to the delimiter-separated data storage file with the corresponding data storage file name, the following is included: In response to a data save file retrieval command pushed by the management terminal, retrieve one or more data save file names contained in the data save file retrieval command; Retrieve the target delimited data file corresponding to the data file name from the delimiter-separated data file library; The data files separated by the delimiters are encapsulated into a data file set, and the data file set is pushed to the management terminal.

8. A network data storage device, characterized in that, include: The dynamic filename acquisition module is used to acquire the target data type of one or more delimiter-separated data saving templates, detect delimiter-separated data saving templates with dynamically customized filenames, and use the dynamically customized filenames as the data saving filenames of the delimiter-separated data saving templates, wherein the dynamically customized filenames are data saving filenames that can be edited at any time; The configuration file name acquisition module is used to acquire the pre-configuration file name of the delimiter-separated data saving template that does not have the dynamically customized file name, and use the pre-configuration file name as the data saving file name of the delimiter-separated data saving template. The data exporter generation module generates a data exporter for each of the delimiter-separated data saving templates, using the target data type and the data saving file name as the export data type and data saving path respectively. The module maps and stores each of the data exporters and their corresponding data saving file names in the exporter pool, so that the data exporters can be reused for the network data export and saving processing of the corresponding network data entities. The data export and save module is used to obtain one or more network data entities downloaded from the target website, and use the data exporters in the exporter pool to export the network data corresponding to the target save data type of each delimiter-separated data save template in each network data entity to the delimiter-separated data save file of the corresponding data save file name.

9. A network data storage device, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Citation Information

Patent Citations

  • Template-based data exporting method and device, medium and equipment

    CN113901098A

  • Data processing method and device, electronic equipment and storage medium

    CN116257501A