Remote data extraction method and device based on Web platform, storage medium and equipment

Abstract

The invention discloses a remote data extraction method and device based on a Web platform, a storage medium and equipment, and the method comprises the steps: collecting related data in a remote data server, and writing the related data into a preset file; the related data comprises basic information of all medium test data files under a target directory; downloading the preset file to the local, and analyzing the preset file into a preset form according to the basic information; and displaying all file data in the preset file in the preset form, so that a tester can extract basic information of a remote medium test data file in a faster and more convenient manner, and analyze the medium test data.

Description

Technical Field

[0001] The present invention relates to the technical field of memory automated testing, and in particular to a remote data extraction method, device, storage medium and equipment based on a Web platform. Background Art

[0002] In order to meet the needs of data analysts for data analysis of data files stored in remote media, data analysis software development usually needs to include remote media test data basic information synchronization, data screening, analysis, and visualization functions.

[0003] In the prior art, since the test data files are large in volume, and the traditional technology needs to acquire data from local files, the process of downloading data files and parsing is slow. At the same time, in terms of data screening, the traditional method is to provide the screening content of each file and let the user screen the files one by one, and then extract the screened data for analysis and visualization. The operation process is relatively cumbersome, and therefore cannot meet the user's needs for rapid screening and acquisition of remote file data. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a remote data extraction method, device, storage medium and equipment based on a Web platform, so that testers can extract and analyze remote file data in a faster and more convenient way.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: A remote data extraction method based on a Web platform, comprising: Collect relevant data from the remote data server and write the relevant data into a preset file; The relevant data includes basic information of all media test data files in the target directory; Downloading the preset file to a local computer, and parsing the preset file into a preset format according to the basic information; All file data in the preset file are displayed in the preset format.

[0006] In order to solve the above technical problems, another technical solution adopted by the present invention is: A remote data extraction device based on a Web platform, comprising: A collection module is used to collect relevant data in a remote data server and write the relevant data into a preset file; the relevant data includes basic information of all media test data files in a target directory; A download module, used for downloading the preset file to a local computer and parsing the preset file into a preset format according to the basic information; The display module is used to display all file data in the preset file in the preset form.

[0007] In order to solve the above technical problems, another technical solution adopted by the present invention is: A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned remote data extraction method based on a Web platform.

[0008] In order to solve the above technical problems, another technical solution adopted by the present invention is: An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the above-mentioned remote data extraction method based on a Web platform is implemented.

[0009] The beneficial effect of the present invention lies in: by collecting relevant data from a remote data server and writing it into a preset file, wherein the relevant data includes basic information of all media test data files under a target directory, the preset file is downloaded to a local computer and parsed into a preset format based on the basic information, so that when performing data analysis, the parsed media test data can be directly obtained from the local computer and displayed, thereby improving data acquisition efficiency; thereby enabling testers to extract basic information of remote media test data files and analyze the media test data in a faster and more convenient manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of a remote data extraction method based on a Web platform in an embodiment of the present invention; Figure 2 This is a step of acquiring remote medium test data in a remote data extraction method based on a Web platform in an embodiment of the present invention; Figure 3 It is an operation interface for acquiring remote medium test data in a remote data extraction method based on a Web platform in an embodiment of the present invention; Figure 4 This is a step of screening medium test data in a remote data extraction method based on a Web platform in an embodiment of the present invention; Figure 5 It is a screening operation interface for medium test data of a remote data extraction method based on a Web platform in an embodiment of the present invention; Figure 6 Another screening operation interface for medium test data in a remote data extraction method based on a Web platform in an embodiment of the present invention; Figure 7An operation interface for a user to select a target file in a remote data extraction method based on a Web platform in an embodiment of the present invention; Figure 8 It is an operation interface for completing the medium test data screening in a remote data extraction method based on a Web platform in an embodiment of the present invention; Fig. 9 Schematic diagram of the structure of a remote data extraction device based on a Web platform in an embodiment of the present invention; Fig.10 FIG. 4 is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0011] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0012] Please refer to Figure 1 , a remote data extraction method based on a Web platform, comprising: Collect relevant data from the remote data server and write the relevant data into a preset file; The relevant data includes basic information of all media test data files in the target directory; Downloading the preset file to a local computer, and parsing the preset file into a preset format according to the basic information; All file data in the preset file are displayed in the preset format.

[0013] From the above description, it can be seen that the beneficial effect of the present invention is that: by collecting relevant data from the remote data server and writing it into a preset file, and the relevant data includes the basic information of all media test data files under the target directory, the preset file is downloaded to the local and parsed into a preset format based on the basic information, so that when performing data analysis, the parsed media test data can be directly obtained from the local and displayed, thereby improving data acquisition efficiency; thereby enabling testers to extract the basic information of remote media test data files and analyze the media test data in a faster and more convenient way.

[0014] Furthermore, the basic information includes at least one of a test data type, a test particle type, a test type, a test data screening condition column, and a test data data source column.

[0015] From the above description, it can be seen that by obtaining data related to media testing such as test data type, test particle type, test type, test data screening condition column and test data data source column, a data basis is provided for subsequent media test analysis on the Web platform.

[0016] Further, parsing the preset file into a preset format according to the basic information includes: Parsing the preset file into the form of data type / particle type / test type / file flag to obtain parsed data; The parsed data, the test data screening condition column and the test data data source column are saved in a database.

[0017] From the above description, it can be seen that by parsing the data into the form of data type / granule type / test type / file flag, the relevant data can be quickly obtained based on the preset form when the data is subsequently queried or extracted through the database; at the same time, the test data filtering condition column and the test data data source column are saved to the database, so that the data can be effectively filtered through the test data filtering condition column later.

[0018] Furthermore, after displaying all file data in the preset file in the preset form, the method further includes: Acquire all target file data selected by the user, and mark one piece of data of the same type in all the target file data for the user to use as a screening condition; All the filtering conditions selected by the user are obtained, and target data is obtained according to all the filtering conditions and all the target data is visualized.

[0019] From the above description, it can be seen that when filtering data, by marking a piece of data of the same type in the target file data for the user to use as a filtering condition, the user is prevented from having to specify conditions for multiple columns of data under multiple files each time, thereby reducing the complexity of the user's operation of the filtering conditions and improving the efficiency of data filtering; and after completing the data filtering, visualization is performed based on all the filtering conditions and the target data.

[0020] Furthermore, the remote data server is installed with a script; The relevant data collected in the remote data server includes: The script is used to collect basic information of all data files in the target directory to obtain the relevant data.

[0021] It can be seen from the above description that by installing a script on the remote data server and collecting relevant data through the script and writing it into a preset file, the efficiency of obtaining remote data can be improved.

[0022] Furthermore, acquiring target data according to all the screening conditions and visualizing all the target data includes: Arrange and combine all the screening conditions in a preset manner to form at least two screening combinations, and perform data query according to all the screening combinations; Associating the queried target data with the corresponding screening condition; A table header is generated according to all the filtering conditions, and the target data corresponding to the filtering conditions are arranged according to the table header to obtain a visual data table.

[0023] From the above description, it can be seen that by arranging and combining all the filtering conditions in a preset manner to form at least two filtering combinations, all the data required for analysis can be filtered out; at the same time, by using the filtering conditions as table headers and the target data as column data or row data corresponding to the filtering conditions, a visual data table is obtained; thus, the visual data table enables users to analyze the data more intuitively.

[0024] Furthermore, after displaying all file data in the preset file in the preset form, the method further includes: Obtain a user pointer movement instruction, determine whether the pointer is located on any of the filtering conditions, and if so, obtain and display the directory information corresponding to the filtering condition.

[0025] From the above description, it can be seen that by obtaining the user pointer movement instruction and displaying the directory information corresponding to the data when the pointer is located on any of the filtering conditions, when the user controls the mouse to move to any data, it can be displayed in which files the data exists.

[0026] Furthermore, after marking one piece of data of the same type in all the target file data for the user to use as a screening condition, the method further includes: Marking the screening condition generated by at least two target file data in a first preset manner; Marking the screening condition mark generated by one of the target file data in a second preset manner; The first preset manner and the second preset manner are different marking manners.

[0027] It can be seen from the above description that by marking the filter conditions containing different numbers of items in the first preset manner and the second preset manner respectively, the user can more intuitively know that the filter conditions contain the same data of multiple target file data.

[0028] Another embodiment of the present invention provides a remote data extraction device based on a Web platform, comprising: The collection module is used to collect relevant data from the remote data server and write the relevant data into a preset file; the relevant data includes the basic information of all media test data files in the target directory. A download module, used for downloading the preset file to a local computer and parsing the preset file into a preset format according to the basic information; The display module is used to display all file data in the preset file in the preset form.

[0029] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the steps in the above-mentioned remote data extraction method based on a Web platform.

[0030] Another embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements each step of the above-mentioned remote data extraction method based on a Web platform when executing the computer program.

[0031] The Web-based remote data extraction method, device, storage medium, and equipment provided by the present invention can be applied to the data extraction and analysis scenario of remote medium particle test data, which is described below through specific implementation methods: Embodiment 1 Please refer to Figure 1 , a remote data extraction method based on a Web platform, comprising steps S1-S5, specifically as follows: Please refer to Figure 2 , steps S1-S2 optimize the access and screening efficiency of remote media test data: S1. Collect relevant data in the remote data server and write the relevant data into a preset file; wherein, the remote data server is installed with a script; when collecting relevant data in the remote data server, the basic information of all data files under the target directory is collected through the script to obtain relevant data; for example, if the data file is a medium test data file, the relevant data includes the basic information of all medium test data files under the target directory, including basic information such as test data type, test particle type, test type, test data screening condition column and test data data source column, and the above-mentioned relevant data is written into a json file, and the integrity of data transmission can be guaranteed through the json file.

[0032] S2. Download the preset file to the local computer, and parse the preset file into a preset format according to the basic information. Specifically, the json file can be downloaded to the local computer remotely based on the Web service and the data can be parsed. The file data can be parsed into the form of data type / granule type / test type / file flag, and saved together with the filter conditions and data source column data into the database, for example, the data can be stored in a MongoDB database, or in other types of databases, so that users can access the basic information data faster.

[0033] In the actual operation process, Figure 3 As shown, after the user clicks the remote data option on the front-end Web service, the data collection script of the data storage service is triggered to execute, and the collected data is written to a json file; in other optional implementations, the Web service can also be controlled to automatically trigger the execution of the data collection script of the data storage service by setting a time interval; and by detecting whether the data in the remote data server is updated, when it is detected that the relevant data is updated, the data collection script of the data storage service is triggered to execute, so that the system data can be kept up to date at all times.

[0034] Please refer to Figure 4 , steps S3-S5 improve the medium test data screening analysis and visualization process: S3. Display all file data in the preset file in a preset form; for example, the front end displays all files through the Web for users to choose. For example, display in the form of a file tree, with the data type as the first layer, the particle type as the second layer, the test type as the third layer, and the file mark (including file name, test characteristics, etc.) as the fourth layer.

[0035] S4, obtain all target file data selected by the user, and mark a piece of data of the same type in all target file data for the user to use as a filtering condition; wherein, after the user selects, the system will extract all the filtering conditions of the file selected by the user, and mark the filtering conditions generated by at least two target file data in a first preset manner, and mark the filtering condition mark generated by one target file data in a second preset manner, the first preset manner and the second preset manner are different marking methods, thereby greatly improving the user's speed of filtering multi-file medium data; the screening can be based on the following judgments: 1. Whether it is a directory layer node; 2. Whether it is a duplicate data node; 3. In which files does the current value exist; and the filter value represented by the current node is indicated by a label. For example, the same value is marked in red, and different values ​​are marked in black; thereby, the values ​​corresponding to the filter column with the same key are put together for the user to select, and finally the filter conditions selected by the user are collected. At the same time, in the process of the user operating the mouse, the user pointer movement instruction is obtained, and it is determined whether the position of the pointer is on any filter condition. If so, the directory information corresponding to the filter condition is obtained and displayed, so that when the mouse moves to the data, it is displayed in which files the data exists.

[0036] Please refer to Figure 5 As shown, the left side of the interface is a file tree part that can be selected multiple times, including multiple media data files, that is, this embodiment processes multiple media data files, and multiple corresponding files can be selected when selecting. Figure 6When the user selects a file, the interface will display all corresponding filtering conditions for the currently selected file, and the user can make further selections based on needs; when there is a selected file in the file tree, all file data will be integrated and labeled, including the corresponding file name, path, whether it is repeated, whether it is a column name and other annotation information; that is, after processing the remote media file data source, the data source is processed into the format that the media needs to present; and when displaying, it is directly presented as the corresponding directory level and the file name is processed as a test feature to ensure that the media personnel can quickly lock the required file, or directly display the file name when the mouse moves to the corresponding file layer. At the same time, the conditions are effective for all currently selected files, and the system will mark the same data according to the file selected by the user. For example, the same data in different files is marked in red or bold, and different values ​​are black or not bold. Figure 6 As shown, the data 300 and 62 corresponding to the target item HB (marked in bold), and the data 153 and 775 corresponding to the target item SB (marked in bold) are integrated data. Figure 7 As shown, when the user moves the mouse to data 0 (marked in bold), the directory information corresponding to data 0 will be displayed; wherein, data 0 is contained in two directories, that is, there are two target files that respectively contain relevant information about data 0.

[0037] S5. Obtain all the filtering conditions selected by the user, obtain the target data according to all the filtering conditions, and visualize all the target data. Specifically: S51. Arrange and combine all the screening conditions in a preset manner to form at least two screening combinations, and perform data query based on all the screening combinations; for example, the server may arrange and combine the conditions selected by the user in order to form multiple conditions for querying data.

[0038] S52, concatenate the queried target data with the corresponding filtering conditions to obtain feedback data, and perform data visualization analysis based on the feedback data; that is, concatenate the filtering conditions with the corresponding data when returning the data and return the corresponding data, so that the user can analyze and visualize the returned conditional data. When performing visualization, a table header can be generated based on all filtering conditions, and the target data corresponding to the filtering conditions can be arranged according to the table header to obtain a visualized data table. Figure 8As shown, when the user selects the corresponding filter condition and clicks OK, the front-end Web displays the data that the current file meets the current condition, which can be displayed in the form of table data, with the condition column on the left and the data of multiple files under the condition on the right. At the same time, it is also possible to analyze and visualize the exported data, such as analyzing and visualizing according to the filter condition, and supporting visualization in the form of supporting bar charts, scatter plots, line charts, etc. For example, in actual operation: the medium data filter condition selected by the user, such as Nand and FrameSize (representing the type of cutoff particles and counting the number of error bits in 1K or 4K as a data unit), then filter each selected file according to the condition, when a column has multiple values, sequentially arrange and combine to construct multiple conditions and filter them one by one, and finally return the filtered data and display it in a table or further visualize it according to the type of graph, such as generating supporting bar charts, scatter plots, line charts, and finally return to the page and draw a picture for display.

[0039] Embodiment 2 Please refer to Fig. 9 , a remote data extraction device based on a Web platform, comprising: A collection module is used to collect relevant data from a remote data server and write the relevant data into a preset file; the relevant data includes basic information of all media test data files in a target directory; The download module is used to download the preset file to the local computer and parse the preset file into a preset format according to the basic information; The display module is used to display all file data in a preset file in a preset form.

[0040] Embodiment 3 A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the various steps of a remote data extraction method based on a Web platform as described in Embodiment 1.

[0041] Embodiment 4 Please refer to Fig.10 An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of a remote data extraction method based on a Web platform as described in Example 1 is implemented.

[0042] In summary, the remote data extraction method, device, storage medium and equipment based on the Web platform provided by the present invention realize data extraction and analysis of remote medium particle test data based on the Web platform, by collecting relevant data in the remote data server and writing it into a preset file, and downloading the preset file to the local and parsing it into a preset form, so that when performing data analysis, the parsed data can be directly obtained from the local and displayed, thereby improving data acquisition efficiency; and when filtering data, by marking a piece of data of the same type in the target file data for the user to use as a filtering condition, it is avoided that the user needs to specify conditions for multiple columns of data under multiple files each time, thereby reducing the complexity of the user's operation filtering conditions and improving data filtering efficiency; and after completing data filtering, visualization is performed according to all filtering conditions and target data, saving the time of media analysts for manual extraction, analysis, and visualization of remote media test data; thereby allowing testers to extract, analyze and visualize remote file data in a faster and more convenient way.

[0043] In the above embodiments provided in the present application, it should be understood that the disclosed methods, devices, computer-readable storage media, and electronic devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple components or modules can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or components or modules, which can be electrical, mechanical or other forms.

[0044] The components described as separate components may or may not be physically separated, and the components shown as components may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the components may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0045] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each component may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0046] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0047] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0048] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0049] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A remote data extraction method based on a Web platform, characterized in that: include: Collect relevant data from the remote data server and write the relevant data into a preset file; The relevant data includes basic information of all media test data files in the target directory; Downloading the preset file to a local computer, and parsing the preset file into a preset format according to the basic information; All file data in the preset file are displayed in the preset format.

2. A remote data extraction method based on a Web platform according to claim 1, characterized in that: The basic information includes at least one of a test data type, a test particle type, a test type, a test data screening condition column, and a test data data source column.

3. A remote data extraction method based on a Web platform according to claim 2, characterized in that: The parsing the preset file into a preset format according to the basic information includes: Parsing the preset file into the form of data type / particle type / test type / file flag to obtain parsed data; The parsed data, the test data screening condition column and the test data data source column are saved in a database.

4. A remote data extraction method based on a Web platform according to claim 1, characterized in that: After displaying all the file data in the preset file in the preset form, the following step is further included: Acquire all target file data selected by the user, and mark one piece of data of the same type in all the target file data for the user to use as a screening condition; All the filtering conditions selected by the user are obtained, and target data is obtained according to all the filtering conditions and all the target data is visualized.

5. A remote data extraction method based on a Web platform according to claim 4, characterized in that: The acquiring of target data according to all the screening conditions and the visualization of all the target data include: Arrange and combine all the screening conditions in a preset manner to form at least two screening combinations, and perform data query according to all the screening combinations; Associating the queried target data with the corresponding screening condition; A table header is generated according to all the filtering conditions, and the target data associated with the filtering conditions are arranged according to the table header to obtain a visual data table.

6. A remote data extraction method based on a Web platform according to claim 4, characterized in that: After displaying all the file data in the preset file in the preset form, the following step is further included: Obtain a user pointer movement instruction, determine whether the pointer is located on any of the filtering conditions, and if so, obtain and display the directory information corresponding to the filtering condition.

7. A remote data extraction method based on a Web platform according to claim 4, characterized in that: After marking a piece of data of the same type in all the target file data for the user to use as a screening condition, the method further includes: Marking the screening condition generated by at least two target file data in a first preset manner; Marking the screening condition mark generated by one of the target file data in a second preset manner; The first preset manner and the second preset manner are different marking manners.

8. A remote data extraction device based on a Web platform, characterized in that: include: A collection module, used to collect relevant data from a remote data server and write the relevant data into a preset file; The relevant data includes basic information of all media test data files in the target directory; A download module, used for downloading the preset file to a local computer and parsing the preset file into a preset format according to the basic information; The display module is used to display all file data in the preset file in the preset form.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of a remote data extraction method based on a Web platform as described in any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the processor implements each step of the remote data extraction method based on a Web platform as described in any one of claims 1 to 7.

Images