Unstructured data acquisition and management method, system and device and storage medium

Abstract

The invention discloses an unstructured data acquisition and management method, system and device and a storage medium, belongs to the technical field of data processing, and solves the technical problem that unstructured data is difficult to process efficiently in the prior art. The method comprises the following steps of: pulling a file from an OSS file server through a Dolphincheduler component, storing the file in a local server, and then uploading the file to an HDFS (Hadoop Distributed File System); performing user and user group read-write authority management on a file path of the HDFS through a ranger authority management system, and writing file metadata information into a hudi table; and publishing the collected unstructured data to an asset portal for a user to apply for downloading and use. According to the method, customized adaptation of unstructured data acquisition and management is carried out through Dolphincheduler, including automatic acquisition of unstructured data, authority management, data integration and release and use, so that flexible processing and management of different data types in a big data environment are met, and the unstructured data is efficiently processed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a method, system, device and storage medium for collecting and managing unstructured data. Background Art

[0002] Most current data integration is aimed at synchronizing structured data, that is, data that can be clearly represented in the form of tables or databases. However, many data are unstructured, such as files, images, audio and video, and there are still some challenges in the collection and management of these unstructured data.

[0003] Existing technologies have deficiencies in the collection, integration, storage and authority management of unstructured data, making it difficult to process unstructured data efficiently. Summary of the invention

[0004] The technical problem to be solved by the present invention is to address the above-mentioned deficiencies in the prior art. One purpose of the present invention is to provide a method for collecting and managing unstructured data, which can efficiently process unstructured data.

[0005] A second objective of the present invention is to provide an unstructured data collection and management system.

[0006] A third object of the present invention is to provide a computer device.

[0007] A fourth objective of the present invention is to provide a computer storage medium.

[0008] In order to achieve the above-mentioned purpose one, the present invention provides an unstructured data collection and management method, which pulls files from the OSS file server through the Dolphinscheduler component and stores them in the local server, and then uploads them to the HDFS system; manages the read and write permissions of users and user groups on the file path of HDFS through the ranger permission management system, and writes the file metadata information into the hudi table; publishes the collected unstructured data to the asset portal for users to apply for download and use.

[0009] As a further improvement, the following steps are specifically included:

[0010] Step 1. Configure the OSS data source for data connection, obtain the OSS access address, accessKey, accessSecret, and test the connection to ensure that the file can be pulled in subsequent steps;

[0011] Step 2. Configure the HDFS write path. You can create a folder on the page. First, call the ranger query to determine whether the currently logged-in user has the operation permission for the corresponding directory. If the user has permission, save the file metadata to the hudi table and create a folder in HDFS synchronously for storing the collected unstructured data later.

[0012] Step 3. Create an unstructured collection task, select the OSS data source configured in step 1, obtain the bucket to be collected, select the files to be collected according to the read rule, select the same-name file processing rule, select the HDFS write path configured in step 2, set the timing period, and confirm that the task configuration is completed;

[0013] Step 4. Add OSS collection components and file upload HDFS components. When the Dolphinscheduler task is started, the worker service will execute the OSS collection components and file upload HDFS components in sequence. The OSS collection component is responsible for calling the OSS interface to download files according to the configuration information, setting the data volume of each shard to download, and preventing memory overflow. After the OSS collection component downloads the file to the local directory of the server, it organizes the file information and some business information corresponding to the local directory address, and passes it to the next executed file upload HDFS component. At this point, the execution of the OSS collection component ends.

[0014] Step 5. On the content management page, query the file metadata information of the hudi table and display it in pages. Internal managers can use buttons to operate files, tag files, and download files. The release button can be used to publish files to the asset portal for other business personnel to apply for download and use. The release process needs to go through the approval process;

[0015] Step 6. All the document information that has been approved for release is displayed on the asset portal. Business personnel can apply for the required documents, which will also go through the approval process. After approval, they can be directly downloaded and used.

[0016] Furthermore, in step 4, the data volume of one shard is 20M.

[0017] Furthermore, in step 4, Dolphinscheduler starts to execute the file upload HDFS component, parses the parameters passed by the previous OSS collection component, finds the file currently to be uploaded to HDFS locally on the server, queries HDFS for the existence of a file with the same name according to the HDFS write path configured on the page, and then determines the upload strategy according to the renaming method to execute the file upload;

[0018] After the file is uploaded successfully, the file metadata information is written into the hudi table. The metadata information records the complete storage path, file size, collection personnel and collection time of the file, which makes it convenient for subsequent data developers to read the hudi table for data cleaning ETL process.

[0019] In order to achieve the above-mentioned second objective, the present invention provides an unstructured data collection and management system, comprising:

[0020] Configure the data source module to configure the OSS data source for data connection, obtain the OSS access address, accessKey, accessSecret, and test the link to ensure that the subsequent steps can pull the file;

[0021] Configure the write path module, which is used to configure the HDFS write path. You can create a folder through the page. First, call the ranger query to determine whether the currently logged-in user has the operation permission for the corresponding directory. If the user has the permission, save the file metadata to the hudi table and create a folder in HDFS synchronously for storing the collected unstructured data later.

[0022] Create a task module to create an unstructured collection task. Select the OSS data source configured in step 1, obtain the bucket to be collected, select the files to be collected according to the reading rules, select the processing rules for files with the same name, select the HDFS write path configured in step 2, set the timing period, and confirm that the task configuration is completed.

[0023] Added component modules to add OSS collection components and file upload HDFS components;

[0024] When the Dolphinscheduler task is started, the worker service will execute the OSS collection component and the file upload HDFS component in sequence. The OSS collection component is responsible for calling the OSS interface to download files according to the configuration information, setting the data volume of each shard to download, and preventing memory overflow. After the OSS collection component downloads the file to the local directory of the server, it organizes the file information and some business information corresponding to the local directory address, and passes it to the next executed file upload HDFS component. At this point, the execution of the OSS collection component ends;

[0025] Dolphinscheduler starts executing the file upload HDFS component, parses the parameters passed by the previous OSS collection component, finds the file that needs to be uploaded to HDFS locally on the server, checks whether there is a file with the same name in HDFS according to the HDFS write path configured on the page, and then determines the upload strategy based on the renaming method to execute the file upload.

[0026] The content management module is used to query the file metadata information of the hudi table on the content management page and display it in pages. Internal managers can operate files, tag files, and download files through buttons. The release button can be used to publish files to the asset portal for other business personnel to apply for download and use. The release process needs to go through the approval process;

[0027] The display module is used to display all the document information that has been approved for release on the asset portal. Business personnel can apply for the required documents, which will also go through the approval process and can be directly downloaded and used after approval.

[0028] In order to achieve the third objective above, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned unstructured data collection and management method when executing the computer program.

[0029] In order to achieve the fourth objective above, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned unstructured data collection and management method is implemented.

[0030] Beneficial Effects

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] The present invention performs customized adaptation of unstructured data collection and management through Dolphin Scheduler, including automatic collection of unstructured data, authority management, data integration, and publishing and use, so as to meet the flexible processing and management of different data types in a big data environment, improve the integration efficiency of unstructured data, improve the integration and storage methods of unstructured data, and strengthen the authority management and control of unstructured data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the present invention;

[0034] Figure 2 Diagram of adding OSS data source type for data connection;

[0035] Figure 3 This is a schematic diagram of file path management;

[0036] Figure 4 To create a schematic diagram of the unstructured acquisition task.

[0037] Figure 5 This is a schematic diagram of the newly added OSS collection component and file upload HDFS component. DETAILED DESCRIPTION

[0038] The present invention will be further described below with reference to the specific embodiments in the accompanying drawings.

[0039] See also Figure 1 to Figure 5 , a method for collecting and managing unstructured data, pulls files from the OSS file server through the Dolphinscheduler component and stores them on the local server, and then uploads them to the HDFS system; manages the read and write permissions of users and user groups on the HDFS file path through the ranger permission management system, writes the file metadata information into the hudi table; publishes the collected unstructured data to the asset portal for users to apply for download and use. Compared with relational databases such as MySQL, hudi can maintain good performance on large amounts of data.

[0040] The specific steps include steps 1 to 6:

[0041] Step 1. Figure 2 As shown in the figure, configure the OSS data source for data connection, obtain the OSS access address, accessKey, accessSecret, and test the link to ensure that the subsequent steps can pull the file.

[0042] Step 2. Figure 3 As shown, configure the HDFS write path. You can create a folder through the page. First call the ranger query to determine whether the currently logged-in user has the operation permission for the corresponding directory. If the user has permission, save the file metadata to the hudi table and synchronously create a folder in HDFS for storing the collected unstructured data later.

[0043] Step 3. Figure 4 As shown in the figure, create an unstructured collection task, select the OSS data source configured in step 1, obtain the bucket to be collected, select the files to be collected according to the reading rules, select the processing rules for files with the same name, select the HDFS write path configured in step 2, set the timing period, and confirm that the task configuration is completed.

[0044] Step 4. Figure 5 As shown in the figure, OSS collection components and file upload HDFS components are added.

[0045] When the Dolphinscheduler task is started, the worker service will execute the OSS collection component and the file upload HDFS component in sequence. The OSS collection component is responsible for calling the OSS interface to download files according to the configured information, setting the data volume of a shard to download a shard, and preventing memory overflow. In this embodiment, the data volume of a shard is 20M.

[0046] After the OSS collection component downloads the file to the local directory of the server, it organizes the file information and some business information corresponding to the local directory address and passes it to the next executed file upload HDFS component. At this point, the execution of the OSS collection component ends.

[0047] Dolphinscheduler starts executing the file upload HDFS component, parses the parameters passed by the previous OSS collection component, finds the file that needs to be uploaded to HDFS locally on the server, queries HDFS for files with the same name based on the HDFS write path configured on the page, determines the upload strategy based on the renaming method, and executes the file upload.

[0048] After the file is uploaded successfully, the file metadata information is written into the hudi table. The metadata information records the complete storage path, file size, collection personnel and collection time of the file, which is convenient for subsequent data developers (people working in the field of data development) to read the hudi table for data cleaning ETL process (ETL process includes three main steps: data extraction (Extract), data transformation (Transform) and data loading (Load)).

[0049] Step 5. On the content management page, query the file metadata information of the hudi table and display it in pages. Internal managers can use buttons to operate files, tag files, and download files. The publish button can be used to publish files to the asset portal for other business personnel to apply for download and use. The release process is required.

[0050] Step 6. All the document information that has been approved for release is displayed on the asset portal. Business personnel can apply for the required documents, which will also go through the approval process. After approval, they can be directly downloaded and used.

[0051] An unstructured data collection and management system, comprising:

[0052] Configure the data source module to configure the OSS data source for data connection, obtain the OSS access address, accessKey, accessSecret, and test the link to ensure that the subsequent steps can pull the file;

[0053] Configure the write path module, which is used to configure the HDFS write path. You can create a folder through the page. First, call the ranger query to determine whether the currently logged-in user has the operation permission for the corresponding directory. If the user has the permission, save the file metadata to the hudi table and create a folder in HDFS synchronously for storing the collected unstructured data later.

[0054] Create a task module to create an unstructured collection task. Select the OSS data source configured in step 1, obtain the bucket to be collected, select the files to be collected according to the reading rules, select the processing rules for files with the same name, select the HDFS write path configured in step 2, set the timing period, and confirm that the task configuration is completed.

[0055] Added component modules to add OSS collection components and file upload HDFS components;

[0056] When the Dolphinscheduler task is started, the worker service will execute the OSS collection component and the file upload HDFS component in sequence. The OSS collection component is responsible for calling the OSS interface to download files according to the configuration information, setting the data volume of each shard to download, and preventing memory overflow. After the OSS collection component downloads the file to the local directory of the server, it organizes the file information and some business information corresponding to the local directory address, and passes it to the next executed file upload HDFS component. At this point, the execution of the OSS collection component ends;

[0057] Dolphinscheduler starts executing the file upload HDFS component, parses the parameters passed by the previous OSS collection component, finds the file that needs to be uploaded to HDFS locally on the server, checks whether there is a file with the same name in HDFS according to the HDFS write path configured on the page, and then determines the upload strategy based on the renaming method to execute the file upload.

[0058] The content management module is used to query the file metadata information of the hudi table on the content management page and display it in pages. Internal managers can operate files, tag files, and download files through buttons. The release button can be used to publish files to the asset portal for other business personnel to apply for download and use. The release process needs to go through the approval process;

[0059] The display module is used to display all the document information that has been approved for release on the asset portal. Business personnel can apply for the required documents, which will also go through the approval process and can be directly downloaded and used after approval.

[0060] A computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the above-mentioned unstructured data collection and management method is implemented.

[0061] A computer-readable storage medium stores a computer program, which implements the above-mentioned unstructured data collection and management method when executed by a processor.

[0062] The above are only preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the structure of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A method for collecting and managing unstructured data, characterized in that: The Dolphinscheduler component is used to pull files from the OSS file server and store them on the local server, and then upload them to the HDFS system. The ranger permission management system is used to manage the read and write permissions of users and user groups on the HDFS file path, and the file metadata information is written to the hudi table. The collected unstructured data is published to the asset portal for users to apply for download and use.

2. The unstructured data collection and management method according to claim 1, characterized in that: The specific steps include: Step 1. Configure the OSS data source for data connection, obtain the OSS access address, accessKey, accessSecret, and test the connection to ensure that the file can be pulled in subsequent steps; Step 2. Configure the HDFS write path. You can create a folder on the page. First, call the ranger query to determine whether the currently logged-in user has the operation permission for the corresponding directory. If the user has permission, save the file metadata to the hudi table and create a folder in HDFS synchronously for storing the collected unstructured data later. Step 3. Create an unstructured collection task, select the OSS data source configured in step 1, obtain the bucket to be collected, select the files to be collected according to the read rule, select the same-name file processing rule, select the HDFS write path configured in step 2, set the timing period, and confirm that the task configuration is completed; Step 4. Add OSS collection components and file upload HDFS components. When the Dolphinscheduler task is started, the worker service will execute the OSS collection components and file upload HDFS components in sequence. The OSS collection component is responsible for calling the OSS interface to download files according to the configuration information, setting the data volume of each shard to download, and preventing memory overflow. After the OSS collection component downloads the file to the local directory of the server, it organizes the file information and some business information corresponding to the local directory address, and passes it to the next executed file upload HDFS component. At this point, the execution of the OSS collection component ends. Step 5. On the content management page, query the file metadata information of the hudi table and display it in pages. Internal managers can use buttons to operate files, tag files, and download files. The release button can be used to publish files to the asset portal for other business personnel to apply for download and use. The release process needs to go through the approval process; Step 6. All the document information that has been approved for release is displayed on the asset portal. Business personnel can apply for the required documents, which will also go through the approval process. After approval, they can be directly downloaded and used.

3. The unstructured data collection and management method according to claim 1, characterized in that: In step 4, the data volume of a shard is 20M.

4. The unstructured data collection and management method according to claim 1, characterized in that: In step 4, Dolphinscheduler starts to execute the file upload HDFS component, parses the parameters passed by the previous OSS collection component, finds the file that needs to be uploaded to HDFS locally on the server, checks whether there is a file with the same name in HDFS according to the HDFS write path configured on the page, and then determines the upload strategy according to the renaming method to execute the file upload; After the file is uploaded successfully, the file metadata information is written into the hudi table. The metadata information records the complete storage path, file size, collection personnel and collection time of the file, which makes it convenient for subsequent data developers to read the hudi table for data cleaning ETL process.

5. An unstructured data collection and management system, characterized in that: include: Configure the data source module to configure the OSS data source for data connection, obtain the OSS access address, accessKey, accessSecret, and test the link to ensure that the subsequent steps can pull the file; Configure the write path module, which is used to configure the HDFS write path. You can create a folder through the page. First, call the ranger query to determine whether the currently logged-in user has the operation permission for the corresponding directory. If the user has the permission, save the file metadata to the hudi table and create a folder in HDFS synchronously for storing the collected unstructured data later. Create a task module to create an unstructured collection task. Select the OSS data source configured in step 1, obtain the bucket to be collected, select the files to be collected according to the reading rules, select the processing rules for files with the same name, select the HDFS write path configured in step 2, set the timing period, and confirm that the task configuration is completed. Add component modules to add OSS collection components and file upload HDFS components. When the Dolphinscheduler task is started, the worker service will execute the OSS collection components and file upload HDFS components in sequence. The OSS collection component is responsible for calling the OSS interface to download files according to the configuration information, setting the data volume of each shard to download, and preventing memory overflow. After the OSS collection component downloads the file to the local directory of the server, it organizes the file information and some business information corresponding to the local directory address, and passes it to the next executed file upload HDFS component. At this point, the execution of the OSS collection component ends. Dolphinscheduler starts executing the file upload HDFS component, parses the parameters passed by the previous OSS collection component, finds the file that needs to be uploaded to HDFS locally on the server, checks whether there is a file with the same name in HDFS according to the HDFS write path configured on the page, and then determines the upload strategy based on the renaming method to execute the file upload. The content management module is used to query the file metadata information of the hudi table on the content management page and display it in pages. Internal managers can operate files, tag files, and download files through buttons. The release button can be used to publish files to the asset portal for other business personnel to apply for download and use. The release process needs to go through the approval process; The display module is used to display all the document information that has been approved for release on the asset portal. Business personnel can apply for the required documents, which will also go through the approval process and can be directly downloaded and used after approval.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, an unstructured data collection and management method as described in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, an unstructured data collection and management method as described in any one of claims 1 to 4 is implemented.

Images