Distributed data processing method and device, electronic equipment and program product

Abstract

The invention belongs to the technical field of big data, and provides a distributed data processing method and device, electronic equipment and a program product, and the method comprises the steps: determining a data query request as a structured data query request or an unstructured data query request in response to the data query request received by a distributed file Hadoop platform; unstructured data stored in a Hadoop platform is queried based on the unstructured data query request through a standard unified API interface, the Hadoop platform is obtained by expanding a distributed file system HDFS based on an open source file system JuiceFS, and the standard unified API interface is a presentation layer state transition RESTful interface constructed or provided by the JuiceFS; querying the structured data stored in the Hadoop platform based on the structured data query request through a user-defined programming API interface, wherein the user-defined programming API interface is realized based on an open source programming language; and returning a query result generated based on the structured data or the unstructured data in a predetermined data format.

Description

Technical Field

[0001] The present application belongs to the field of big data technology, and more specifically, to a distributed data processing method, device, electronic device and program product. Background Art

[0002] With the rapid development of big data technology, enterprises are facing more and more challenges in data collection, storage, processing and analysis. The data types of big data platforms are diverse, including both structured data and unstructured data. However, as the amount of data continues to grow, the inability to access this data efficiently and flexibly has become a major bottleneck facing enterprises.

[0003] In addition, existing big data platforms have many limitations in data sharing. For example, sharing structured data usually requires cumbersome SQL queries, and unstructured data is difficult for ordinary users to quickly obtain because it is stored in complex file systems. In addition, the way data is displayed is relatively simple and lacks flexible customization functions, making it difficult to meet the needs of different business scenarios. For non-technical users, these problems have greatly increased the cost of data sharing and reduced work efficiency. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a distributed data processing method, device, electronic device and program product, aiming to solve the current technical problem of being unable to efficiently and flexibly access data in a big data platform.

[0005] To achieve the above object, according to a first aspect of the present application, a distributed data processing method is provided, the method comprising:

[0006] In response to a data query request received by the distributed file Hadoop platform, determining whether the data query request is a structured data query request or an unstructured data query request;

[0007] If the data query request is an unstructured data query request, query the unstructured data stored in the Hadoop platform based on the unstructured data query request through a standard unified API interface, wherein the Hadoop platform is obtained by extending the distributed file system HDFS based on the open source file system JuiceFS, and the standard unified API interface is a presentation layer state transfer RESTful interface constructed or provided by JuiceFS;

[0008] If the data query request is a structured data query request, querying the structured data stored in the Hadoop platform based on the structured data query request through a custom programming API interface, wherein the custom programming API interface is implemented based on an open source programming language;

[0009] Return the query result generated based on the structured data or the unstructured data in a predetermined data format, wherein the predetermined data format includes: JSON data format.

[0010] Optionally, in a possible implementation manner of the first aspect, querying the structured data stored in the Hadoop platform based on the structured data query request through a custom programming API interface includes:

[0011] Parsing the structured data query request through a custom programming API interface to obtain query parameters and SQL statement placeholders in the structured data query request;

[0012] Send the query parameters and the SQL statement placeholder to a data query Hive driver module;

[0013] By means of the Hive driver module, the query parameters are converted into a structured query SQL statement according to the SQL statement placeholder;

[0014] The SQL statement is used to query the data warehouse Hive tool of the Hadoop platform to obtain the structured data stored in the data warehouse Hive tool.

[0015] Optionally, in a possible implementation manner of the first aspect, the using the SQL statement to query the data warehouse Hive tool of the Hadoop platform to query and obtain the structured data stored in the data warehouse Hive tool includes:

[0016] The data warehouse Hive tool of the Hadoop platform is queried by using the SQL statement through the data query Hive engine or the Spark engine to obtain the structured data stored in the data warehouse Hive tool.

[0017] Optionally, in a possible implementation manner of the first aspect, converting the query parameter into a structured query SQL statement according to the SQL statement placeholder by the Hive driver module includes:

[0018] By means of the Hive driver module, the SQL statement placeholder is matched with the query parameter using a regular expression to determine whether a key value corresponding to the SQL statement placeholder exists in the query parameter;

[0019] If a key value corresponding to the SQL statement placeholder exists in the query parameter, the SQL statement placeholder is replaced with the key value to obtain the SQL statement.

[0020] Optionally, in a possible implementation manner of the first aspect, querying the unstructured data stored in the Hadoop platform based on the unstructured data query request through a standard unified API interface includes:

[0021] Querying metadata corresponding to the unstructured data query request in the key-value pair storage database in the Hadoop platform through a standard unified API interface, wherein the standard unified API interface includes: a storage service S3 protocol interface;

[0022] Unstructured data corresponding to the metadata is queried from the HDFS system, wherein each file data in the HDFS system is divided into one or more data blocks Chunks for storage.

[0023] Optionally, in a possible implementation manner of the first aspect, after returning the query result generated based on the structured data or the unstructured data in a predetermined data format, the method further includes:

[0024] In response to the query result display request, determine the page display style that is pre-customized and generated by the user based on the style sheet language CSS and / or the web page development language JS script;

[0025] The query result in the predetermined data format is displayed using the page display style.

[0026] Optionally, in a possible implementation manner of the first aspect, after returning the query result generated based on the structured data or the unstructured data in a predetermined data format, the method further includes:

[0027] In response to a user's sharing link customization operation, determining the validity period and access rights of the data sharing link pre-customized and generated by the user;

[0028] generating a data sharing link for the user according to the validity period and the access permission;

[0029] In response to an accessing user clicking on the data sharing link within the validity period, the query result in the predetermined data format is shared and displayed, wherein the accessing user is a user having the access authority.

[0030] Optionally, in a possible implementation manner of the first aspect, the method further includes:

[0031] In the process of sharing and displaying the query result in the predetermined data format, recording the access log of the access user to the query result, wherein the access log includes: the IP address, access time and operation type of the access user;

[0032] When the validity period of the data sharing link expires, the data sharing link is marked as invalid, wherein the data sharing link in the invalid state is prohibited from being accessed.

[0033] According to a second aspect of the present application, a distributed data processing device is provided, the device comprising:

[0034] A request unit, configured to respond to a data query request received by the distributed file Hadoop platform and determine whether the data query request is a structured data query request or an unstructured data query request;

[0035] An unstructured query unit, configured to query the unstructured data stored in the Hadoop platform based on the unstructured data query request through a standard unified API interface if the data query request is an unstructured data query request, wherein the Hadoop platform is obtained by extending the distributed file system HDFS based on the open source file system JuiceFS, and the standard unified API interface is a presentation layer state transfer RESTful interface constructed or provided by JuiceFS;

[0036] A structured query unit, configured to query the structured data stored in the Hadoop platform based on the structured data query request through a custom programming API interface if the data query request is a structured data query request, wherein the custom programming API interface is implemented based on an open source programming language;

[0037] The returning unit is used to return the query result generated based on the structured data or the unstructured data in a predetermined data format, wherein the predetermined data format includes: JSON data format.

[0038] Optionally, in a possible implementation manner of the second aspect, the structured query unit includes:

[0039] A custom programming API interface module, used to parse the structured data query request, obtain query parameters and SQL statement placeholders in the structured data query request; and send the query parameters and the SQL statement placeholders to a data query Hive driver module;

[0040] The Hive driver module is used to convert the query parameters into a structured query SQL statement according to the SQL statement placeholder;

[0041] The data query engine is used to use the SQL statement to query the data warehouse Hive tool of the Hadoop platform to obtain the structured data stored in the data warehouse Hive tool. The data query engine includes: a Hive engine or a Spark engine.

[0042] Optionally, in a possible implementation manner of the second aspect,

[0043] The Hive driver module is further used to match the SQL statement placeholder with the query parameter using a regular expression to determine whether there is a key value corresponding to the SQL statement placeholder in the query parameter; if there is a key value corresponding to the SQL statement placeholder in the query parameter, the SQL statement placeholder is replaced with the key value to obtain the SQL statement.

[0044] Optionally, in a possible implementation of the second aspect, the unstructured query unit is further specifically used to query metadata corresponding to the unstructured data query request in the key-value pair storage database in the Hadoop platform through a standard unified API interface; and query the unstructured data corresponding to the metadata from the HDFS system, wherein the standard unified API interface includes: a storage service S3 protocol interface, and each file data in the HDFS system is divided into one or more data blocks Chunks for storage.

[0045] Optionally, in a possible implementation manner of the second aspect, the device further includes:

[0046] The display unit is used to respond to the query result display request, determine the page display style that the user has pre-customized based on the style sheet language CSS and / or the web page development language JS script; and use the page display style to display the query result in the predetermined data format.

[0047] Optionally, in a possible implementation manner of the second aspect, the device further includes:

[0048] A sharing unit is used to determine the validity period and access rights of the data sharing link pre-customized and generated by the user in response to the user's sharing link customization operation; generate the user's data sharing link based on the validity period and the access rights; in response to the visiting user clicking on the data sharing link within the validity period, share and display the query results in the predetermined data format, wherein the visiting user is a user with the access rights.

[0049] Optionally, in a possible implementation manner of the second aspect, the device further includes:

[0050] A recording unit, used to record the access log of the access user to the query result in the process of sharing and displaying the query result in the predetermined data format, wherein the access log includes: the IP address, access time and operation type of the access user;

[0051] The access prohibiting unit is used to mark the data sharing link as an invalid state when the validity period of the data sharing link expires, wherein the data sharing link in the invalid state is prohibited from being accessed.

[0052] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0053] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements any of the methods described in one of the embodiments.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any of the methods described above.

[0055] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes any one of the methods described in the first aspect.

[0056] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0057] The embodiments of the present application provide a distributed data processing method, device, electronic device and program product. The method determines whether the data query request is a structured data query request or an unstructured data query request by responding to a data query request received by a distributed file Hadoop platform; if the data query request is an unstructured data query request, the unstructured data stored in the Hadoop platform is queried based on the unstructured data query request through a standard unified API interface. Specifically, the Hadoop platform is obtained by extending the distributed file system HDFS based on the open source file system JuiceFS, and the standard unified API interface is a presentation layer state transfer RESTful interface constructed or provided by JuiceFS; if the data query request is a structured data query request, the structured data stored in the Hadoop platform is queried based on the structured data query request through a custom programming API interface, and the custom programming API interface is implemented based on an open source programming language; and the query result generated based on the structured data or the unstructured data is returned in a predetermined data format (such as JSON or other formats).

[0058] The distributed data processing method provided in the example of this application classifies data query requests, adopts corresponding efficient processing methods for different types of query requests, and then returns the query results in a predetermined data format. It effectively solves the problems that may be encountered when processing different types of data queries on the distributed file Hadoop platform, such as low query efficiency and difficulty in obtaining data, so as to achieve efficient and flexible access to different types of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 It is a flow chart of a distributed data processing method provided in an embodiment of the present application;

[0061] Figure 2 It is a flowchart of an optional distributed data processing method provided in an embodiment of the present application;

[0062] Figure 3 It is a flowchart of an optional distributed data processing method provided in an embodiment of the present application;

[0063] Figure 4 It is a flowchart of an optional distributed data processing method provided in an embodiment of the present application;

[0064] Figure 5 It is a flowchart of an optional distributed data processing method provided in an embodiment of the present application;

[0065] Figure 6 It is a schematic diagram of the structure of an optional big data platform provided in an embodiment of the present application;

[0066] Figure 7 is a structural diagram of a distributed data processing device provided in an embodiment of the present application;

[0067] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0069] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0070] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0071] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0072] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0073] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0074] First, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0075] Representational State Transfer (RESTful) is a software architecture style that uses unified interface rules (such as using standard HTTP methods to operate resources), takes resources as the core, and implements HTTP protocol-based interaction between clients and servers in a stateless manner. It is used to efficiently build interoperable network application program interfaces (APIs).

[0076] JuiceFS is an open source distributed file system that adopts a data and metadata separation architecture. It can store metadata in the database and data in object storage. It has the characteristics of high performance and strong compatibility, and is suitable for a variety of data storage and application scenarios.

[0077] Representational State Transfer Application Programming Interface (Restful API) is an API designed in accordance with the RESTful architectural style.

[0078] Structured Query Language (SQL) is a standard language for interacting with databases.

[0079] JSON (JavaScript Object Notation) is JavaScript object representation, which is a lightweight data exchange format.

[0080] Redis cluster is a distributed deployment method of Redis database. Redis itself is a high-performance key-value pair storage database, commonly used in scenarios such as caching and message queues. By building a cluster, distributed storage and processing of data can be achieved.

[0081] Simple Storage Service (S3) is an object storage protocol that stores data in the form of objects. Each object contains the data itself, metadata (such as object name, size, creation time, etc.) and a unique identifier.

[0082] Chunk is a relatively independent part divided from a complete file according to certain rules. It is a basic unit in the process of file storage and transmission.

[0083] Kyuubi is a distributed SQL query engine mainly used in big data ecosystems.

[0084] Go is an open source programming language that is widely used in network programming, distributed systems, cloud computing and other fields. It is often used to develop high-performance, high-reliability server-side programs and various tool software.

[0085] Hadoop Distributed File System (HDFS) is the core component of the Apache Hadoop project. It is based on a master-slave architecture, with name nodes managing metadata and data nodes storing data blocks. It is a highly reliable and scalable distributed file system designed specifically for processing large-scale data.

[0086] Hive is a data warehouse tool based on Hadoop, which is mainly used to store, manage and query large-scale data. It uses Hadoop's distributed file system HDFS to store data and uses Hadoop's computing framework to realize data calculation and processing.

[0087] Spark is a fast and general distributed computing framework for large-scale data processing and analysis. It has efficient in-memory computing capabilities, supports multiple programming languages, including Java, Python, Scala, etc., and can be used as a query engine to perform data query tasks.

[0088] Cascading Style Sheets (CSS) is a style sheet language used to describe the presentation style of documents such as HTML (Hypertext Markup Language) or XML (Extensible Markup Language). It is used to control the appearance of web page elements, such as setting the font, color, and size of text, defining the layout of elements (such as spacing, alignment, etc.), and determining style attributes such as background color and images.

[0089] JavaScript, or JS for short, is a high-level programming language widely used in web development. It usually runs in client browsers and can now also be used for server-side development (such as Node.js).

[0090] Lightweight Directory Access Protocol (LDAP) is a network protocol used to access and manipulate information in directory services. Directory services are usually used to store and manage structured data such as user information, organizational structure, and device information.

[0091] OAuth 2.0 is an authorization framework used to implement authorization and resource sharing between different applications, websites or services. It allows users to authorize third-party applications to access some of their resources in a main application (such as a social networking platform) without providing their main application account password to the third-party application.

[0092] Security Assertion Markup Language (SAML) is an XML-based language used to transfer authentication and authorization information between different network entities. It is mainly used to implement cross-domain authentication and authorization between enterprises and applications.

[0093] The above is a brief introduction to the nouns involved in the embodiments of the present application, which will not be repeated below.

[0094] This application example provides an example of a distributed data processing method. Please refer to Figure 1 As shown, Figure 1 A schematic flow chart of a distributed data processing method provided by the present application is shown. As an example but not a limitation, the method can be applied to or run in an electronic device. The method includes:

[0095] S101, in response to a data query request received by a distributed file Hadoop platform, determining whether the data query request is a structured data query request or an unstructured data query request.

[0096] S102: If the data query request is an unstructured data query request, query the unstructured data stored in the Hadoop platform based on the unstructured data query request through a standard unified API interface.

[0097] Among them, the Hadoop platform is obtained by extending the distributed file system HDFS based on the open source file system JuiceFS, and the standard unified API interface is the presentation layer state transfer RESTful interface built or provided by JuiceFS.

[0098] S103: If the data query request is a structured data query request, query the structured data stored in the Hadoop platform based on the structured data query request through a custom programming API interface.

[0099] Among them, the custom programming API interface is implemented based on open source programming language.

[0100] S104, returning the query result generated based on the structured data or the unstructured data in a predetermined data format, where the predetermined data format includes: JSON data format.

[0101] In this application example, when the distributed file Hadoop platform receives a data query request, the data query request is first classified to determine whether the data query request is a structured data query request or an unstructured data query request. This is because structured data and unstructured data differ in storage methods, processing logic, etc. By clarifying the type of data query request, a targeted processing method can be used to accurately obtain and return data that meets the requirements, thereby improving the efficiency and accuracy of data processing.

[0102] As an optional example, if it is determined that the distributed file Hadoop platform receives an unstructured data query request, the query operation is performed through a standard unified API interface. In the example of this application, the Hadoop platform is based on the open source file system JuiceFS to expand the distributed file system HDFS, and specifically expands the storage capacity and flexibility of the Hadoop platform through the combination of JuiceFS and HDFS. The characteristics of JuiceFS (such as the data and metadata separation architecture, etc.) complement the distributed storage capabilities of HDFS, enabling the platform to better cope with data storage needs of different types and scales, and provide a reliable and efficient environment for the storage of unstructured data.

[0103] The above-mentioned standard unified API interface is a presentation layer state transfer RESTful interface built or provided by JuiceFS, for example, an S3 protocol interface. The S3 protocol interface adopts a RESTful interface with a unified interface specification and standard HTTP method operations (such as GET for obtaining resources, PUT for updating resources, etc.). By adopting the S3 protocol interface, it is possible to conveniently and accurately find and query the unstructured data stored in the Hadoop platform based on the unstructured data query request, thereby solving the problem that it is difficult for ordinary users to quickly obtain the unstructured data because the unstructured data is stored in a complex file system, making the query of unstructured data more standardized, convenient and efficient.

[0104] As an optional example, when it is determined that the data query request is a structured data query request, the structured data stored in the Hadoop platform is queried through a custom programming API interface. The custom programming API interface is based on an open source programming language, for example, Go language. Go language has the characteristics of simplicity, efficiency, strong concurrency, etc., and is suitable for developing high-performance API interfaces.

[0105] Through the customized API interface, the required structured data can be accurately obtained from the Hadoop platform according to the specific requirements of the structured data query request. Compared with the traditional and cumbersome SQL query method, this API-based processing method is more flexible and efficient. It can be customized according to specific business needs, and can better adapt to the query needs of structured data in different scenarios, overcoming the limitation that the original structured data sharing requires cumbersome SQL queries.

[0106] Finally, no matter whether it is a structured data query request or an unstructured data query request, the query result generated based on the queried structured data or unstructured data will be returned in a predetermined data format (for example, a lightweight data exchange JSON data format). JSON data format is a lightweight data exchange format that facilitates data interaction, has a simple and clear data structure (consisting of objects, arrays, strings, numbers, Boolean values, and null, etc.), is easy to read and write, and is well compatible with multiple programming languages.

[0107] The query results are returned in JSON data format, so that users in different application scenarios can easily receive and further process these results. Whether in Web applications, mobile applications or other types of software systems, the JSON data format can be easily parsed and used, thereby ensuring smooth interaction of data between different systems and users. The above-mentioned way of returning query results improves the versatility and ease of use of the data processing method provided by the example of this application, so that the distributed data processing method can better adapt to various application scenarios and user needs, and can easily process the received query results regardless of the programming language or development environment used by the user.

[0108] To summarize, the distributed data processing method provided in the example of this application classifies query requests, adopts corresponding efficient processing methods for different types of requests, and finally returns the query results in a suitable format. It effectively solves the problems that may be encountered when processing different types of data queries on the distributed file Hadoop platform, such as low query efficiency and difficulty in obtaining data, and improves the overall performance of data processing and user experience.

[0109] For a possible implementation, please refer to Figure 2 As shown, Figure 2 A schematic flow chart of an optional distributed data processing method provided by the present application is shown, which queries structured data stored in the Hadoop platform based on a structured data query request through a custom programming API interface, including:

[0110] S201, parsing a structured data query request through a custom programming API interface to obtain query parameters and SQL statement placeholders in the structured data query request.

[0111] S202, sending query parameters and SQL statement placeholders to the data query Hive driver module.

[0112] S203, through the Hive driver module, convert the query parameters into a structured query SQL statement according to the SQL statement placeholder.

[0113] S204, using SQL statements to query the data warehouse Hive tool of the Hadoop platform to obtain structured data stored in the data warehouse Hive tool.

[0114] When processing a structured data query request through a custom programming API interface, the structured data query request is first parsed. For example, in actual application scenarios, the structured data query request sent by the user often contains a variety of information, such as: specific query conditions, data fields to be obtained, etc. This information is combined in the request in a specific format or according to certain rules. By parsing the request, key query parameters and SQL statement placeholders can be extracted.

[0115] For example, a user sends a data query request: "Query the order information with sales greater than 1000 and region 'BJ', and the query time period is #{start_time} to #{end_time}". Here, "sales greater than 1000" and "region 'BJ'" are query parameters, and "#{start_time}" and "#{end_time}" are SQL statement placeholders; the parsing operation is to accurately identify and separate these different types of information for subsequent targeted processing.

[0116] After parsing the structured data query request and obtaining the query parameters and SQL statement placeholders, this information will be sent to the data query Hive driver module. In the entire data query process, the Hive driver module is specifically used to connect the upper-level query request and the underlying data storage and query tool (the Hive tool for the data warehouse of the Hadoop platform). The query parameters and SQL statement placeholders are sent to the Hive driver module. After receiving the query parameters and SQL statement placeholders, the Hive driver module converts the query parameters into structured query SQL statements according to the SQL statement placeholders.

[0117] Still using the above example to explain, the Hive driver module uses the query parameters "sales greater than 1000" and "region is 'BJ'" according to the SQL syntax rules and the replacement logic of the "#{start_time}" and "#{end_time}" placeholders to generate a complete, SQL-compliant structured query SQL statement. For example, a statement like this is generated: "SELECT * FROM orders WHERE sales>1000 AND region = 'BJ' AND order_time BETWEEN #{start_time} AND #{end_time}" (assuming that orders is the order table, sales is the sales field, region is the region field, and order_time is the order time field).

[0118] Through the above conversion operation, the request information with placeholders initially sent by the user can be converted into SQL statements that can be directly executed in the data warehouse Hive tool, so as to facilitate subsequent query of the structured data actually stored in the Hadoop platform.

[0119] In a possible implementation, a SQL statement is used to query the data warehouse Hive tool of the Hadoop platform to obtain structured data stored in the data warehouse Hive tool, including:

[0120] Through the data query Hive engine or Spark engine, use SQL statements to query the data warehouse Hive tool of the Hadoop platform to query the structured data stored in the data warehouse Hive tool.

[0121] In this application example, when querying the Hive tool, a data warehouse of the Hadoop platform, to obtain the stored structured data, two optional query engines are provided, namely the Hive engine and the Spark engine. It should be understood that in this application example, whether the Hive engine or the Spark engine is selected, the previously generated SQL statement will be used to query the Hive tool, a data warehouse of the Hadoop platform.

[0122] As an explanation but not limitation, the Hive engine is a data warehouse tool based on Hadoop, and its Hive engine is the core mechanism for data processing, query and other operations. It mainly uses Hadoop's distributed file system (HDFS) to store data, and uses Hadoop's MapReduce computing framework (now it can also be combined with other computing frameworks such as Spark) to realize data calculation and processing. The Hive engine uses a query language similar to SQL (HiveQL), which is easy to use for users familiar with SQL. It can easily query the structured data stored in the Hive tool, such as querying sales data within a certain period of time, counting the number of users in different regions, etc.

[0123] As an explanation rather than a limitation, the Spark engine is a fast, general-purpose distributed computing framework with efficient in-memory computing capabilities. Compared with the traditional MapReduce framework, it can provide faster processing speeds in many scenarios. The Spark engine can also be used to query structured data in the Hive tool of the Hadoop platform's data warehouse. It supports multiple programming languages, including Java, Python, Scala, etc., which enables developers with different programming backgrounds to easily use the Spark engine for data query operations. Moreover, the Spark engine can be used not only for simple data queries, but also for various data analysis and processing operations, such as machine learning, graph computing, etc., and has strong versatility.

[0124] For example, in one example, when the Hive engine is used, it searches, filters, counts, and performs other operations in the structured data stored in the Hive tool according to its own processing mechanism and rules and the requirements of the SQL statement. For example, if the SQL statement is to query the sales volume of a specific product in several regions, the Hive engine will match the values ​​of the region field and the product field in the corresponding sales data table, and then count the sales volume records that meet the conditions.

[0125] Similarly, in another example, when the Spark engine is selected, it will also process the structured data in the Hive tool according to the provisions of the SQL statement by virtue of its powerful computing power and memory computing advantages. For example, for large-scale sales data queries, the Spark engine can analyze and process the data in memory more quickly and obtain query results quickly. Compared with the Hive engine in some cases, it has higher processing efficiency, especially in scenarios where massive data is processed and high processing speed is required.

[0126] In this application example, query operations are performed through either of these two engines. The ultimate goal is to query the Hive tool in the Hadoop platform's data warehouse to obtain structured data that meets the requirements of SQL statements. These data can be further used for analysis, display or other related business operations.

[0127] In summary, this application example provides two options, Hive engine and Spark engine, to perform query operations, so that when querying structured data in the Hive tool of the Hadoop platform's data warehouse, you can flexibly select the appropriate engine based on specific business needs, data scale, and processing speed requirements, thereby obtaining the required structured data more efficiently and accurately.

[0128] For a possible implementation, please refer to Figure 3 As shown, Figure 3A schematic flow chart of an optional distributed data processing method provided by the present application is shown, which converts query parameters into structured query SQL statements according to SQL statement placeholders through a Hive driver module, including:

[0129] S301, using a Hive driver module, matching the SQL statement placeholder with the query parameter using a regular expression to determine whether a key value corresponding to the SQL statement placeholder exists in the query parameter;

[0130] S302: If there is a key value corresponding to the SQL statement placeholder in the query parameter, the SQL statement placeholder is replaced with the key value to obtain the SQL statement.

[0131] In this application example, in the process of converting query parameters into structured query SQL statements, the Hive driver module first uses regular expressions to match SQL statement placeholders and query parameters.

[0132] It should be understood that regular expressions are a powerful text processing tool that can match text content according to specific patterns. Here, the Hive driver module uses the pattern matching ability of regular expressions to carefully compare the placeholders in the SQL statement (such as #{start_time}, #{end_time}) with the received query parameters.

[0133] For example, suppose there is a SQL statement placeholder #{product_id}, and the query parameter is a dictionary containing multiple key-value pairs, one of which is "product_id" and the value is "12345". By matching the regular expression, we can accurately find out whether there is a key-value pair corresponding to the placeholder #{product_id} in the query parameter, that is, whether there is a key "product_id".

[0134] After matching the regular expression, if it is found that there is a key value corresponding to the SQL statement placeholder in the query parameter, the corresponding key value is used to replace the placeholder in the SQL statement to obtain a complete SQL statement.

[0135] Continuing with the above example, once the query parameter "product_id":"12345" is determined to correspond to the #{product_id} placeholder, all #{product_id} in the SQL statement are replaced with "12345". If the original SQL statement template is "SELECT*FROM products WHERE product_id=#{product_id}", after the replacement operation is completed, the complete SQL statement "SELECT*FROM productsWHERE product_id='12345'" will be obtained.

[0136] Through the above optional implementation, the Hive driver module can accurately convert the SQL statement template with placeholders into a structured query SQL statement that can be directly executed in the data warehouse Hive tool according to the query parameters provided by the user, so that the appropriate query engine (such as Hive engine or Spark engine) can be used to query and obtain the required structured data. The matching and replacement mechanism based on regular expressions makes the processing of structured data query requests more flexible and accurate, and can better adapt to the query needs of different users in different scenarios.

[0137] For a possible implementation, please refer to Figure 4 As shown, Figure 4 A schematic flow chart of an optional distributed data processing method provided by the present application is shown, which queries unstructured data stored in the Hadoop platform based on an unstructured data query request through a standard unified API interface, including:

[0138] S401, querying metadata corresponding to the unstructured data query request in the key-value pair storage database in the Hadoop platform through a standard unified API interface.

[0139] S402, querying the unstructured data corresponding to the metadata from the HDFS system, wherein each file data in the HDFS system is divided into one or more data blocks Chunks for storage.

[0140] When processing an unstructured data query request through a standard unified API interface, the metadata corresponding to the request is queried first. The metadata is stored in a key-value pair storage database in the Hadoop platform (for example, Redis, PostgreSQL, etcd, TiKV, and other databases). It should be understood that in the entire data storage architecture, the key-value pair storage database is used to store metadata, which contains various descriptive information about the unstructured data, such as the file's name, size, creation time, storage location, and other attributes.

[0141] When receiving an unstructured data query request, it interacts with the key-value storage database through the S3 protocol interface, and uses the key information in the query request (such as the file name, specific identifier, etc.) to find the corresponding metadata record in the key-value storage database. For example, if the query request is to find relevant information about a specific video file, the metadata record containing the detailed attributes of the video file is searched in the key-value storage database based on the name or other unique identifier of the video file.

[0142] After obtaining the metadata corresponding to the unstructured data query request from the key-value storage database, the next step is to query the actual unstructured data from the HDFS system based on the metadata. It should be noted that each file data in the HDFS system is divided into one or more data blocks (Chunks) for storage. This division method helps to improve data storage and processing efficiency, especially for large-scale unstructured data.

[0143] When the metadata is queried, it will contain key information about the storage location of the unstructured data in the HDFS system (such as the data block number, storage node, etc.). Based on the metadata, the corresponding unstructured data can be accurately located in the HDFS system. For example, if the metadata indicates that part of the data of a certain image file is stored in certain specific data blocks (determined by the data block number) of the HDFS system and is located on certain data nodes, the complete data of the image file can be found and obtained in the HDFS system.

[0144] Through the above implementation method, the two-step operation of first querying the key-value storage database to obtain metadata and then querying the unstructured data from the HDFS system based on the metadata can accurately obtain the required unstructured data from the Hadoop platform based on the unstructured data query request, utilizing the convenience of storing metadata in the key-value storage database and the advantage of the HDFS system for large-scale data storage, and further optimizing the data storage and query process by dividing the data into chunks, making the query of unstructured data more efficient and accurate.

[0145] In a possible implementation, after returning the query result generated based on the structured data or the unstructured data in a predetermined data format, the method further includes:

[0146] In response to the query result display request, determine the page display style that is pre-customized and generated by the user based on the style sheet language CSS and / or the web page development language JS script;

[0147] The query results in a predetermined data format are displayed using a page display style.

[0148] In the example of the present application, after returning the query results generated based on structured data or unstructured data in a predetermined data format, when a query result display request is received, it can be determined that the page display style that the user has customized in advance based on the style sheet language CSS and / or the web page development language JS script. For example, in previous operations (through the interface beautification and development module and other related functions of the unified management platform), the user used CSS and JS to customize the appearance and interaction mode of the data display page according to the user's needs. When there is a query result display request, the system will identify the custom page display style based on CSS and JS that the user has previously set, so that the query results can be displayed in this style later.

[0149] CSS is mainly used to control the style of page elements, such as setting the font, color, and size of text, defining the layout of elements (such as spacing, alignment, etc.), and determining style attributes such as background color and images. JS can achieve dynamic effects on web pages, such as changes in elements when the mouse hovers, expansion and collapse of menus, etc. It can also interact with users, such as obtaining and processing user input data.

[0150] Optionally, the query results in the predetermined data format include various data information, such as query status, data content, success information, and total number of records for structured data query results (for example: {"code":200,"data":{...},"msg":"Execution successful"}). For unstructured data query results, relevant data information will also be presented in a JSON format.

[0151] In addition, in this application example, the presentation of the query results on the page can be adjusted according to the CSS style set by the user, such as displaying the text content according to the set font, color, layout, etc., to make the data clearer and more beautiful. At the same time, with the help of dynamic effects and interactive functions realized by JS scripts, users can experience interactive effects such as clicking on a data item to expand detailed information and dynamic loading of data when scrolling the page when viewing the query results.

[0152] Through the above optional implementation methods, not only can the query results be accurately displayed to the user, but the display can also be made more personalized and interactive in a user-defined manner, further improving the user experience and making the data display more in line with the user's needs and expectations. On the basis of satisfying data sharing and query functions, it provides users with better visual effects and interactive experience.

[0153] For a possible implementation, please refer to Figure 5 As shown, Figure 5A schematic flow chart of an optional distributed data processing method provided by the present application is shown. After returning the query result generated based on structured data or unstructured data in a predetermined data format, the method further includes:

[0154] S501, in response to a user's sharing link customization operation, determining the validity period and access rights of the data sharing link pre-customized and generated by the user;

[0155] S502, generating a user's data sharing link according to the validity period and access rights;

[0156] S503, in response to the access user clicking on the data sharing link within the validity period, sharing and displaying the query results in a predetermined data format, wherein the access user is a user with access rights.

[0157] In this application example, after returning the query results generated based on structured data or unstructured data in a predetermined data format, when the system receives the user's customized sharing link operation, the first thing to do is to determine the validity period and access rights of the data sharing link that the user pre-customized and generated.

[0158] As an example and not a limitation, users can set the validity period of the shared link according to their own needs. For example, users can choose different time periods such as 1 hour, 1 day, 1 week, etc. as the validity period, so that users can flexibly control the accessible time of the data and prevent the data from being accessed by others for a long time without restriction, thereby ensuring the security and privacy of the data to a certain extent. For example, if a user only temporarily needs to share the data query results of a project report with others, a shorter validity period can be set, such as 1 hour, to ensure that the link automatically expires after the sharing purpose is achieved, preventing the data from being obtained by unnecessary personnel later.

[0159] In addition, in this application example, users can also set access permissions to decide who can access the data displayed by the shared link. For example, users can set it to public access, that is, any user can click the link to view the data without additional authentication; or they can set it to restricted access, for example, only a specific user group (through some kind of identity authentication mechanism, such as association with the user account system within the enterprise) can access it. By setting access permissions, users can further accurately control the scope of data sharing, protect the privacy and security of data, and prevent sensitive information from being obtained by unauthorized personnel.

[0160] After determining the validity period and access rights, a data sharing link for the user is generated based on these settings. The sharing link is not just a simple URL, but contains information about relevant settings such as the validity period and access rights. For example, during the link generation process, the parameters of the validity period and access rights are embedded into the link through a certain encoding method so that they can be accurately identified and verified by the system during subsequent visits. In this way, when the visiting user clicks the link, it can quickly determine whether the user meets the access rights requirements and whether the link is still within the validity period.

[0161] When the accessing user clicks on the data sharing link within the validity period, and the accessing user is a user with access rights, the system will share and display the query results generated in the JSON data format. In other words, only the accessing user who meets all the conditions set by the user before (within the validity period and with access rights) can successfully see the query results presented in the JSON data format. The query results in the predetermined data format contain various relevant data information (such as the query status, data content, success information, and total number of records for the structured data query results mentioned above; for unstructured data query results, the relevant data will also be displayed in the corresponding JSON structure). Furthermore, users can easily share the data query results with others according to the rules they set, while ensuring the security and controllability of data sharing, and meeting the data sharing needs in different scenarios.

[0162] By using the above example of this application, by allowing users to customize the validity period and access permissions of data sharing links, generating links based on these settings, and displaying query results when conditions are met, users are provided with a more flexible, secure and controllable data sharing method, further improving the practicality and user experience of data sharing services.

[0163] In a possible implementation, the method further includes:

[0164] In the process of sharing and displaying the query results in a predetermined data format, recording the access log of the access user to the query results, wherein the access log includes: the IP address, access time and operation type of the access user;

[0165] When the validity period of the data sharing link expires, the data sharing link is marked as invalid, wherein the data sharing link in the invalid state is prohibited from being accessed.

[0166] In the process of sharing and displaying the query results in a predetermined data format to qualified access users, the system will perform the operation of recording access logs.

[0167] Recorded information content: The recorded access log contains several key information, namely the IP address of the accessing user, access time and operation type.

[0168] Optionally, the IP address is an important identifier for identifying the location of the accessing user on the network. By recording the IP address, on the one hand, we can understand from which regions the data is accessed, which is helpful for analyzing the dissemination range and audience distribution of the data; on the other hand, when abnormal access occurs (such as suspected malicious attacks, unauthorized large-scale access, etc.), the IP address can be used as an important clue for tracking and troubleshooting problems.

[0169] Optionally, the access time is used to clearly record the specific time point of each access user's access to the query results, so as to facilitate the subsequent analysis of the data's access frequency, peak period and other usage patterns. For example, an enterprise can determine in which time periods users are more interested in specific data based on the access time, so as to reasonably arrange data update, maintenance and other related work.

[0170] Optionally, the operation type is the specific operation performed by the accessing user when viewing the query results, such as simply browsing the page, downloading data, sorting and filtering data, etc. It should be understood that understanding the operation type helps to gain a deeper understanding of the user's use and needs for data, and then optimize the data display method based on these feedbacks, provide functions that better meet user needs, etc.

[0171] In this application example, recording access logs can provide important basis for data management, security monitoring and subsequent service optimization, so that data owners or managers can fully understand the use of data, timely discover potential security risks or abnormal behaviors, and continuously improve data sharing services based on records to better meet user needs.

[0172] When the validity period of a data sharing link expires, the system will mark the data sharing link as invalid, and access to the data sharing link in this invalid state is prohibited. For example, setting an validity period and marking the link invalid after expiration can effectively prevent data from being accessed by others after the sharing period set by the user has expired, avoiding the risk of data leakage caused by the long-term validity of the sharing link, and further ensuring the security and privacy of the data.

[0173] Through the above optional implementation methods, users can more accurately control the scope and duration of data sharing, ensuring that the data is only accessed by the desired group of people within the desired time period. Once the validity period expires, even if someone accidentally obtains the link, they will no longer be able to successfully access the relevant query results, so that the entire data sharing process is always under the control of the user. While ensuring that data can be shared and displayed according to user expectations, it can further strengthen the security management and controllability of data sharing, providing users with a safer, more reliable and convenient data sharing experience.

[0174] The distributed data processing method example provided in this application example is applicable to scenarios of large-scale data sharing and access in multiple fields, and can be widely used in the following fields in particular:

[0175] (1) Enterprise data sharing platform

[0176] Many companies face the need to share a large amount of structured and unstructured data. This application example simplifies the process of data access and sharing through a unified Restfu style interface, which is suitable for application in data sharing scenarios within the enterprise or across departments, especially scenarios that require flexible access to multiple data types, such as sharing and analysis of financial data and business data.

[0177] (2) Big Data Analysis Platform

[0178] In big data analysis, it is often necessary to process complex structured and unstructured data. This application example can quickly obtain data through a standardized API interface, reducing the complexity of data processing, and is suitable for big data analysis platforms and related data mining and analysis systems. It is particularly suitable for scenarios that require real-time or batch processing of large-scale data.

[0179] (3) Cloud storage and cloud computing services

[0180] This application example integrates JuiceFS and S3 protocols to support cloud storage and sharing of large-scale unstructured data, and is suitable for cloud storage service providers as an optimized storage and data sharing solution. It can provide enterprise users with more flexible storage and data access methods and reduce data management and storage costs.

[0181] (4) Scientific research and education

[0182] This application example can be applied to the sharing and analysis of large-scale data in the fields of scientific research and education. It is especially suitable for research institutes and universities that need to store and process large amounts of experimental data, paper data, and educational resources. Through the solution of this application example, these data can be efficiently and securely shared and managed in the academic and education systems.

[0183] (5) Industrial control systems and industrial control security

[0184] In the field of industrial control systems (ICS) and industrial control security, this application example can help enterprises achieve centralized management and sharing of large-scale industrial data. Through the S3 protocol provided by JuiceFS, unstructured data in industrial control systems (such as log files, sensor data, etc.) can be securely stored and shared, and structured data (such as database information in the control system) can also be accessed and analyzed through API query interfaces. This application example also supports the aggregation and rapid response of multi-source data in industrial control networks through customized interfaces, improves the data processing capabilities and network security of industrial control safety systems, and helps enterprises monitor and quickly respond to threats in industrial control environments.

[0185] The distributed data processing method provided in the example of this application enables users to achieve unified management and sharing of structured and unstructured data in a variety of application scenarios, greatly improving the efficiency of data acquisition and analysis and meeting the needs of modern data-intensive businesses.

[0186] The following is an example of a sales data sharing and analysis system for an enterprise, providing a specific implementation method based on the above distributed data processing method, and providing a method for using the highly customizable big data sharing service based on the RESTful style in actual scenarios:

[0187] For example, a company has a large amount of sales data, including structured data (such as sales order tables, customer information tables, etc.) and unstructured data (such as promotional pictures, videos of sales activities, and sales logs, etc.). As the business develops, the company needs a system that can efficiently manage, flexibly share, and conveniently analyze this data to meet the diverse data needs of different departments (such as sales departments, marketing departments, data analysis departments, etc.), while improving the convenience and flexibility of data sharing, and being able to easily expand and customize functions according to business changes. For example, the system architecture construction specifically uses the company's existing Hadoop big data platform as the infrastructure, which already has certain storage and computing capabilities and can meet the processing needs of large-scale data.

[0188] Integrate JuiceFS and self-developed Restful API interface to realize unstructured data processing and structured data processing. Specifically, for unstructured data processing: store unstructured data (such as promotional pictures, videos and sales logs of sales activities) on Hadoop HDFS. In order to better manage and access these unstructured data, use JuiceFS to expand HDFS and build a storage platform that supports the S3 protocol. JuiceFS formats the data and stores it in HDFS, and stores the metadata of the file in the Redis cluster (as described in the previous article, each file consists of one or more Chunks, which facilitates divide-and-conquer processing and improves performance). JuiceFS provides an S3 interface, allowing users to access unstructured data stored in HDFS through the standard S3 protocol. For example, if the marketing department wants to obtain a promotional video for a sales event, it can access the corresponding unstructured data by sending a standard S3 API request (such as a GET request for downloading a video).

[0189] For structured data processing: Structured data (such as sales order tables, customer information tables, etc.) are stored in Hive on the Hadoop platform. In order to simplify the access to structured data, a custom API interface was developed. This service connects and communicates with the Hive engine through a customized Hive driver module (UnifiedEngine) implemented in the Go language, and supports Spark as a query engine.

[0190] As for the storage and access of unstructured data, as described above, unstructured data is stored on HDFS extended by JuiceFS. JuiceFS is responsible for formatting and storing data, and storing metadata in the Redis cluster for quick subsequent location and access to data. Users (such as marketing personnel who need to obtain promotional materials) access unstructured data by sending standard S3API requests. The RESTful-style S3 protocol interface provided by JuiceFS will process requests from users, query the corresponding metadata in the Redis cluster based on the information in the request (such as file name, bucket name, etc.), and then extract and return the corresponding unstructured data from HDFS based on the metadata to the user.

[0191] For structured data storage and access, structured data is stored in Hive and organized in the form of table structure, which is convenient for data management and query operations. For querying structured data, users (such as data analysis department personnel who need to analyze sales order data) send requests with SQL query parameters through the API interface. For example, the request contains something like "query for sales order information with sales greater than a certain amount within a certain time period, where the time period is represented by placeholders #{start_time} and #{end_time}". The system will first use the customized Hive driver based on the request parameters to dynamically fill in and parse the SQL query statement with the actual parameters provided by the user (such as the specific time period value) by identifying the SQL statement placeholders. Then, the customized Hive driver sends the query request to the Hive engine or Spark engine based on the parsed SQL statement (select according to the actual situation, such as the Spark engine can be selected when the data volume is large and the speed requirement is high). The Hive engine or Spark engine will execute the corresponding query operation on the Hadoop cluster and extract structured data from the structured data table.

[0192] For unstructured data queries, when marketing personnel want to view all promotional images for a sales event, they send a standard S3 API request (such as a LIST request to list all objects in a bucket) to the S3 protocol interface provided by JuiceFS. The interface will process the request, first query the image metadata corresponding to the request in the Redis cluster, and then extract and return all qualified promotional images from HDFS based on the metadata to the user.

[0193] For structured data query, suppose the data analysis department wants to query the sales order information with sales greater than 100,000 yuan in the past month. Send a request with SQL query parameters through the API interface, which contains placeholders #{start_time} and #{end_time} to represent the start and end time of the query period respectively. After the system receives the request, the custom Hive driver first replaces the placeholders with the actual parameters provided by the user (specific start and end time) according to the passed SQL statement placeholders to generate the corresponding query statement. Then send the query statement to the Hive engine or Spark engine for execution. After execution, the query results will be packaged in JSON data format and returned to the user through the API interface for display or further processing.

[0194] In this application example, we still take the enterprise sales data sharing and analysis system as an example. The system relies on the enterprise's existing Hadoop cluster environment, and the data is stored in HDFS. According to the development of the enterprise's business and the growth of data volume, users can flexibly expand the scale of the Hadoop cluster according to the system load to meet the needs of large-scale data processing. For example, when an enterprise carries out large-scale promotional activities and the amount of sales data increases sharply, the number of nodes in the Hadoop cluster can be increased to improve the system's storage and computing capabilities.

[0195] JuiceFS is deployed on top of HDFS and provides S3 protocol support for storage and access of unstructured data. After deployment, JuiceFS will be tightly integrated with Hadoop to uniformly manage data files in HDFS. For example, in daily operations, JuiceFS is responsible for storing unstructured data such as newly generated sales promotion materials in HDFS according to its set method, and manages related metadata to ensure efficient storage and access of data.

[0196] Please refer to Figure 6 As shown, Figure 6 This paper shows a schematic diagram of an optional big data platform structure to explain a highly customizable big data sharing service based on the RESTful style and its implementation method, aiming to solve the problem of insufficient flexibility and convenience in data sharing and access in existing big data platforms. The big data platform can include but is not limited to the following modules:

[0197] (1) Unstructured data sharing module

[0198] The present invention builds an S3-compatible unstructured data sharing module based on JuiceFS, HDFS, and Redis cluster. HDFS is selected as the underlying storage to ensure data security and scalability, and it can be smoothly connected and compatible with the existing big data platform; on the basis of JuiceFS, its S3 function is repackaged to implement customized verification and authentication protocols to connect with the module.

[0199] When storing data, the metadata of the data will be stored in Redis for subsequent access and retrieval by JuiceFS. At the same time, unlike traditional file systems that can only use local disks to store data and corresponding metadata, JuiceFS will format the data and store it in HDFS, and at the same time store the file's metadata in Redis.

[0200] Each file consists of one or more Chunks, and each Chunk is up to 64M. No matter how large the file is, all reads and writes will be located to the corresponding Chunk based on its offset (that is, the file location where the read and write operations are generated). With the divide-and-conquer design, JuiceFS also has excellent performance for large files. When accessing, it is first authenticated through the S3 general protocol, and the authentication request is finally handled by the custom verification and authentication module. After passing, the S3 protocol correctly outputs unstructured data.

[0201] By introducing the S3 protocol, users can easily access unstructured data stored in HDFS using the standard RESTful API interface. As part of the RESTful style, the S3 protocol aims to provide a simple, flexible, and efficient solution that allows data to be easily stored and accessed via HTTP. The S3 protocol supports several basic operations, usually interacting using HTTP requests:

[0202] 1.PUT: upload data to the specified bucket, creating or replacing an object.

[0203] 2.GET: Download data from the specified bucket and obtain the content of the object.

[0204] 3.DELETE: Delete the object in the specified bucket.

[0205] 4.LIST: List all objects in the bucket and get the contents of the bucket.

[0206] S3 protocol requests usually use standard HTTP methods (such as GET, PUT, DELETE, etc.), including necessary header information and parameters. For example, when a PUT request is used to upload an object, it is usually necessary to specify the content type and length, and authentication information is required; the response contains the HTTP status code, response header, and response body. The status code indicates whether the operation is successful (such as 200 for success and 404 for object not found), and the response body usually contains the requested object data (in a GET request) or other information (such as ETag, indicating the version of the object).

[0207] Based on the S3 protocol, a unified RESTful API interface is provided, allowing users to easily upload, download and manage unstructured data such as pictures, videos, log files, etc. through standard HTTP requests. This consistency simplifies the data operation process and lowers the threshold for use. At the same time, the S3 protocol is designed to handle unstructured data of any quantity and size. Users can dynamically expand storage capacity as needed without worrying about the limitations of the underlying infrastructure. Accessing unstructured data based on the S3 protocol greatly improves the convenience of accessing unstructured data.

[0208] (2) Structured data sharing module

[0209] In order to solve the problem of sharing structured data (such as data tables in HIVE), the present invention designs a self-developed API service module. First, a request containing a SQL query template and corresponding parameters sent by the user is received. The SQL template contained in the request may contain placeholders, such as #{device_id} and #{start_time}, which will be dynamically replaced with the actual parameters provided by the user in subsequent steps. Use regular expressions to match the placeholders in the SQL query template. For each matched placeholder, the module will check the parameter dictionary provided by the user to find out whether there is a corresponding key value.

[0210] If the corresponding key value is found, the placeholder is replaced with the value to achieve dynamic filling; after filling the parameters, a simple syntax check is performed on the generated SQL query to ensure that the generated SQL complies with the SQL standard, and then the filled and checked SQL query is passed to the corresponding execution engine, using Kyuubi or Hive according to different strategies. The execution engine is responsible for executing the SQL query and returning the result. After execution is complete, the module performs universal serialization on the returned results. The results will be formatted into a unified JSON structure to facilitate user understanding and use. The serialized results will contain fields such as query status, data content, success information, and total number of records. An optional example is as follows:

[0211]

[0212]

[0213] This design greatly reduces the threshold for non-technical users to access structured data and simplifies the data extraction process.

[0214] (3) Unified management platform Figure 6 As shown, the present invention provides a unified management platform (Unified Management Platform), which specifically includes the following modules:

[0215] 1. User authentication module: supports multiple advanced authentication mechanisms, such as LDAP, OAuth 2.0, SAML, etc. After the user logs in, the system generates a unique token for subsequent identity authentication to ensure the security and effectiveness of data access.

[0216] 2. API Management Module

[0217] Users can manage all shared APIs in this module, including APIs for unstructured and structured data, and support viewing, editing, and deleting API settings. A visual interface is provided to facilitate users to efficiently manage their data interaction interfaces.

[0218] 3. Data sharing and link generation module: This module is responsible for the management and generation of data sharing links for users. Users can select multiple sharing APIs and generate unified data sharing links in a process-based manner.

[0219] Unstructured data: Convert user requests into S3 download links. Uploading and retrieving existing data must be approved and configured on the platform to ensure the security and compliance of data sharing.

[0220] Structured data: The table structure data is parsed and converted through the data conversion module, and finally displayed in the form of a response body, which users can directly view and operate.

[0221] 4. Interface beautification and development module

[0222] Allow users to fill in custom CSS and JavaScript to beautify the display interface of the shared link. Provide an interface so that the user's JS can interact with the platform functions, support data operations and the realization of dynamic effects. There is also a development and debugging page where users can preview and adjust their designs in real time.

[0223] 5. Data conversion module

[0224] Provides parsing and conversion functions for structured and unstructured data. The platform ensures that user requests are processed correctly and generates corresponding sharing links. This module is responsible for formatting and structuring data, allowing users to share their data efficiently. The platform not only supports sharing of unstructured and structured data, but also supports personalized page display and API docking. Users can customize the appearance and interaction of data display pages through predefined CSS and JavaScript, further improving the flexibility and adaptability of data sharing.

[0225] (4) Highly customizable page display function

[0226] Users can easily generate sharing links to share data through the unified management platform. The platform supports users to achieve personalized display by customizing page styles, and users can meet diverse display needs without developing additional modules. This function is suitable for a variety of application scenarios, such as project collaboration, data exchange and results display, and can effectively improve the visual effect and interactive experience of data sharing.

[0227] After selecting the data to be shared, users can generate a sharing link through simple operations. At the same time, the platform allows users to customize the validity period of the link, providing multiple options (such as 1 hour, 1 day, 1 week, etc.) to flexibly manage data access. Users can also set access rights and choose public or restricted access to ensure the security and privacy of information.

[0228] During the data sharing process, the system will automatically record access logs, detailing the visitor's IP address, access time, and operation type. When the validity period of the shared link expires, the system will automatically mark the link as invalid to prevent unauthorized access. This series of functions not only improves the user's operational convenience, but also effectively ensures the security of data, and with the intuitive user interface, provides users with a smooth operating experience.

[0229] The API interface provided in the example of this application is developed based on the Go language and seamlessly connects to the Hadoop cluster. The service uses a customized Hive driver (Unified Engine) to communicate with the Hive engine (HiveEngine) and the Spark engine (SparkEngine), and by identifying special placeholder structures and maintaining corresponding relationships, the user request parameters are converted into query SQL statements and executed to ensure the efficiency of structured data queries. The API interface can be lightweight deployed through container technologies such as Docker, which is convenient for maintenance and expansion. For example, when an enterprise needs to deploy the API interface in different server environments, it can use Docker containers for rapid deployment, and can adjust the configuration parameters of the container according to actual needs to meet the operating requirements in different environments.

[0230] It can be seen from the above embodiments that the highly customizable big data sharing service based on the Restful style and its implementation method proposed in the example of this application can effectively meet the diverse needs of enterprises in data management, sharing and analysis, improve the flexibility, convenience and customizability of data sharing, and provide strong support for the data operation of enterprises.

[0231] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0232] Corresponding to the distributed data processing method described in the above embodiment, Figure 7 is a schematic diagram of the structure of a distributed data processing device provided in an embodiment of the present application. The device can be implemented as part or all of a computer device by software, hardware, or a combination of both. The computer device can be Figure 8 Electronic equipment shown.

[0233] Reference Figure 7 , the distributed data processing device comprises:

[0234] The request unit 601 is used to respond to the data query request received by the distributed file Hadoop platform and determine whether the data query request is a structured data query request or an unstructured data query request;

[0235] The unstructured query unit 602 is used for querying the unstructured data stored in the Hadoop platform based on the unstructured data query request through a standard unified API interface if the data query request is an unstructured data query request, wherein the Hadoop platform is obtained by extending the distributed file system HDFS based on the open source file system JuiceFS, and the standard unified API interface is a presentation layer state transfer RESTful interface constructed or provided by JuiceFS;

[0236] A structured query unit 603 is configured to query the structured data stored in the Hadoop platform based on the structured data query request through a custom programming API interface if the data query request is a structured data query request, wherein the custom programming API interface is implemented based on an open source programming language;

[0237] The returning unit 604 is used to return the query result generated based on the structured data or unstructured data in a predetermined data format, and the predetermined data format includes: JSON data format.

[0238] Further, based on any of the above embodiments, as an example of the present application, the structured query unit includes:

[0239] A custom programming API interface module is used to parse the structured data query request, obtain the query parameters and SQL statement placeholders in the structured data query request, and send the query parameters and SQL statement placeholders to the data query Hive driver module;

[0240] The Hive driver module is used to convert the query parameters into a structured query SQL statement according to the SQL statement placeholder;

[0241] The data query engine is used to use the above SQL statements to query the data warehouse Hive tool of the above Hadoop platform to query and obtain the structured data stored in the above data warehouse Hive tool. The above data query engine includes: Hive engine or Spark engine.

[0242] Further, based on any of the above embodiments, as an example of this application,

[0243] The Hive driver module is further used to match the SQL statement placeholder with the query parameter using a regular expression to determine whether a key value corresponding to the SQL statement placeholder exists in the query parameter; if a key value corresponding to the SQL statement placeholder exists in the query parameter, the SQL statement placeholder is replaced with the key value to obtain the SQL statement.

[0244] Further, based on any of the above embodiments, as an example of the present application, the unstructured query unit is specifically used to query the metadata corresponding to the unstructured data query request in the key-value pair storage database in the Hadoop platform through a standard unified API interface; and query the unstructured data corresponding to the metadata from the HDFS system, wherein the standard unified API interface includes: a storage service S3 protocol interface, and each file data in the HDFS system is divided into one or more data blocks Chunk for storage.

[0245] Further, based on any of the above embodiments, as an example of the present application, the above device also includes:

[0246] The display unit is used to respond to the query result display request, determine the page display style that the user has pre-customized based on the style sheet language CSS and / or the web page development language JS script; and use the above page display style to display the query results in the above predetermined data format.

[0247] Further, based on any of the above embodiments, as an example of the present application, the above device also includes:

[0248] The sharing unit is used to determine the validity period and access rights of the data sharing link pre-customized and generated by the above-mentioned user in response to the user's sharing link customization operation; generate the data sharing link of the above-mentioned user according to the above-mentioned validity period and the above-mentioned access rights; in response to the visiting user clicking on the above-mentioned data sharing link within the above-mentioned validity period, share and display the query results in the above-mentioned predetermined data format, wherein the above-mentioned visiting user is a user with the above-mentioned access rights.

[0249] Further, based on any of the above embodiments, as an example of the present application, the above device also includes:

[0250] A recording unit, used to record the access log of the access user to the query result in the process of sharing and displaying the query result in the predetermined data format, wherein the access log includes: the IP address, access time and operation type of the access user;

[0251] The access prohibiting unit is used to mark the data sharing link as invalid when the validity period of the data sharing link expires, wherein the data sharing link in the invalid state is prohibited from being accessed.

[0252] It should be noted that any implementation of the distributed data processing device corresponds to any implementation of the distributed data processing method. The technical effects corresponding to any implementation of the distributed data processing device can refer to the technical effects corresponding to any implementation of the above-mentioned distributed data processing method, which will not be repeated here.

[0253] The distributed data processing device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0254] The functional units and modules in the above embodiments may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit, and the above integrated units may be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0255] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0256] An embodiment of the present application further provides an electronic device, the electronic device comprising one or more processors and a memory;

[0257] The memory is coupled to one or more processors, and the memory is used to store computer program codes, the computer program codes include computer instructions, and one or more processors call the computer instructions to enable the electronic device to execute the distributed data processing method shown above.

[0258] Figure 8The schematic diagram of the structure of an electronic device provided in the embodiment of the present application is that the electronic device 700 can be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, or a communication device such as a server, a storage device, a base station, or a smart car, etc. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

[0259] The memory 701 can be used to store computer software programs 702 and modules, and the processor 703 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 701. The memory 701 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the memory 701 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0260] Among them, the processor 703 may include one or more processors such as a central processing unit, an application processor (AP), a baseband processor, etc. The processor may be the nerve center and command center of the wireless router. The processor 703 may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions. The memory 701 may be used to store computer executable program codes, and the executable program codes include instructions. The processor 703 executes various functional applications and data processing of the network device by running the instructions stored in the memory. The memory 701 may include a program storage area and a data storage area, such as storing data of a sound signal to be played. For example, the memory may be a double rate synchronous dynamic random access memory DDR or a flash memory Flash.

[0261] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored; when the computer-readable storage medium is run on an electronic device, the electronic device executes the distributed data processing method shown above.

[0262] The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0263] The embodiment of the present application also provides a computer program product including computer instructions. When the computer program product is run on an electronic device, the electronic device can execute the distributed data processing method shown above.

[0264] The computer storage medium and computer program product provided in the above-mentioned embodiments of the present application are used to execute the method provided above. Therefore, the beneficial effects that can be achieved can refer to the corresponding beneficial effects of the method provided above, and will not be repeated here.

[0265] In the above embodiments, it can also be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (such as: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (such as: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)).

[0266] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0267] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments applied for herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0268] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0269] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0270] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A distributed data processing method, characterized in that: include: In response to a data query request received by a distributed file Hadoop platform, determining that the data query request is a structured data query request or an unstructured data query request, wherein the Hadoop platform is obtained by extending a distributed file system HDFS based on an open source file system JuiceFS; If the data query request is an unstructured data query request, query the unstructured data stored in the Hadoop platform based on the unstructured data query request through a standard unified API interface, wherein the standard unified API interface is a presentation layer state transfer RESTful interface constructed or provided by JuiceFS; If the data query request is a structured data query request, querying the structured data stored in the Hadoop platform based on the structured data query request through a custom programming API interface, wherein the custom programming API interface is implemented based on an open source programming language; Return the query result generated based on the structured data or the unstructured data in a predetermined data format, wherein the predetermined data format includes: JSON data format.

2. The method according to claim 1, characterized in that The querying of the structured data stored in the Hadoop platform based on the structured data query request through the custom programming API interface includes: Parsing the structured data query request through a custom programming API interface to obtain query parameters and SQL statement placeholders in the structured data query request; Send the query parameters and the SQL statement placeholder to a data query Hive driver module; By means of the Hive driver module, the query parameters are converted into a structured query SQL statement according to the SQL statement placeholder; The SQL statement is used to query the data warehouse Hive tool of the Hadoop platform to obtain the structured data stored in the data warehouse Hive tool.

3. The method according to claim 2, characterized in that The using the SQL statement to query the data warehouse Hive tool of the Hadoop platform to query and obtain the structured data stored in the data warehouse Hive tool includes: The data warehouse Hive tool of the Hadoop platform is queried by using the SQL statement through the data query Hive engine or the Spark engine to obtain the structured data stored in the data warehouse Hive tool.

4. The method according to claim 1, characterized in that: The Hive driver module converts the query parameters into a structured query SQL statement according to the SQL statement placeholder, including: By means of the Hive driver module, the SQL statement placeholder is matched with the query parameter using a regular expression to determine whether a key value corresponding to the SQL statement placeholder exists in the query parameter; If a key value corresponding to the SQL statement placeholder exists in the query parameter, the SQL statement placeholder is replaced with the key value to obtain the SQL statement.

5. The method according to claim 1, characterized in that The querying of the unstructured data stored in the Hadoop platform based on the unstructured data query request through the standard unified API interface includes: Through the standard unified API interface, query the metadata corresponding to the unstructured data query request in the key-value pair storage database in the Hadoop platform, wherein the standard unified API interface includes: a storage service S3 protocol interface; Unstructured data corresponding to the metadata is queried from the HDFS system, wherein each file data in the HDFS system is divided into one or more data blocks Chunks for storage.

6. The method according to any one of claims 1 to 5, characterized in that After returning the query result generated based on the structured data or the unstructured data in a predetermined data format, the method further includes: In response to a query result display request, determine a page display style that is pre-customized and generated by a user based on a style sheet language CSS and / or a web page development language JS script; use the page display style to display the query result in the predetermined data format; and / or In response to a user's sharing link customization operation, determine the validity period and access rights of the data sharing link pre-customized and generated by the user; generate the user's data sharing link based on the validity period and the access rights; in response to the visiting user clicking on the data sharing link within the validity period, share and display the query results in the predetermined data format, wherein the visiting user is a user with the access rights.

7. The method according to claim 6, characterized in that The method further comprises: In the process of sharing and displaying the query result in the predetermined data format, recording the access log of the access user to the query result, wherein the access log includes: the IP address, access time and operation type of the access user; When the validity period of the data sharing link expires, the data sharing link is marked as invalid, wherein the data sharing link in the invalid state is prohibited from being accessed.

8. A distributed data processing device, characterized in that: include: A request unit, configured to respond to a data query request received by the distributed file Hadoop platform and determine whether the data query request is a structured data query request or an unstructured data query request; An unstructured query unit, configured to query the unstructured data stored in the Hadoop platform based on the unstructured data query request through a standard unified API interface if the data query request is an unstructured data query request, wherein the Hadoop platform is obtained by extending the distributed file system HDFS based on the open source file system JuiceFS, and the standard unified API interface is a presentation layer state transfer RESTful interface constructed or provided by JuiceFS; A structured query unit, configured to query the structured data stored in the Hadoop platform based on the structured data query request through a custom programming API interface if the data query request is a structured data query request, wherein the custom programming API interface is implemented based on an open source programming language; The returning unit is used to return the query result generated based on the structured data or the unstructured data in a predetermined data format.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed, causes the method according to any one of claims 1 to 7 to be performed.

Images