Server big data platform service expansion method

Abstract

The invention discloses a server big data platform service capacity expansion method, which comprises the following processes: hardware and software environment preparation is carried out, and prepared server hardware equipment comprises at least two solid-state disks and at least eight common disks; operating system deployment and disk configuration are carried out on the server, a solid-state disk is set to be in an RAID1 mode, and a common disk is set to be in a JBOD mode; performing disk partitioning and system configuration optimization, and closing a swap partition; a daemons service and an agent service are installed, and a configuration file is modified to set service parameters; the method comprises the following steps: performing data processing, acquiring data from a data source by a daemons service, transmitting the acquired data through an agent service, storing the transmitted data into a distributed file system and a relational database, establishing an index, and finally analyzing and processing the data by utilizing a distributed computing framework and outputting a result. A server big data platform service expansion process can be realized.

Description

Technical Field

[0001] This invention belongs to the field of cloud computing and relates to a method for expanding the service capacity of a server big data platform. Background Technology

[0002] With the advent of the big data era, enterprises are experiencing explosive growth in data volume. After a period of operation, the computing power and storage capacity of existing big data platforms often reach their bottlenecks, failing to meet the ever-increasing business demands. Therefore, expanding the service capacity of big data platforms, i.e., adding new server nodes, has become a routine and critical operational task.

[0003] However, existing server expansion methods generally have the following drawbacks: Inefficient and time-consuming server expansion: Traditional server expansion processes rely heavily on manual operations. Maintenance personnel need to manually install the operating system, configure the network, partition the disk, and install various dependent software and core services for each new server. The entire process is cumbersome and time-consuming. When expanding to include dozens or even hundreds of servers at once, this manual approach becomes a major bottleneck, unable to quickly respond to the growing business demands.

[0004] Inconsistent configurations pose stability risks: During manual configuration, inconsistencies can easily arise between server nodes due to negligence or operational differences. For example, subtle differences in operating system versions, kernel parameters, software dependency library versions, and service configuration files can trigger difficult-to-diagnose "ghost" issues in the cluster, leading to service instability and even data processing errors, seriously jeopardizing the stability of the entire platform.

[0005] Unoptimized system performance and low resource utilization: Generalized server installation processes typically do not undergo in-depth optimization for big data application scenarios. For example, improper configuration of disk arrays (RAID / JBOD), failure to disable performance-impacting swap partitions, and lack of tuning of network and file system parameters will all lead to underutilization of server hardware resources, resulting in significant performance degradation for big data platforms when handling high-concurrency read / write and large-scale computing tasks.

[0006] Disjointed data flow and insufficient security: During capacity expansion, the addition of new nodes often focuses only on building the basic environment, neglecting seamless integration with the entire data processing workflow. The configurations of data collection, transmission, storage, and analysis are relatively independent, lacking unified security strategies and efficient transmission mechanisms. For example, data may not be encrypted during transmission, or there may be a lack of effective breakpoint resumption mechanisms when the network is interrupted. This not only affects data processing efficiency but also introduces the security risk of data leakage. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for expanding server big data platform services, which can realize an automated, standardized, high-performance and secure server big data platform service expansion process.

[0008] To achieve the above objectives, the present invention employs the following technical solution: A method for expanding the service capacity of a server big data platform includes the following processes: Prepare the hardware and software environment, including at least two solid-state drives and at least eight regular disks for the server hardware. Deploy the operating system and configure the disks on the server, setting the solid-state drive to RAID1 mode and the regular disk to JBOD mode; Perform disk partitioning and system configuration optimization, and disable the swap partition; Install the daemons and agent services, and modify the configuration files to set the service parameters; For data processing, the daemons service collects data from the data source, transmits the collected data through the agent service, stores the transmitted data in a distributed file system and a relational database, and creates indexes. Finally, it uses a distributed computing framework to analyze and process the data and output the results.

[0009] Preferably, when preparing the hardware and software environment, Deepin Remote Boot Manager is installed, and the server is connected through Deepin Remote Boot Manager for automated deployment.

[0010] Preferably, after connecting to the server through the Deep Remote Boot Manager, authentication can be performed using an SMS verification code before logging in.

[0011] Preferably, when performing disk partitioning and system configuration optimization, it also includes modifying the hostname and hosts file of the host, replacing it with a domestic yum source, disabling SELinux and IPv6 login, disabling the firewall, configuring and starting the NTP time synchronization service, and mounting the disk.

[0012] Preferably, after installing the daemons and agent services, the system also configures the services to start automatically on boot and performs an initialization test.

[0013] Preferably, when the daemons service collects data from data sources, it uses multi-threading technology to concurrently process collection tasks from multiple data sources and performs preliminary data cleaning during the collection process.

[0014] Preferably, when the agent service transmits data, it uses an encrypted TCP / IP protocol combined with a custom compression algorithm, and utilizes breakpoint resume technology during transmission.

[0015] Preferably, the custom compression algorithm is an adaptive algorithm that samples and analyzes data blocks and selects the compression algorithm according to the type of data block.

[0016] Preferably, when creating indexes, B-Tree indexes and bitmap indexes are created based on data characteristics and query requirements.

[0017] Preferably, the data analysis and result output also includes quality assessment of the processing results, and finally outputting the processing results in the form of a report.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention automates the entire process from operating system installation and disk configuration to service deployment by employing a deep remote boot manager and automated configuration scripts. Maintenance personnel can complete batch and standardized expansion operations without manual intervention on each server, reducing the expansion cycle from days to hours, significantly improving operational efficiency and enabling rapid support for the elastic expansion needs of businesses.

[0019] This invention ensures that all newly added server nodes maintain strict consistency in hardware configuration (such as RAID1 / JBOD mode), operating system, kernel parameters, software environment, and service configuration through a unified automated process. This fundamentally eliminates the risk of configuration inconsistencies caused by human error, avoids potential operational failures, and significantly enhances the stability and reliability of the entire big data platform.

[0020] This invention proposes a series of in-depth optimization measures to address the high concurrency and high throughput characteristics of big data applications. For example, it employs solid-state drives to build a RAID1 array to improve system response speed and reliability, disables the SWAP partition to avoid performance jitter, and adjusts network and file system parameters to adapt to large data read / write loads. These optimization measures fully utilize the server's hardware performance, thereby effectively improving the computing and I / O performance of the entire big data platform.

[0021] This invention not only completes the server deployment but also constructs an end-to-end data processing workflow from data acquisition, encrypted transmission, efficient storage to distributed analysis. It employs multi-factor authentication to ensure access security, encryption and adaptive compression algorithms to guarantee secure and efficient data transmission, and breakpoint resumption technology to ensure data integrity. This integrated design ensures the secure and efficient flow of data within the expanded platform architecture, providing solid and reliable support for upper-layer data applications. Attached Figure Description

[0022] Figure 1 This is a flowchart of the server big data platform service expansion method of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] like Figure 1 As shown in this embodiment, a method for expanding the capacity of a server big data platform service includes the following steps: The prerequisites are: ensure the hardware meets the requirements; the server must have at least two solid-state drives (SSDs) and eight regular-purpose hard drives, and the network connection of the hardware must be stable with sufficient network bandwidth to support data transmission. Additionally, prepare third-party tools for connecting to the server, such as Deepin Remote Boot Manager, and ensure that the tool is correctly installed and functioning properly.

[0026] 1. Hardware and software environment preparation: Based on the high-concurrency read / write and massive storage requirements of the big data platform, carefully selected server hardware equipment met the requirements. Two high-performance solid-state drives (SSDs) offer low latency and high random read / write performance, primarily used for the operating system and frequently accessed core data storage; eight large-capacity hard disks (HDDs) focus on providing low-cost, massive storage, suitable for archived and cold data storage. Network equipment selected includes switches and network interface cards supporting 10 Gigabit Ethernet to ensure high-speed and stable data transmission. Furthermore, a deep remote boot manager (such as an iPXE+TFTP solution) is installed as the server connection software. This software supports diskless boot technology, facilitating unified deployment and management of large-scale servers.

[0027] Specifically, in hardware selection, solid-state drives (SSDs) are preferably NVMe (Non-Volatile Memory Express) interface types to maximize I / O throughput. Network equipment includes not only 10 Gigabit switches and network interface cards (NICs), but also pre-configured VLANs (Virtual Local Area Networks) to logically isolate the big data computing network from the management and business networks, reducing the risks of network storms and unauthorized access. The Deep Remote Boot Manager works collaboratively with a TFTP (Trivial File Transfer Protocol) server, a DHCP (Dynamic Host Configuration Protocol) server, and an HTTP server. The DHCP server assigns an IP address to the new server and informs the TFTP server of its address. The new server loads the iPXE boot program from the TFTP server via the PXE (Preboot Execution Environment) firmware. The iPXE then retrieves the operating system installation image and automated installation configuration files (such as Kickstart files) from the HTTP server according to a preset script, thus achieving zero-contact automated deployment.

[0028] 2. Secure login and identity verification: After establishing a connection with the backend server using Deepin Remote Boot Manager, login is performed via an HTML page. To ensure login security, SMS verification code authentication is used. The system will send a verification code containing 6 random digits to a pre-bound mobile phone number. Users must enter the correct verification code within a specified time (e.g., 5 minutes) to successfully log in, effectively preventing unauthorized users from intruding into the system.

[0029] To further enhance security, the HTML page for user login is provided via HTTPS (Hypertext Transfer Protocol Secure), ensuring that login credentials are encrypted using TLS / SSL (Transport Layer Security / Secure Sockets Layer) during transmission. The SMS verification code generation and verification process is as follows: The backend server generates a unique token containing a 6-digit number and a timestamp. The digit portion is sent to the user's mobile phone via the integrated SMS gateway API, while the token is stored in a high-speed cache database such as Redis with a 5-minute expiration time. After the user submits the verification code, the backend server verifies its match with the cached token and its validity. Upon successful verification, the token is destroyed, ensuring one-time use and preventing replay attacks.

[0030] 3. Operating system deployment and disk configuration: Deploy a CentOS 7 system on the server. To improve the reliability of the system disk and data security, two solid-state drives (SSDs) are configured in RAID 1 mode. This mode achieves redundancy through data mirroring; if one hard drive fails, the other can continue to provide service, ensuring continuous system operation. Eight regular disks are configured in JBOD mode to maximize disk capacity and meet the needs of large data storage. After system installation, carefully check the system's automatic optimization parameters, such as file system caching strategies and process scheduling algorithms, and fine-tune them according to the actual application scenario to ensure the system is in optimal operating condition.

[0031] 4. Disk partitioning and system configuration optimization: Disk partitioning: Partition the disk appropriately according to data storage and application needs. For example, allocate separate partitions for the system root directory ( / ), user home directories ( / home), and temporary file directories ( / tmp), while reserving some space for future expansion.

[0032] During the disk partitioning phase, the XFS file system is used to format the partitions because it offers better performance support for large files and high-concurrency read / write operations. The specific partitioning scheme is as follows: On the RAID1 array, a / partition (50GB), a / boot partition (1GB), a / var partition (100GB, used for log storage), and a separate physical volume are created for potential future expansion. The remaining 8 JBOD disks are initialized as physical volumes and added to a large volume group. Logical volumes will be created from this volume group and mounted to directories such as / data / 1, / data / 2, etc., according to the needs of the data nodes.

[0033] 5. User Account Management: Create user accounts with different permissions. For example, the administrator account has the highest privileges and can perform system configuration and management operations; the data processing account only has data read, write and processing permissions, restricting its access to core system files to ensure system security.

[0034] User account management follows the principle of least privilege. For example, a user and user group named "hadoop" are created specifically for running big data components such as Hadoop and Spark. This user's home directory is set on the data disk, and read, write, and execute permissions are granted to the relevant data directories. Simultaneously, by configuring the sudoers file, the administrator commands that the "hadoop" user can execute are strictly limited, allowing only necessary commands such as starting and stopping services and viewing configurations, while prohibiting modification of critical system-level configurations.

[0035] 6. Network and service configuration: Modify the hostname and hosts files of the host machine to facilitate host identification and communication in the network environment; replace with a high-speed domestic yum repository to speed up package download and installation; disable SELinux and IPv6 login to avoid service anomalies caused by security policies or network configuration issues; disable the firewall (or configure firewall rules according to actual needs) to ensure smooth communication between various components of the big data platform; configure and start the NTP time synchronization service to ensure the accuracy of the system time and avoid data synchronization problems caused by time inconsistencies; disable the swap partition to prevent frequent use of disk swap space when memory is insufficient, which would affect system performance; mount disks according to data storage needs, such as mounting large-capacity storage disks to specified data storage directories.

[0036] This step is performed using an automated script (such as an Ansible Playbook or a shell script) to ensure consistency in configuration across all new servers. For example, by executing `hostnamectl set-hostname`. <new-hostname>Set the hostname. Turn off SELinux by modifying the SELinux parameter in the / etc / selinux / config file to disabled and restart. The specific command to turn off the firewall is systemctl stop firewalld and systemctl disable firewalld. When configuring the NTP service, specify the internal NTP master server address in the / etc / chrony.conf file, and execute systemctl start chronyd and systemctl enable chronyd. To turn off the swap partition, execute the swapoff -a command and comment out all swap type mount lines in the / etc / fstab file, as big data applications such as Hadoop and HBase manage memory themselves, and using swap will severely reduce performance.

[0037] 7. Service installation and configuration: Use the rpm command to install daemons and agent services, which are the core components of data collection and transmission in the big data platform. After installation, carefully modify the config.ini configuration file to set service parameters, including service listening address, port number, database connection information, etc. Perform the service startup setting to ensure that the service can automatically run after system restart, and perform initialization testing to check whether the service starts normally and whether communication with related components is normal.

[0038] Before service installation, first configure the local Yum repository or point to the internal Nexus / Artifactory server to accelerate rpm package installation and perform version control. The modification of the config.ini configuration file is also completed by an automated tool, which dynamically generates the final configuration file from the configuration template and variable list according to the server's role (e.g., data collection node, data calculation node). For example, for the daemons service, the source_db_host parameter in its configuration file will be filled according to the data source that the current node needs to collect. The service startup at boot is implemented by the systemctl enabledaemons.service and systemctl enable agent.service commands. Initialization testing includes checking whether the service process exists, whether the listening port is in the LISTEN state (using the netstat -tulnp command), and checking the service startup log (using journalctl -u<service_name>) to confirm that there is no error information output.

[0039] 8. Data processing whole process: Data collection: During data processing, daemons service collects data from multiple databases (such as MySQL, Oracle) and file systems (such as NFS, CIFS) of the enterprise. To improve collection efficiency, multi-threading technology is used, which can concurrently process multiple data source collection tasks, significantly shortening data collection time. During the collection process, preliminary cleaning of data is performed, such as removing duplicate data and handling missing values, to improve data quality.

[0040] Specifically, daemons service has a configurable thread pool built-in, the size of which is dynamically set according to the number of server CPU cores. When the service starts, it will load multiple data source collection plugins, each of which is responsible for a type of data source (such as MySQL plugin, Oracle plugin). Each collection task is executed as an independent thread in the thread pool. The rules for preliminary data cleaning are defined in an external JSON or XML file, and daemons service loads these rules in real time while collecting data to filter and convert the data stream, for example, using regular expressions to match and correct non-standard date formats, or filling missing numerical fields with default value 0.

[0041] Data transmission: The collected data is transmitted by agent service using encrypted TCP / IP protocol combined with custom compression algorithm. The encryption protocol ensures the security of the data during transmission, preventing data leakage; the custom compression algorithm can effectively reduce data transmission volume and reduce network bandwidth occupancy. During transmission, the resume transmission technology is used, which can continue transmission from the breakpoint when the network is interrupted or transmission is abnormal, ensuring data integrity and no loss.

[0042] The service first preprocesses the batch of collected data blocks before transmission. The custom compression algorithm is an adaptive algorithm that first samples and analyzes the data blocks to determine whether they are text type data or binary type data. For text data, high-speed compression algorithms such as Snappy or LZ4 are preferred; for structured or highly repetitive data, higher compression ratio algorithms such as Gzip are used, thereby balancing compression ratio and CPU overhead. The resume transmission technology generates an MD5 checksum for each data block at the sending end and records the offset of the successfully sent data block. The receiving end checks the integrity of the data block after receiving it and confirms to the sending end. When the transmission is interrupted, the agent service reinitiates the transmission request from the last successful confirmation offset.

[0043] Data storage and indexing: The transmitted data is stored in a distributed file system (such as HDFS) and a relational database (such as PostgreSQL). To improve data query efficiency, indexes such as B-Tree index and bitmap index are established during storage according to data characteristics and query requirements.

[0044] Data is distributed to different storage systems according to its characteristics. Unstructured or semi-structured raw log data is stored in HDFS in columnar storage formats such as Parquet or ORC, which naturally support compression and predicate pushdown, greatly improving the efficiency of subsequent Spark analysis. At the same time, key indicators and dimension information extracted from raw data are loaded into tables in PostgreSQL. In PostgreSQL, not only B-Tree indexes are created for high-cardinality columns (such as user ID), but also Bitmap Indexes are created for low-cardinality columns (such as gender, region) to optimize the performance of different types of aggregation queries.

[0045] Data analysis and result output: Use the Spark distributed computing framework to analyze and process data. Spark has strong in-memory computing capabilities and can quickly process large-scale data. Use data mining algorithms (such as clustering analysis and association rule mining) to extract valuable information. Quality assessment of the processing results is performed by judging data accuracy, completeness, consistency, etc. to ensure the reliability of the results. Finally, the processing results are output in the form of reports to the enterprise's network shared folder for relevant personnel to query and use, providing data support for enterprise decision-making.

[0046] The above sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0047] In the above embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0048] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the device embodiment described above is only schematic. For example, the division of the units can be a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection between each represented or discussed element can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other form.

[0049] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or also can be distributed to multiple units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to the actual needs.

[0050] The above description is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, can make several improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

[0051] It should be understood that the above description is for illustration only and is not intended to be limiting. Many embodiments and many applications other than those described herein will be readily apparent to those skilled in the art from this description. The scope of the application should therefore not be determined with reference to the above description, but instead should be determined with reference to the appended claims along with their full scope of equivalents. For purposes of completeness, all articles and references, including patents and patent documents, are incorporated herein by reference in their entirety. Nothing herein is to be construed as an admission that the subject matter disclosed herein is not entitled to antedate such disclosure by virtue of prior application.

Claims

1. A method for expanding the service capacity of a server big data platform, characterized in that, Includes the following processes: Prepare the hardware and software environment, including at least two solid-state drives and at least eight regular disks for the server hardware. Deploy the operating system and configure the disks on the server, setting the solid-state drive to RAID1 mode and the regular disk to JBOD mode; Perform disk partitioning and system configuration optimization, and disable the swap partition; Install the daemons and agent services, and modify the configuration files to set the service parameters; For data processing, the daemons service collects data from the data source. The collected data is then transmitted through the agent service. After transmission, the data is stored in a distributed file system and a relational database, and an index is created. Finally, a distributed computing framework is used to analyze and process the data and output the results.

2. The server big data platform service expansion method according to claim 1, characterized in that, When preparing the hardware and software environment, install Deepin Remote Boot Manager, connect to the server through Deepin Remote Boot Manager, and perform automated deployment.

3. The server big data platform service expansion method according to claim 1, characterized in that, After connecting to the server through Deepin Remote Boot Manager, you can log in only after verifying your identity with an SMS verification code.

4. The server big data platform service expansion method according to claim 1, characterized in that, Disk partitioning and system configuration optimization also include modifying the hostname and hosts file of the host, replacing it with a domestic yum repository, disabling SELinux and IPv6 login, disabling the firewall, configuring and starting the NTP time synchronization service, and mounting the disk.

5. The server big data platform service expansion method according to claim 1, characterized in that, After installing the daemons and agent services, the system also configures these services to start automatically on boot and performs an initialization test.

6. The server big data platform service expansion method according to claim 1, characterized in that, When the daemons service collects data from data sources, it uses multi-threading technology to concurrently process collection tasks from multiple data sources and performs preliminary data cleaning during the collection process.

7. The server big data platform service expansion method according to claim 1, characterized in that, When transmitting data, the agent service uses an encrypted TCP / IP protocol combined with a custom compression algorithm and utilizes breakpoint resume technology during transmission.

8. The server big data platform service expansion method according to claim 1, characterized in that, The custom compression algorithm is an adaptive algorithm that samples and analyzes data blocks and selects the compression algorithm based on the type of data block.

9. The server big data platform service expansion method according to claim 1, characterized in that, When creating indexes, B-Tree indexes and bitmap indexes are created based on data characteristics and query requirements.

10. The server big data platform service expansion method according to claim 1, characterized in that, The data analysis and results output also include quality assessment of the processing results, and finally outputting the processing results in the form of reports.

Images