A book data acquisition cloud data processing system

Abstract

This invention discloses a cloud-based data processing system for book data acquisition, belonging to the field of data processing. The system includes: a data acquisition module, a cloud storage module, a data processing engine, an intelligent analysis module, and a security protection module. The data acquisition module supports automatic acquisition and preprocessing of multi-source heterogeneous book data; the cloud storage module adopts a distributed storage architecture to achieve highly reliable data storage and elastic expansion; the data processing engine, based on the MapReduce framework, performs parallel processing and cleaning of the acquired book data; the intelligent analysis module utilizes natural language processing and machine learning algorithms to deeply mine and classify book content; and the security protection module implements data encryption, access control, and privacy protection. This invention achieves efficient acquisition, processing, and analysis of book data through a cloud architecture, solving the problems of low efficiency and poor scalability in traditional book data processing, and has advantages such as strong processing capabilities, low storage costs, and high intelligence.

Description

Technical Field

[0001] This invention relates to data processing, and more particularly to a cloud-based data processing system for book data acquisition. Background Technology

[0002] With the rapid development of digital library construction and the e-book industry, book data is experiencing explosive growth. Currently, massive amounts of book data are scattered across various data sources, including traditional library management systems, major e-book platforms (such as Amazon Kindle and Dangdang eBooks), open-access digital libraries (such as Google Books and Project Gutenberg), and various academic databases (such as CNKI, Wanfang, and IEEE Xplore). These data sources use diverse formats, including PDF, EPUB, MOBI, TXT, Word documents, and structured metadata records (such as MARC and Dublin Core).

[0003] However, existing book data processing solutions have the following technical problems: (1) Low data collection efficiency and high integration difficulty. Traditional book data collection methods mainly rely on manual input or single web crawler tools, which are difficult to cope with the collection needs of multiple heterogeneous data sources. Different data sources have different interface standards, data formats and access permissions, requiring the development of special adapters, which increases the complexity of system development and maintenance costs.

[0004] (2) Poor data storage scalability and high cost. Traditional relational databases are difficult to store and process massive amounts of full-text book data, especially unstructured data. With a centralized storage architecture, the system's scalability is limited, and it cannot be elastically expanded according to the growth of data volume, resulting in high storage and maintenance costs.

[0005] (3) Insufficient data processing capabilities and low efficiency. Data cleaning, format conversion, and quality assessment of book data are inefficient on single machines or small-scale clusters, making it difficult to meet real-time or near-real-time processing requirements. In particular, batch processing of large-scale book data often takes several days or even weeks using traditional methods.

[0006] (4) Insufficient depth of data mining and low level of intelligence. The existing system mainly focuses on metadata management and retrieval when utilizing book data, lacking in-depth mining and intelligent analysis of the book content itself. It is difficult to achieve semantic understanding, knowledge extraction and intelligent recommendation of book content, and cannot meet users' needs for knowledge discovery and intelligent services.

[0007] (5) Insufficient data security protection and high privacy risks. Book data often contains users' reading behavior data and personal privacy information. The existing data encryption and access control mechanisms are relatively simple and difficult to cope with complex security threats and compliance requirements.

[0008] Therefore, there is an urgent need for a cloud data processing system that can efficiently collect, store, process, and analyze book data in order to solve the above-mentioned technical problems and enhance the utilization value and service capabilities of book data. Summary of the Invention

[0009] Purpose of the invention: The purpose of this invention is to provide a cloud data processing system for book data acquisition, which achieves efficient acquisition, elastic storage, parallel processing and intelligent analysis of book data through cloud architecture, and solves the problems of low data acquisition efficiency, poor storage scalability, weak processing capabilities and low level of intelligence in the existing technology.

[0010] Technical solution: A cloud data processing system for book data acquisition includes a data acquisition module, a cloud storage module, a data processing engine, an intelligent analysis module, and a security protection module.

[0011] Data Acquisition Module: Supports automatic acquisition of book data from multiple data sources (including book databases, e-book platforms, digital libraries, academic databases, etc.). This module adopts an adapter pattern, configuring a dedicated acquisition adapter for each data source, and supports various acquisition methods such as HTTP API interface, direct database connection, FTP file transfer, and web crawling. The acquired data includes basic book information (ISBN, title, author, publisher, publication date), metadata (abstract, keywords, classification number), full-text book data, and user review data. This module supports breakpoint resume, incremental acquisition, and distributed acquisition, and can handle large-scale concurrent acquisition tasks.

[0012] The cloud storage module employs a distributed storage architecture, combining the HDFS (Hadoop Distributed File System) distributed file system and distributed databases (such as HBase and MongoDB) to achieve tiered storage of book data. Structured data (basic book information and metadata) is stored in the distributed database, while unstructured data (full text and cover images) is stored in the HDFS file system. This module uses a three-tier storage strategy: hot data (frequently accessed) is stored in an in-memory database (Redis), warm data (periodically accessed) is stored in the distributed database, and cold data (rarely accessed) is stored in HDFS. It automatically migrates and archives data based on access frequency and time characteristics, reducing storage costs. This module also supports automatic data sharding, redundant backup (three replicas by default), and load balancing to ensure high availability and reliability of the data.

[0013] Data processing engine: Based on the MapReduce parallel computing framework, it enables parallel processing of collected book data. This engine comprises three core units: Data cleaning unit: Identifies and removes duplicate data using hash algorithms; detects and processes missing values ​​using statistical methods and rule engines (interpolation, default value filling); and identifies anomalous data (format errors, logical errors) using anomaly detection algorithms.

[0014] Format conversion unit: Supports mutual conversion between multiple document formats (PDF to TXT, EPUB to HTML, etc.) and generates a unified XML standard format.

[0015] Quality Inspection Unit: Performs quality assessment on the processed data, including integrity checks, consistency checks, and accuracy checks, and generates a data quality report.

[0016] Intelligent Analysis Module: Utilizing Natural Language Processing (NLP) and machine learning algorithms, this module performs in-depth mining and intelligent analysis of book content. It comprises three core units: Natural Language Processing Unit: Employs a BERT (Bidirectional Encoder Representations from Transformers) pre-trained model to perform word segmentation, part-of-speech tagging, named entity recognition (NER), syntactic analysis, and semantic understanding on book text, generating vector representations (embeddings) of the book content to support subsequent semantic retrieval and similarity calculation.

[0017] Knowledge Extraction Unit: Based on a deep learning-based entity relationship extraction model, it extracts entities such as names of people, places, organizations, keywords, topics, and events from book content, and constructs semantic relationships between entities (such as the relationship between "author"-"work"-"topic") to generate a structured knowledge base and a book knowledge graph.

[0018] Book classification unit: Combining multi-label classification algorithms (such as TextCNN, BERT classifier) ​​and topic models (such as LDA), books are classified and labeled in multiple dimensions, including subject classification (Chinese Library Classification, Dewey Decimal Classification), genre classification (novels, essays, popular science, history, etc.), reading difficulty level (beginner, intermediate, professional) and reader profile tags (children, teenagers, adults, professionals).

[0019] Security Protection Module: Ensures the security of book data during acquisition, transmission, storage, and processing. This module comprises three core units: Data encryption unit: It uses a combination of symmetric encryption (AES-256) and asymmetric encryption (RSA-2048) to encrypt the transmission (SSL / TLS) and storage of sensitive book data, and supports dynamic updates and rotation of keys.

[0020] Access Control Unit: Employs an attribute-based access control model (ABAC) that dynamically calculates access permissions and implements access policies based on multiple attributes such as user identity, role, time, location, and device, enabling fine-grained access control.

[0021] Audit Log Unit: Records all user operations, system access, data changes, and other log information, supports log querying, analysis, and tracing, and meets compliance audit requirements.

[0022] The modules communicate asynchronously and transfer data through message queue middleware (such as Kafka and RabbitMQ), building a high-concurrency, highly available cloud data processing platform. The system also includes a task scheduling module, which dynamically schedules data processing tasks based on system load and data priority, supporting task priority configuration, load balancing, and automatic fault recovery.

[0023] Beneficial effects: (1) This invention significantly improves the efficiency and coverage of book data collection through a multi-source heterogeneous data collection architecture and a distributed collection strategy.

[0024] (2) The present invention adopts distributed cloud storage and a three-level storage strategy to achieve elastic expansion and cost optimization of storage resources.

[0025] (3) Based on the MapReduce parallel computing framework, this invention realizes efficient processing of massive book data, improving processing efficiency by more than 10 times.

[0026] (4) This invention utilizes NLP and machine learning algorithms to achieve in-depth mining and knowledge discovery of book content, providing support for knowledge services and intelligent recommendations.

[0027] (5) This invention comprehensively protects data security and user privacy through encryption, access control and auditing mechanisms, and meets compliance requirements. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall architecture of the cloud data processing system for book data acquisition; Figure 2 This is a schematic diagram of the three-tier storage architecture of the cloud storage module; Figure 3 This is a flowchart of the data processing engine. Figure 4This is a flowchart of the book knowledge graph construction process in the intelligent analysis module. Detailed Implementation

[0029] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Example 1

[0030] like Figure 1 As shown, the present invention provides a cloud data processing system for book data acquisition. The system is deployed on a cloud platform (such as Alibaba Cloud, AWS, Tencent Cloud) and adopts a microservice architecture, including a data acquisition module, a cloud storage module, a data processing engine, an intelligent analysis module, a security protection module, and a task scheduling module.

[0031] The data acquisition module includes an acquisition scheduler, an adapter manager, and multiple acquisition adapters.

[0032] The data acquisition scheduler is responsible for scheduling and managing data acquisition tasks, supporting three modes: scheduled data acquisition, manually triggered data acquisition, and event-driven data acquisition. Based on priority queues and load balancing algorithms, the scheduler allocates data acquisition tasks to idle data acquisition adapters.

[0033] The Adapter Manager manages the lifecycle of all data acquisition adapters, including adapter registration, loading, unloading, and status monitoring. The Adapter Manager supports dynamic plugging and unplugging of adapters, allowing new data sources to be added without restarting the system.

[0034] A data acquisition adapter is a data acquisition component specific to a data source; each data source is configured with a dedicated adapter. In this embodiment, the system is configured with the following data acquisition adapters: The CNKI Academic Database Adapter accesses the CNKI Academic Database via an HTTP API interface to collect academic books, including academic papers and journal articles. This adapter supports paginated queries, batch downloads, and incremental updates.

[0035] Dangdang eBook Adapter: This adapter uses web crawling technology to scrape book information from the Dangdang.com eBook platform, including basic book information, metadata, synopsis, table of contents, and user reviews. It employs anti-crawler strategies (User-Agent rotation, IP proxy pool, and request frequency control) to avoid being blocked.

[0036] Project Gutenberg Adapter: Connects to Project Gutenberg's public FTP server via the FTP protocol to download full-text data of public domain e-books. This adapter supports resume download and incremental synchronization.

[0037] Local Library System Adapter: Connects to the local library's OPAC system via JDBC (Direct Database Connectivity) to collect basic information about the library's collection and borrowing data. This adapter supports multiple database types (MySQL, Oracle, SQL Server).

[0038] The raw data collected by the acquisition adapter is sent to the data processing engine for further processing via a message queue middleware.

[0039] like Figure 2 As shown, the cloud storage module adopts a three-tier storage architecture, including a hot data storage layer, a warm data storage layer, and a cold data storage layer.

[0040] Hot data storage layer: Employs a Redis in-memory database to store system configuration information, user session information, and frequently accessed book metadata (such as popular lists and new book recommendations). The hot data storage layer has a response time in the millisecond range and supports more than 100,000 concurrent accesses per second.

[0041] Warm data storage layer: Employs the HBase distributed database to store basic book information, metadata, index data, and knowledge graph data. HBase uses columnar storage, supporting fast querying and range scanning of massive amounts of data. Data is sharded and stored according to RowKey, which is designed using a combination of "book type, timestamp, and book ID" to ensure the continuity of related data storage and query efficiency.

[0042] Cold data storage layer: Employs the HDFS distributed file system to store full-text book data, raw acquisition files, historical log files, and backup data. HDFS uses a master-slave architecture, with the NameNode managing file system metadata and DataNodes storing the actual data blocks. The default data block size is 128MB, and each data block is stored in three replicas on different DataNodes to ensure data reliability.

[0043] The cloud storage module also includes a data migration manager, which monitors data access frequency and time characteristics, and automatically performs data migration between different storage tiers. The migration strategy is as follows: Hot data that has not been accessed for 30 days will be automatically migrated to the warm data storage layer. Warm data that has not been accessed for 180 days will be automatically migrated to the cold data storage layer. If cold data is accessed frequently, it will be automatically migrated to the warm data storage layer.

[0044] like Figure 3 As shown, the data processing engine is based on the MapReduce parallel computing framework and includes a data cleaning unit, a format conversion unit, and a quality inspection unit.

[0045] Data cleaning unit: Performs the following cleaning operations: Duplicate data detection: The MD5 hash algorithm is used to calculate the hash value of each book data, and duplicate data is identified by comparing the hash table, while the latest data record is retained.

[0046] Missing value handling: Check for missing values ​​in key fields (ISBN, book title, author). For missing values ​​in optional fields, fill them with default values; for missing values ​​in required fields, mark them as invalid data and send them to the manual review queue.

[0047] Outlier detection: Statistical methods (3σ principle, box plot) are used to detect outliers in numerical fields (such as abnormal publication date or price); a rule engine is used to detect format anomalies (such as incorrect ISBN format or incorrect email format).

[0048] The data cleaning unit uses the Spark SQL parallel processing framework, which can process more than 100,000 book data records at the same time, with a processing efficiency of 5,000 records per second.

[0049] Format conversion unit: Performs the following conversion operations: Document format conversion: Use the Apache Tika tool to convert PDF, EPUB, MOBI, and Word documents to plain text format; Text encoding conversion: Convert to UTF-8 encoding uniformly to handle garbled text issues; Structured data generation: Converting unstructured text data into structured XML format. XML Schema defines the complete field structure and type constraints of book data; Index generation: Generates full-text indexes (Elasticsearch) and inverted indexes, supporting fast retrieval.

[0050] Quality Inspection Unit: Performs quality assessment on the processed data and generates a quality report, including the following indicators: Completeness: Fill rate of key fields (target value > 95%); Consistency: Consistency of records with the same ISBN (target value > 99%); Accuracy: Formatting correctness and logical correctness (target value > 98%); Timeliness: The difference between the data update time and the current time (target value < 30 days).

[0051] Data that fails quality inspection is returned to the data cleaning unit for reprocessing or sent to the manual review queue.

[0052] like Figure 4As shown, the intelligent analysis module includes a natural language processing unit, a knowledge extraction unit, and a book classification unit.

[0053] Natural Language Processing Unit: Utilizing a BERT pre-trained model, it performs NLP analysis on the full text of the book. The specific workflow is as follows: Text preprocessing: performing text cleaning (removing HTML tags and special characters), word segmentation, and stop word filtering; Part-of-speech tagging: Using the BERT sequence tagging model, the part of speech (noun, verb, adjective, etc.) of each word is identified. Named Entity Recognition: Using the BERT-NER model, identify entities such as people's names, place names, organization names, and book titles in text; Syntactic analysis: Using the BERT syntactic analysis model, a dependency syntax tree for the sentence is generated; Semantic vector generation: The output vector of BERT's [CLS] tag is used as the semantic representation of the entire book (768-dimensional vector) for semantic similarity calculation.

[0054] After the natural language processing unit completes the processing, it generates structured NLP results (JSON format) and stores them in the HBase database.

[0055] Knowledge Extraction Unit: Based on the entity-relation extraction model, this unit extracts entities and relations from NLP results to construct a book knowledge graph. The specific process is as follows: Entity recognition: Extract entities from NLP results, including names (authors, characters), place names (locations, countries, cities), organization names (publishers, schools), keywords (subject terms), and events (historical events, scientific discoveries). Relationship recognition: Identifying semantic relationships between entities, such as "author-work-article", "work-topic-domain", "person-biography-location", etc. Knowledge Graph Construction: A book knowledge graph is constructed using the Neo4j graph database to store entities and relationships. The knowledge graph supports graph queries (Cypher query language) for knowledge discovery and related recommendations.

[0056] Book classification unit: Combining multi-label classification algorithms and topic models, books are classified in multiple dimensions. The specific process is as follows: Text feature extraction: Text features were extracted using TF-IDF and Word2Vec; Multi-label classification: Use the BERT fine-tuning model for multi-label classification, outputting the subject classification (22 major categories according to the Chinese Library Classification), thematic classification (10 categories), and reading difficulty level (3 levels) of the books. Topic model analysis: Using the LDA topic model, extract 5-10 topic keywords from the book content and generate topic tags; Reader profile tags: Based on user reading behavior data, books are tagged with reader profiles (children, teenagers, adults, professionals).

[0057] The book classification results are stored in an HBase database, and a classification index is generated, supporting the filtering and retrieval of books by classification criteria.

[0058] The security protection module ensures data security and system reliability.

[0059] Data encryption unit: Employs a hybrid encryption strategy. Encryption of transmission: Data transmission is encrypted using the TLS 1.3 protocol, and the certificate uses the RSA-2048 algorithm; Storage encryption: Sensitive fields (user ID number, mobile phone number) are stored using AES-256 encryption, and the key is managed uniformly by the Key Management Service (KMS); Key rotation: The encryption key is automatically rotated every 90 days to ensure key security.

[0060] Access control unit: Employs an attribute-based access control model (ABAC), with access policies based on the following attributes: User attributes: Role (administrator, regular user, visitor), department, level; Resource attributes: resource type (books, metadata, full-text data), sensitivity level; Environmental attributes: access time, access location, device type; Operation attribute: Operation type (read, write, delete, download).

[0061] The access control unit dynamically calculates user access permissions and rejects unauthorized access requests.

[0062] Audit Log Unit: Records the following log information: User operation log: Login, query, download, and modification operations; System access logs: API calls, database access, file access; Security event logs: login failure, permission denied, abnormal access; Data change log: Data insertion, update, and deletion operations.

[0063] Logs are stored in Elasticsearch, supporting real-time querying and analysis, and are retained for 2 years.

[0064] The task scheduling module is based on the Kubernetes container orchestration platform and the Apache Airflow workflow engine to achieve dynamic scheduling and management of tasks.

[0065] The task scheduling module supports the following functions: Task dependency management: Define the dependencies between tasks (such as data acquisition → data processing → intelligent analysis) to ensure that tasks are executed in the correct order; Load balancing: Dynamically allocate tasks to idle nodes based on cluster resource usage; Fault recovery: If a task fails, it will automatically retry (up to 3 times). If it still fails, an alarm notification will be sent. Priority scheduling: Supports setting task priorities (high, medium, low), with high-priority tasks being executed first; Elastic scaling: Automatically expands or shrinks worker nodes based on task queue length and resource usage.

[0066] In this embodiment, the complete workflow of the system is as follows: Step 1: The data acquisition module collects book data. Administrators configure acquisition tasks (data source, acquisition scope, acquisition time) through a web interface. The acquisition scheduler generates acquisition tasks based on the configuration and assigns them to the corresponding acquisition adapters for execution. The acquisition adapters retrieve book data from the data source and send it to the Kafka message queue.

[0067] Step 2: The data processing engine consumes Kafka messages and processes the book data in parallel. The data cleaning unit removes duplicates, handles missing values, and detects outliers; the format conversion unit converts the data into a standard format; and the quality inspection unit assesses the data quality and generates a quality report.

[0068] Step 3: The cloud storage module stores the processed data in the corresponding storage layer. Metadata is stored in HBase, full-text data is stored in HDFS, and frequently accessed data is cached in Redis.

[0069] Step 4: The intelligent analysis module performs intelligent analysis on the newly added book data. The natural language processing unit performs NLP analysis on the full text of the books; the knowledge extraction unit extracts entities and relationships to construct a knowledge graph; and the book classification unit classifies the books in multiple dimensions.

[0070] Step 5: Users access the system via the web interface or API to retrieve and obtain book data. The security module performs identity authentication and access control verification on user requests, and returns the query results after successful verification.

[0071] Step 6: The system regularly performs data maintenance tasks, including data migration (hot → warm → cold), data backup, log archiving, index rebuilding, etc.

[0072] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A cloud data processing system for book data acquisition, characterized in that, include: Data acquisition module, cloud storage module, data processing engine, intelligent analysis module, and security protection module; The data acquisition module is used to collect book data from multiple data sources such as book databases, e-book platforms and digital libraries, including basic book information, metadata, abstract content and full-text data of the book, and supports automatic recognition and standardized conversion of multiple data formats; The cloud storage module adopts a distributed storage architecture, including an HDFS distributed file system and a distributed database, for hierarchical storage of collected book data. Structured data is stored in the distributed database, and unstructured data is stored in the HDFS file system. It also supports automatic data sharding, redundant backup, and load balancing. The data processing engine, based on the MapReduce parallel computing framework, includes a data cleaning unit, a format conversion unit, and a quality detection unit. It is used to perform parallel cleaning, format standardization, and quality assessment on the collected book data, and to identify and correct errors and inconsistencies in the data. The intelligent analysis module includes a natural language processing unit, a knowledge extraction unit, and a book classification unit. It uses deep learning algorithms to perform semantic analysis, entity recognition, relation extraction, and automatic classification of book content, generating a book knowledge graph and a topic index. The security protection module includes a data encryption unit, an access control unit, and an audit log unit. It encrypts the stored and transmitted book data, implements fine-grained access control based on role and permission management, and records all operation logs. The cloud storage module, data processing engine, intelligent analysis module, and security protection module achieve asynchronous communication and data flow through message queue middleware, building a high-concurrency, highly available cloud data processing platform.

2. The cloud data processing system for book data acquisition according to claim 1, characterized in that, The data acquisition module includes multiple acquisition adapters, each corresponding to a data source type. It supports various acquisition methods such as HTTP interface, direct database connection, file upload, and web crawling, and has the functions of resuming interrupted downloads and incremental acquisition.

3. The cloud data processing system for book data acquisition according to claim 1, characterized in that, The cloud storage module adopts a three-tier storage architecture: hot data is stored in an in-memory database, warm data is stored in a distributed database, and cold data is stored in an HDFS file system. It also automatically migrates data based on the access frequency and time characteristics of the data.

4. The cloud data processing system for book data acquisition according to claim 1, characterized in that, The data cleaning unit of the data processing engine includes a duplicate data detection module, a missing value processing module, and an outlier detection module. It identifies duplicate data using a hash algorithm, processes missing values ​​using interpolation, and detects outlier data using statistical methods.

5. The cloud data processing system for book data acquisition according to claim 1, characterized in that, The natural language processing unit of the intelligent analysis module uses the BERT pre-trained model to perform word segmentation, part-of-speech tagging, named entity recognition, and syntactic analysis on the book text, generating a vector representation of the book content.

6. The cloud data processing system for book data acquisition according to claim 1, characterized in that, The knowledge extraction unit of the intelligent analysis module is based on an entity relationship extraction model. It extracts entities such as names of people, places, organizations, keywords, topics, and events from the book content, and constructs semantic relationships between entities to generate a structured knowledge base.

7. The cloud data processing system for book data acquisition according to claim 1, characterized in that, The intelligent analysis module's book classification unit combines multi-label classification algorithms and topic models to classify and label books in multiple dimensions, including subject classification, genre classification, reading difficulty level, and reader profile tags.

8. The cloud data processing system for book data acquisition according to claim 1, characterized in that, The data encryption unit of the security protection module uses a combination of symmetric and asymmetric encryption to encrypt the transmission and storage of sensitive book data, and supports dynamic updates and rotation of keys.

9. The cloud data processing system for book data acquisition according to claim 1, characterized in that, The access control unit of the security protection module adopts an attribute-based access control model, which dynamically calculates access permissions and implements access policies based on multiple dimensions such as user identity, role, time and location.

10. The cloud data processing system for book data acquisition according to claim 1, characterized in that, The system also includes a task scheduling module, which dynamically schedules data processing tasks based on system load and data priority, and supports task priority configuration, load balancing and automatic fault recovery.

Images