A full-type file unified aggregation text content extraction method, system, medium and device

Abstract

The application discloses a full-type file unified aggregation text content extraction method, system, medium and equipment, relates to the technical field of file processing, and the method comprises the following steps: system initialization and configuration, signature verification, extraction judgment, proxy forwarding, content extraction, result storage and return. A single, stable service entry point (Endpoint) and standardized API specification are provided for all upper-layer services; through a standard S3 signature verification mechanism, the safety and reliability of each call are ensured; in combination with an AK / SK system, unified permission control, traffic limitation and post-audit can be conveniently performed; a service-oriented architecture is adopted, and the extraction capabilities of different types of files are provided by independent micro-service modules. When a new file type needs to be supported, only a new content extraction module needs to be developed and registered into a mapping table, and the main process and upper-layer services do not need to be changed, so that the system has high expansibility. In combination with file size judgment, the logic of synchronous extraction and asynchronous extraction is executed.

Description

Technical Field

[0001] The present invention relates to the field of file processing technology, and in particular to a method, system, medium and device for extracting unified aggregated text content from all types of files. Background Art

[0002] In the current information and digital age, data exists in a vast array of files across various business systems. These files come in a variety of formats, including office documents (.docx, .pdf, .pptx), spreadsheets (.xlsx), images (.jpg, .png, .tiff), compressed archives (.zip, .rar), and even specialized formats specific to specific industries. Accurately and efficiently extracting textual information from these structured, semi-structured, and even unstructured files is a critical foundation for enabling enterprise-level applications such as content search, data mining, risk monitoring, knowledge graph construction, and AI model training.

[0003] Existing file content extraction technologies often suffer from one or more of the following issues: Isolated processes and complex integration: Extraction tools or services for different file types (such as PDF parsing libraries, OCR services, and Word components) are typically independent. Upper-level business systems must independently determine file types and write complex logic to call different interfaces or software, resulting in cumbersome development processes, high system coupling, and significant maintenance costs. Poor scalability and maintainability: Supporting a new file type often requires modifying the upper-level business system's call logic and even adjusting the core architecture. Unbalanced processing efficiency and resource utilization: Existing methods typically fail to differentiate between file processing complexity and file size. Using asynchronous processing for small, simple files can cause unnecessary delays. Using synchronous processing for large files (such as scanned PDFs) that require significant computing resources can easily lead to request timeouts and service congestion, severely impacting system stability and user experience. Therefore, the market urgently needs a universal file content extraction method and system that provides unified access, centralized security verification, intelligent routing and distribution, and combines synchronous and asynchronous processing modes to efficiently return results. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a method, system, medium and device for extracting text content from all types of files in a unified and aggregated manner.

[0005] The object of the present invention is achieved through the following technical solutions:

[0006] In a first aspect, the present invention discloses a method for extracting text content from all types of files in a unified and aggregated manner, comprising the following steps:

[0007] S100: System initialization and configuration. Deploy and start the signature verification module, extraction judgment module, proxy forwarding module, object storage module (or connect to an existing S3 service), and at least two specialized content extraction modules on a server cluster. Generate a unique access key (AK) and secret key (SK) pair for each authorized external application or service in the system security database. Configure the "file type-processing service" mapping table and synchronous / asynchronous processing thresholds.

[0008] S200, signature verification: A unified API gateway is established to receive external requests for extracting file information. The signature verification module built into the gateway verifies the request using the standard S3 object storage signature algorithm based on the preset access key AK and security key SK.

[0009] S300, extraction judgment: After the signature verification passes, the extraction judgment module parses the file metadata in the request, queries the "file type-processing service" mapping table preset in the database or configuration center, determines whether the file type supports content extraction, and obtains the corresponding downstream processing service information;

[0010] S400, proxy forwarding: After the extraction is judged to be passed, the proxy forwarding module forwards the request containing the file download address, file name and request ID to the corresponding content extraction module through the internal service call mechanism according to the processing service information found in the mapping table;

[0011] S500, content extraction. After receiving the request, the content extraction module intelligently determines whether to perform synchronous or asynchronous extraction based on the file size and processing complexity. It downloads the file through the interface provided by the object storage module and uses the corresponding parsing library or optical character recognition (OCR) technology to extract the text content.

[0012] S600, result storage and return. The content extraction module will respond to the extracted text content or processing status according to the preset return rules. For synchronous tasks, the extraction results will be returned directly to the requester; for asynchronous tasks, the results will be stored in the object storage module, and a queryable task ID and result address will be returned to the proxy forwarding module.

[0013] Based on the first aspect, configuring the "file type-processing service" mapping table and the synchronous / asynchronous processing threshold in step S100 specifically includes the following steps:

[0014] S101. In a distributed configuration center or database, a mapping table is created and maintained. The mapping table includes a file extension, a service name or address of a content extraction module that processes files of this type, a priority, and a status.

[0015] S102: Preset a file size threshold Sthreshold in the configuration of the content extraction module.

[0016] Based on the first aspect, step S200 specifically includes the following steps:

[0017] S201. The client constructs and sends a request. The external client constructs an HTTPS POST request according to the API specification. The HTTPS POST request includes a request body and a request header.

[0018] The request body includes in JSON format: the pre-signed download address file_url of the file to be processed in the S3 object storage, the original file name file_name, the unique request ID request_id for tracking, and a parameter response_mode that specifies the return type;

[0019] The request header includes the client's AK, a Unix timestamp x-timestamp, and a signature Authorization generated according to the S3 signature algorithm V4;

[0020] S202, the API gateway receives the request, and the unified API gateway of the system receives the extraction request;

[0021] S203: Trigger the signature verification module. The gateway passes the request information to the signature verification module. The information includes the HTTP method, path, query parameters, request body, and request header.

[0022] S204: Check the timestamp. The signature verification module first checks the x-timestamp in the request header and compares it with the server's current time. If the time difference exceeds the preset window, it is determined to be a replay attack and the request is rejected.

[0023] S205: Obtain SK and reconstruct the signature. The module obtains the corresponding SK from the system security database based on the AK in the request header. If the AK is invalid, the request is rejected. Then, based on the S3 signature algorithm specification, the server uses the same information to construct the string to be signed, String to Sign.

[0024] S206: Calculate and compare the signature. Use the obtained SK to encrypt the string to be signed, generate the server signature, and compare it bit by bit with the signature "Authorization" in the client request header.

[0025] S207. Process the verification result. If the signatures are inconsistent, it is determined to be an illegal request, and an HTTP 403 Forbidden error is returned to the client, and the audit log is recorded; if the signatures are consistent, the verification passes.

[0026] Based on the first aspect, step S300 specifically includes the following steps:

[0027] S301: trigger the extraction judgment module. After the signature verification passes, the request information is passed to the extraction judgment module;

[0028] S302, parse the file suffix, and extract the file suffix from the file_name field in the request body by the extraction judgment module;

[0029] S303, query the mapping table, use the extracted suffix name as a keyword to query the configured "file type-processing service" mapping table;

[0030] S304. Determine the retrievability and obtain the service address. Check the query results. If no corresponding record is found or the status of the record is disabled, it is determined that the file type is not supported and a specified error code is returned to the client. If the query is successful, the corresponding service name or address handler_service is obtained.

[0031] Based on the first aspect, step 400 specifically includes the following steps:

[0032] S401. Trigger the proxy forwarding module and transfer the original request information together with the service name or address handler_service obtained in step S304 to the proxy forwarding module;

[0033] S402: Execute internal forwarding. The proxy forwarding module acts as a router between services, constructs a new internal RPC or HTTP request, and sends the task information to the target content extraction module.

[0034] Based on the first aspect, step 500 specifically includes the following steps:

[0035] S501, the content extraction module receives the task;

[0036] S502: Perform synchronous / asynchronous judgment. The extraction module first obtains the metadata of the file from the S3 service through file_url, obtains the file size, and compares the file size with the file size threshold Sthreshold;

[0037] S503: Asynchronous logic: If the file size exceeds the file size threshold Sthreshold, the extraction module returns a response to the proxy forwarding module, including the task ID task_id and the processing status. The task details, including the file_url, are serialized and pushed to a highly available message queue. An independent background worker cluster consumes the tasks in the message queue.

[0038] S504, synchronous logic / asynchronous Worker execution, for small files or large file tasks consumed by background workers, the execution process downloads the file from the S3 object storage module to the local temporary storage space through file_url;

[0039] S505: Call the core engine to perform extraction and perform content extraction based on the module's expertise;

[0040] S506: Format and store the results, clean and organize all the extracted text contents, and merge them into a single text file; upload this text file to a dedicated result storage bucket in the S3 object storage module, and associate the file name with the request_id or task_id.

[0041] Based on the first aspect, step 600 specifically includes the following steps:

[0042] S601, synchronous return: For synchronously executed tasks and the response_mode requested by the client is "direct", after completing step S506, the extraction module directly puts the extracted text content into the HTTP response body and encapsulates it in JSON format, and finally returns it to the client through the proxy forwarding module;

[0043] S602: Return the extracted content URL. For tasks that are executed synchronously but whose response_mode is "url" or all asynchronous tasks, the system updates the task status after step S506 is completed. For synchronous tasks, the system directly returns a response containing the S3 address. For asynchronous tasks, the system waits for client queries.

[0044] S603, asynchronous result query. For asynchronous tasks, after receiving the task_id returned in step S503, the client obtains the task progress by polling the status query API. When the task is completed, the API returns a success status and the pre-signed download address of the final result file in the S3 object storage.

[0045] In a second aspect, the present invention discloses a system for extracting text content from all types of files in a unified and aggregated manner, which is used in the above-mentioned method for extracting text content from all types of files in a unified and aggregated manner, comprising:

[0046] Signature verification module, used to verify timestamps, obtain SK, and reconstruct signatures;

[0047] Extraction judgment module, used to parse the file suffix, judge the extractability and obtain the service address;

[0048] An agent forwarding module, used to send task information to a target content extraction module;

[0049] Object storage module, used to store result files and provide an interface to the content extraction module;

[0050] The content extraction module is used to process one or several file types and extract file content using the corresponding parsing library or optical character recognition (OCR) technology.

[0051] In a third aspect, the present invention further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program implements the above-described method when executed by a processor.

[0052] In a fourth aspect, the present invention further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-described method when executing the computer program.

[0053] The beneficial effects of the present invention are:

[0054] 1) Unified entry and standardization: providing a single, stable service entry point (Endpoint) and standardized API specifications for all upper-level businesses, greatly simplifying the integration complexity and development costs of business systems.

[0055] 2) High security and controllability: The standard S3 signature verification mechanism ensures the security and reliability of every call. Combined with the AK / SK system, unified permission control, traffic restriction, and post-audit can be easily implemented.

[0056] 3) High scalability and flexibility. Using a service-oriented architecture, the extraction capabilities for different file types are provided by independent microservice modules. When supporting new file types, simply develop and register a new content extraction module in the mapping table without modifying the main process or upper-level services, resulting in highly scalable systems.

[0057] 4) High efficiency and flexibility: Innovatively combining file size determination with a combination of synchronous and asynchronous file extraction logic. This provides fast responses for small files and avoids request blocking and timeouts for large files through asynchronous task queues, thereby optimizing system resource utilization, improving overall throughput, and enhancing user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 The present invention is a flowchart of a method for extracting text content from all types of files in a unified and aggregated manner according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0060] The present invention discloses a method, system, medium and device for extracting text content from all types of files in a unified and aggregated manner. At present, file content extraction solutions are extremely complex, lack unified security management and control, have a rigid architecture and poor scalability, and do not differentiate between processing loads, resulting in resource waste and low efficiency. The present invention provides a standard, secure, efficient and easily scalable unified interface for upper-layer applications by constructing an aggregation service gateway that integrates S3 standard security authentication, dynamic routing distribution, and synchronous / asynchronous elastic processing logic, thereby providing a solid and reliable data processing foundation for enterprise-level content retrieval, big data analysis and artificial intelligence model training. It solves the problems of high development cost, poor compatibility, low security and efficiency in the prior art when extracting text content from different types of files. The flow chart of the method is as follows Figure 1 As shown, the specific steps include:

[0061] S100. System initialization and configuration: On a server cluster, deploy and start the signature verification module, extraction judgment module, proxy forwarding module, object storage module or connect to an existing S3 service, and at least two specialized content extraction modules. Each content extraction module is responsible for processing one or several specific file types, for example: an Office document extraction module for processing .docx, .xlsx, and .pptx, and an OCR extraction module for processing .jpg, .png, and .pdf (scanned documents); in the system security database, generate a unique access key (AK) and secret key (SK) for each authorized external application or service. The AK is used to identify the caller, and the SK is used for signature calculation and is not exposed externally; configure the "file type-processing service" mapping table and the synchronous / asynchronous processing threshold;

[0062] S200, signature verification, a unified API gateway is established, an extraction request containing file information transmitted from outside is received, a signature verification module built in the gateway verifies the legality, integrity and timeliness of the request according to a preset access key (AK) and a security key (SK) by using a standard S3 object storage signature algorithm;

[0063] S300, extraction judgment, after the signature verification passes, the extraction judgment module parses the file meta information (such as the file suffix name) in the request, queries a "file type-processing service" mapping table preset in a database or a configuration center, judges whether the file type supports content extraction, and obtains corresponding downstream (proxy forwarding module) processing service information;

[0064] S400, proxy forwarding, after the extraction judgment passes, the proxy forwarding module forwards the request containing the file download address, the file name, the request ID and the like to the corresponding downstream (content extraction module) through an internal service calling mechanism according to the processing service information obtained from the mapping table;

[0065] S500, content extraction, after the content extraction module receives the request, the content extraction module intelligently judges to perform synchronous extraction or asynchronous extraction according to the file size and the processing complexity; the file is downloaded through an interface provided by an object storage module, and the corresponding analysis library or optical character recognition (OCR) technology is used to extract the text content;

[0066] S600, result storage and return, the content extraction module responds according to a preset return rule to the extracted text content or processing status. For a synchronous task, the extraction result can be directly returned to the requester. For an asynchronous task, the result can be stored in an object storage module, and a task ID and a result address for query can be returned to the upstream (proxy forwarding module).

[0067] Exemplarily, the configuration of the "file type-processing service" mapping table and the synchronous / asynchronous processing threshold in step S100 specifically includes the following steps:

[0068] S101, configuring a "file type-processing service" mapping table, a mapping table is created and maintained in a distributed configuration center or a database. The table at least includes the following fields: file_extension (file suffix name, such as.pdf), handler_service (service name or address of the content extraction module for processing the file of this type, such as ocr-extraction-service), priority (priority, used for decision-making when the suffix name conflicts), status (status, enabled / disabled);

[0069] S102, configure a synchronous / asynchronous processing threshold, in the configuration of the content extraction module, a file size threshold Sthreshold is preset, for example, Sthreshold=20MB. This threshold is used for subsequent judgment whether to adopt synchronous processing or asynchronous processing.

[0070] Exemplarily, step S200 specifically includes the following steps:

[0071] S201, the client constructs and sends a request, an external client (an upper-layer business system) constructs an HTTPS POST request according to an API specification, and the HTTPS POST request includes a request body Body and a request header Header;

[0072] The request body Body includes, in a JSON format: a pre-signed download address file_url of a to-be-processed file in an S3 object storage, an original file name file_name, a unique request ID request_id for tracking, and a parameter response_mode ("direct" or "url") for specifying a return type;

[0073] The request header Header includes an AK of the client, a Unix timestamp x-timestamp, and a signature Authorization generated according to an S3 signature algorithm V4;

[0074] S202, the API gateway receives the request, a unified API gateway of the system receives the extraction request;

[0075] S203, trigger the signature verification module, the gateway passes information of the request to the signature verification module, and the information includes an HTTP method, a path, a query parameter, a request body Body, and a request header Header;

[0076] S204, verify the timestamp, the signature verification module first checks the x-timestamp in the request header Header, and compares it with the current time of the server; if the time difference exceeds a preset window (for example, 5 minutes), it is determined as a replay attack, and the request is rejected;

[0077] S205, obtain the SK and reconstruct the signature, the module obtains the corresponding SK from the system security database according to the AK in the request header; if the AK is invalid, the request is rejected, and then a to-be-signed string String to Sign is constructed on the server using the same information according to the S3 signature algorithm specification;

[0078] S206: Calculate and compare the signature. Use the obtained SK to encrypt the string to be signed, generate the server signature, and compare it bit by bit with the signature "Authorization" in the client request header.

[0079] S207. Process the verification result. If the signatures are inconsistent, it is determined to be an illegal request, and an HTTP 403 Forbidden error is returned to the client, and the audit log is recorded; if the signatures are consistent, the verification passes.

[0080] Exemplarily, step S300 specifically includes the following steps:

[0081] S301: trigger the extraction judgment module. After the signature verification passes, the request information is passed to the extraction judgment module;

[0082] S302, parsing the file suffix, the extraction judgment module parses the file suffix from the file_name field in the request body; for example, extracting ".pdf" from "financial report_Q2.pdf";

[0083] S303, query the mapping table, use the extracted suffix name as a keyword to query the configured "file type-processing service" mapping table;

[0084] S304. Determine the retrievability and obtain the service address. Check the query results. If no corresponding record is found or the record status is disabled, it is determined that the file type is not supported and a specified error code (such as 415 Unsupported Media Type) is returned to the client. If the query is successful, the corresponding service name or address handler_service is obtained.

[0085] Exemplarily, step 400 specifically includes the following steps:

[0086] S401. Trigger the proxy forwarding module and transfer the original request information together with the service name or address handler_service obtained in step S304 to the proxy forwarding module;

[0087] S402: Execute internal forwarding. The proxy forwarding module acts as a router between services, constructs a new internal RPC or HTTP request, and sends the task information to the target content extraction module.

[0088] Exemplarily, step 500 specifically includes the following steps:

[0089] S501: The content extraction module receives a task; for example, the OCR extraction module receives a task to process "Financial Report_Q2.pdf";

[0090] S502: Perform synchronous / asynchronous judgment. The extraction module first obtains the file metadata (especially Content-Length) from the S3 service through file_url, obtains the file size, and compares the file size with the file size threshold Sthreshold;

[0091] S503: Asynchronous logic: If the file size exceeds the file size threshold Sthreshold, the extraction module returns a response to the proxy forwarding module, including the task ID task_id and the processing status. The task details, including the file_url, are serialized and pushed to a highly available message queue (such as RabbitMQ or Kafka). An independent background worker cluster consumes the tasks in the message queue.

[0092] S504, synchronous logic / asynchronous worker execution. For small files (synchronous logic) or large file tasks consumed by background workers, the execution process downloads the file from the S3 object storage module to local temporary storage space through file_url;

[0093] S505: Calling the core engine to perform extraction, performing content extraction based on the module's expertise; for example: the Office document module calls libraries such as Apache POI or python-docx to parse the document structure and extract plain text; the OCR extraction module performs page-by-page rendering, image processing, and text recognition on PDF or image files;

[0094] S506. Format and store the results. Clean and organize all the extracted text content (e.g., remove extra line breaks) and merge them into a single text file (.txt). Upload this text file to a dedicated result storage bucket in the S3 object storage module. The file name is associated with the request_id or task_id.

[0095] Exemplarily, step 600 specifically includes the following steps:

[0096] S601, synchronous return: For synchronously executed tasks and the response_mode requested by the client is "direct", after completing step S506, the extraction module directly puts the extracted text content into the HTTP response body and encapsulates it in JSON format, and finally returns it to the client through the proxy forwarding module;

[0097] S602: Return the extracted content URL. For tasks executed synchronously but with response_mode set to "url" or all asynchronous tasks, after step S506 is completed, the system updates the task status (for example, in Redis or a database, the task_id status is updated to "completed" and the S3 address of the result file is recorded). For synchronous tasks, a response containing the S3 address is directly returned; for asynchronous tasks, the system waits for client queries.

[0098] S603, asynchronous result query. For asynchronous tasks, after receiving the task_id returned in step S503, the client obtains the task progress by polling the status query API (GET / api / v1 / task_status?id={task_id}). When the task is completed, the API returns a success status and the pre-signed download address of the final result file in the S3 object storage.

[0099] Therefore, this application achieves text content extraction from all file types through a unified interface and process, improving extraction efficiency and accuracy, ensuring the security of the extraction process, and reducing development and maintenance costs. The method features unified access and standardized security authentication technology. By establishing a single API gateway with a built-in signature verification module based on the S3 standard, all extraction requests for different file types are consolidated and standardized AK / SK signature authentication is enforced. This not only greatly simplifies the integration of upper-layer applications but, more importantly, establishes a centralized and reliable security barrier, resolving security management challenges inherent in existing technologies caused by fragmented interfaces and varying authentication methods. Adaptive processing technology based on file size is introduced in the content extraction module, with synchronous / asynchronous processing decision logic based on file size thresholds. This intelligently selects the optimal processing mode based on the specific file load (measured by size): synchronous processing for small files ensures fast responses, while asynchronous processing for large files, via message queues, avoids request blocking and timeouts. This technology enables refined and intelligent management of system resources, significantly improving overall system throughput and stability.

[0100] Specifically, the present invention discloses a system for extracting text content from all types of files in a unified and aggregated manner, which is used in the above-mentioned method for extracting text content from all types of files in a unified and aggregated manner, including:

[0101] Signature verification module, used to verify timestamps, obtain SK, and reconstruct signatures;

[0102] Extraction judgment module, used to parse the file suffix, judge the extractability and obtain the service address;

[0103] An agent forwarding module, used to send task information to a target content extraction module;

[0104] Object storage module, used to store result files and provide an interface to the content extraction module;

[0105] The content extraction module is used to process one or several file types and extract file content using the corresponding parsing library or optical character recognition (OCR) technology.

[0106] Illustratively, the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0107] Exemplarily, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0108] Illustratively, the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0109] Exemplarily, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0110] In summary, the current extraction tool processes for different file formats are isolated and complex to integrate; various services lack unified security authentication and management, posing a risk of data leakage; the system architecture is rigid and difficult to expand to support new file types; and the processing logic does not distinguish between file size and complexity, resulting in low processing efficiency and waste of resources. The present invention overcomes many of the shortcomings of file content extraction solutions in the prior art. By establishing a centralized service that integrates security, distribution, and flexible processing, it achieves standardized, efficient, secure, and highly scalable content extraction for all types of files, thereby providing stable and reliable data support for downstream big data analysis, content retrieval, and artificial intelligence applications.

[0111] The foregoing description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A method for extracting text content from all types of files in a unified aggregated manner, characterized in that: The following steps are involved: S100: System initialization and configuration. Deploy and start the signature verification module, extraction judgment module, proxy forwarding module, object storage module (or connect to an existing S3 service), and at least two specialized content extraction modules on a server cluster. Generate a unique access key (AK) and secret key (SK) pair in the system security database for each authorized external application or service. Configure the "file type-processing service" mapping table and synchronous / asynchronous processing thresholds. S200, signature verification: A unified API gateway is established to receive external requests for extracting file information. The signature verification module built into the gateway verifies the request using the standard S3 object storage signature algorithm based on the preset access key AK and security key SK. S300, extraction judgment: After the signature verification passes, the extraction judgment module parses the file metadata in the request, queries the "file type-processing service" mapping table preset in the database or configuration center, determines whether the file type supports content extraction, and obtains the corresponding downstream processing service information; S400, proxy forwarding: After the extraction is judged to be passed, the proxy forwarding module forwards the request containing the file download address, file name and request ID to the corresponding content extraction module through the internal service call mechanism according to the processing service information found in the mapping table; S500, content extraction. After receiving the request, the content extraction module intelligently determines whether to perform synchronous or asynchronous extraction based on the file size and processing complexity. It downloads the file through the interface provided by the object storage module and uses the corresponding parsing library or optical character recognition (OCR) technology to extract the text content. S600, result storage and return: the content extraction module responds to the extracted text content or processing status according to the preset return rules. For synchronous tasks, the extraction result is directly returned to the requester; For asynchronous tasks, the results are stored in the object storage module, and a queryable task ID and result address are returned to the proxy forwarding module; Step S200 specifically includes the following steps: S201. The client constructs and sends a request. The external client constructs an HTTPS POST request according to the API specification. The HTTPS POST request includes a request body and a request header. The request body includes in JSON format: the pre-signed download address file_url of the file to be processed in the S3 object storage, the original file name file_name, the unique request ID request_id for tracking, and a parameter response_mode that specifies the return type; The request header includes the client's AK, a Unix timestamp x-timestamp, and a signature Authorization generated according to the S3 signature algorithm V4; S202, the API gateway receives the request, and the unified API gateway of the system receives the extraction request; S203: Trigger the signature verification module. The gateway passes the request information to the signature verification module. The information includes the HTTP method, path, query parameters, request body, and request header. S204: Check the timestamp. The signature verification module first checks the x-timestamp in the request header and compares it with the server's current time. If the time difference exceeds the preset window, it is determined to be a replay attack and the request is rejected. S205: Obtain SK and reconstruct the signature. The module obtains the corresponding SK from the system security database based on the AK in the request header. If the AK is invalid, the request is rejected. Then, based on the S3 signature algorithm specification, the server uses the same information to construct the string to be signed, String to Sign. S206: Calculate and compare the signature. Use the obtained SK to encrypt the string to be signed, generate the server signature, and compare it bit by bit with the signature "Authorization" in the client request header. S207. Process the verification result. If the signatures are inconsistent, it is determined to be an illegal request, and an HTTP 403 Forbidden error is returned to the client, and the audit log is recorded; if the signatures are consistent, the verification passes.

2. A method for extracting text content from all types of files in a unified aggregated manner according to claim 1, characterized in that: The configuration of the "file type-processing service" mapping table and the synchronous / asynchronous processing threshold in step S100 specifically includes the following steps: S101. In a distributed configuration center or database, a mapping table is created and maintained. The mapping table includes a file extension, a service name or address of a content extraction module that processes files of this type, a priority, and a status. S102: Preset a file size threshold Sthreshold in the configuration of the content extraction module.

3. A method for extracting text content from all types of files in a unified aggregated manner according to claim 2, characterized in that: Step S300 specifically includes the following steps: S301: trigger the extraction judgment module. After the signature verification passes, the request information is passed to the extraction judgment module; S302, parse the file suffix, and extract the file suffix from the file_name field in the request body by the extraction judgment module; S303, query the mapping table, use the extracted suffix name as a keyword to query the configured "file type-processing service" mapping table; S304. Determine the retrievability and obtain the service address. Check the query results. If no corresponding record is found or the status of the record is disabled, it is determined that the file type is not supported and a specified error code is returned to the client. If the query is successful, the corresponding service name or address handler_service is obtained.

4. A method for extracting text content from all types of files in a unified aggregated manner according to claim 3, characterized in that: Step 400 specifically includes the following steps: S401. Trigger the proxy forwarding module and transfer the original request information together with the service name or address handler_service obtained in step S304 to the proxy forwarding module; S402: Execute internal forwarding. The proxy forwarding module acts as a router between services, constructs a new internal RPC or HTTP request, and sends the task information to the target content extraction module.

5. A method for extracting text content from all types of files in a unified aggregated manner according to claim 4, characterized in that: Step 500 specifically includes the following steps: S501, the content extraction module receives the task; S502: Perform synchronous / asynchronous judgment. The extraction module first obtains the metadata of the file from the S3 service through file_url, obtains the file size, and compares the file size with the file size threshold Sthreshold; S503: Asynchronous logic: If the file size exceeds the file size threshold Sthreshold, the extraction module returns a response to the proxy forwarding module, including the task ID task_id and the processing status. The task details, including the file_url, are serialized and pushed to a highly available message queue. An independent background worker cluster consumes the tasks in the message queue. S504, synchronous logic / asynchronous Worker execution, for small files or large file tasks consumed by background workers, the execution process downloads the file from the S3 object storage module to the local temporary storage space through file_url; S505: Call the core engine to perform extraction and perform content extraction based on the module's expertise; S506: Format and store the results, clean and organize all the extracted text contents, and merge them into a single text file; upload this text file to a dedicated result storage bucket in the S3 object storage module, and associate the file name with the request_id or task_id.

6. A method for extracting text content from all types of files in a unified aggregated manner according to claim 5, characterized in that: Step 600 specifically includes the following steps: S601, synchronous return: For synchronously executed tasks and the response_mode requested by the client is "direct", after completing step S506, the extraction module directly puts the extracted text content into the HTTP response body and encapsulates it in JSON format, and finally returns it to the client through the proxy forwarding module; S602: Return the extracted content URL. For tasks that are executed synchronously but whose response_mode is "url" or all asynchronous tasks, the system updates the task status after step S506 is completed. For synchronous tasks, the system directly returns a response containing the S3 address. For asynchronous tasks, the system waits for client queries. S603, asynchronous result query. For asynchronous tasks, after receiving the task_id returned in step S503, the client obtains the task progress by polling the status query API. When the task is completed, the API returns a success status and the pre-signed download address of the final result file in the S3 object storage.

7. A system for extracting text content from all types of documents in a unified aggregated manner, applying the method for extracting text content from all types of documents in a unified aggregated manner as claimed in any one of claims 1 to 6, characterized in that: include: Signature verification module, used to verify timestamps, obtain SK, and reconstruct signatures; Extraction judgment module, used to parse file suffixes, judge extractability and obtain service addresses; An agent forwarding module, used to send task information to a target content extraction module; Object storage module, used to store result files and provide an interface to the content extraction module; Content extraction module, used to process one or several file types; Use the corresponding parsing library or optical character recognition (OCR) technology to extract file content.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the method according to any one of claims 1 to 6 when executed by a processor.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

Images