Unified aggregated text content extraction method and system for full-type files, medium and equipment

Abstract

The invention discloses a method, a system, a medium and equipment for extracting unified aggregated text content of all types of files, and relates to the technical field of file processing, the method comprises the following steps: system initialization and configuration, signature verification, extraction judgment, proxy forwarding, content extraction, and result storage and return. A single and stable service entry point (Endpoint) and a standardized API (Application Program Interface) specification are provided for all upper-layer businesses; through a standard S3 signature verification mechanism, the security and reliability of each call are ensured; in combination with an AK / SK system, unified authority control, flow limitation and post auditing can be conveniently carried out; a service-oriented architecture is adopted, and the extraction capability of different types of files is provided by an independent micro-service module. When a new file type needs to be supported, only a new content extraction module needs to be developed and registered into the mapping table, a main process and an upper layer service do not need to be changed, and the system expansibility is extremely high. And in combination with file size judgment, executing logic of combining synchronous extraction and asynchronous extraction of the file.

Description

Technical Field

[0001] The present invention relates to the field of file processing technology, and in particular to a method, system, medium and device for extracting unified aggregated text content from all types of files. Background Art

[0002] In the current information and digital age, data exists in a vast array of files across various business systems. These files come in a variety of formats, including office documents (.docx, .pdf, .pptx), spreadsheets (.xlsx), images (.jpg, .png, .tiff), compressed archives (.zip, .rar), and even specialized formats specific to specific industries. Accurately and efficiently extracting textual information from these structured, semi-structured, and even unstructured files is a critical foundation for enabling enterprise-level applications such as content search, data mining, risk monitoring, knowledge graph construction, and AI model training.

[0003] Existing file content extraction technologies often suffer from one or more of the following issues: Isolated processes and complex integration: Extraction tools or services for different file types (such as PDF parsing libraries, OCR services, and Word components) are typically independent. Upper-level business systems must independently determine file types and write complex logic to call different interfaces or software, resulting in cumbersome development processes, high system coupling, and significant maintenance costs. Poor scalability and maintainability: Supporting a new file type often requires modifying the upper-level business system's call logic and even adjusting the core architecture. Unbalanced processing efficiency and resource utilization: Existing methods typically fail to differentiate between file processing complexity and file size. Using asynchronous processing for small, simple files can cause unnecessary delays. Using synchronous processing for large files (such as scanned PDFs) that require significant computing resources can easily lead to request timeouts and service congestion, severely impacting system stability and user experience. Therefore, the market urgently needs a universal file content extraction method and system that provides unified access, centralized security verification, intelligent routing and distribution, and combines synchronous and asynchronous processing modes to efficiently return results. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a method, system, medium and device for extracting text content from all types of files in a unified and aggregated manner.

[0005] The object of the present invention is achieved through the following technical solutions: In a first aspect, the present invention discloses a method for extracting text content from all types of files in a unified and aggregated manner, comprising the following steps: S100: System initialization and configuration. Deploy and start the signature verification module, extraction judgment module, proxy forwarding module, object storage module (or connect to an existing S3 service), and at least two specialized content extraction modules on a server cluster. Generate a unique access key (AK) and secret key (SK) pair for each authorized external application or service in the system security database. Configure the "file type-processing service" mapping table and synchronous / asynchronous processing thresholds. S200, signature verification: A unified API gateway is established to receive external requests for extracting file information. The signature verification module built into the gateway verifies the request using the standard S3 object storage signature algorithm based on the preset access key AK and security key SK. S300, extraction judgment: After the signature verification passes, the extraction judgment module parses the file metadata in the request, queries the "file type-processing service" mapping table preset in the database or configuration center, determines whether the file type supports content extraction, and obtains the corresponding downstream processing service information; S400, proxy forwarding: After the extraction is judged to be passed, the proxy forwarding module forwards the request containing the file download address, file name and request ID to the corresponding content extraction module through the internal service call mechanism according to the processing service information found in the mapping table; S500, content extraction. After receiving the request, the content extraction module intelligently determines whether to perform synchronous or asynchronous extraction based on the file size and processing complexity. It downloads the file through the interface provided by the object storage module and uses the corresponding parsing library or optical character recognition (OCR) technology to extract the text content. S600, result storage and return. The content extraction module will respond to the extracted text content or processing status according to the preset return rules. For synchronous tasks, the extraction results will be returned directly to the requester; for asynchronous tasks, the results will be stored in the object storage module, and a queryable task ID and result address will be returned to the proxy forwarding module.

[0006] Based on the first aspect, configuring the "file type-processing service" mapping table and the synchronous / asynchronous processing threshold in step S100 specifically includes the following steps: S101. In a distributed configuration center or database, a mapping table is created and maintained. The mapping table includes a file extension, a service name or address of a content extraction module that processes files of this type, a priority, and a status. S102: Preset a file size threshold Sthreshold in the configuration of the content extraction module.

[0007] Based on the first aspect, step S200 specifically includes the following steps: S201. The client constructs and sends a request. The external client constructs an HTTPS POST request according to the API specification. The HTTPS POST request includes a request body and a request header. The request body includes in JSON format: the pre-signed download address file_url of the file to be processed in the S3 object storage, the original file name file_name, the unique request ID request_id for tracking, and a parameter response_mode that specifies the return type; The request header includes the client's AK, a Unix timestamp x-timestamp, and a signature Authorization generated according to the S3 signature algorithm V4; S202, the API gateway receives the request, and the unified API gateway of the system receives the extraction request; S203: Trigger the signature verification module. The gateway passes the request information to the signature verification module. The information includes the HTTP method, path, query parameters, request body, and request header. S204: Check the timestamp. The signature verification module first checks the x-timestamp in the request header and compares it with the server's current time. If the time difference exceeds the preset window, it is determined to be a replay attack and the request is rejected. S205: Obtain SK and reconstruct the signature. The module obtains the corresponding SK from the system security database based on the AK in the request header. If the AK is invalid, the request is rejected. Then, based on the S3 signature algorithm specification, the server uses the same information to construct the string to be signed, String to Sign. S206: Calculate and compare the signature. Use the obtained SK to encrypt the string to be signed, generate the server signature, and compare it bit by bit with the signature "Authorization" in the client request header. S207. Process the verification result. If the signatures are inconsistent, it is determined to be an illegal request, and an HTTP 403 Forbidden error is returned to the client, and the audit log is recorded; if the signatures are consistent, the verification passes.

[0008] Based on the first aspect, step S300 specifically includes the following steps: S301: trigger the extraction judgment module. After the signature verification passes, the request information is passed to the extraction judgment module; S302, parse the file suffix, and extract the file suffix from the file_name field in the request body by the extraction judgment module; S303, query the mapping table, use the extracted suffix name as a keyword to query the configured "file type-processing service" mapping table; S304. Determine the retrievability and obtain the service address. Check the query results. If no corresponding record is found or the status of the record is disabled, it is determined that the file type is not supported and a specified error code is returned to the client. If the query is successful, the corresponding service name or address handler_service is obtained.

[0009] Based on the first aspect, step 400 specifically includes the following steps: S401. Trigger the proxy forwarding module and transfer the original request information together with the service name or address handler_service obtained in step S304 to the proxy forwarding module; S402: Execute internal forwarding. The proxy forwarding module acts as a router between services, constructs a new internal RPC or HTTP request, and sends the task information to the target content extraction module.

[0010] Based on the first aspect, step 500 specifically includes the following steps: S501, the content extraction module receives the task; S502: Perform synchronous / asynchronous judgment. The extraction module first obtains the metadata of the file from the S3 service through file_url, obtains the file size, and compares the file size with the file size threshold Sthreshold; S503: Asynchronous logic: If the file size exceeds the file size threshold Sthreshold, the extraction module returns a response to the proxy forwarding module, including the task ID task_id and the processing status. The task details, including the file_url, are serialized and pushed to a highly available message queue. An independent background worker cluster consumes the tasks in the message queue. S504, synchronous logic / asynchronous Worker execution, for small files or large file tasks consumed by background workers, the execution process downloads the file from the S3 object storage module to the local temporary storage space through file_url; S505: Call the core engine to perform extraction and perform content extraction based on the module's expertise; S506: Format and store the results, clean and organize all the extracted text contents, and merge them into a single text file; upload this text file to a dedicated result storage bucket in the S3 object storage module, and associate the file name with the request_id or task_id.

[0011] Based on the first aspect, step 600 specifically includes the following steps: S601, synchronous return: For synchronously executed tasks and the response_mode requested by the client is "direct", after completing step S506, the extraction module directly puts the extracted text content into the HTTP response body and encapsulates it in JSON format, and finally returns it to the client through the proxy forwarding module; S602: Return the extracted content URL. For tasks that are executed synchronously but whose response_mode is "url" or all asynchronous tasks, the system updates the task status after step S506 is completed. For synchronous tasks, the system directly returns a response containing the S3 address. For asynchronous tasks, the system waits for client queries. S603, asynchronous result query. For asynchronous tasks, after receiving the task_id returned in step S503, the client obtains the task progress by polling the status query API. When the task is completed, the API returns a success status and the pre-signed download address of the final result file in the S3 object storage.

[0012] In a second aspect, the present invention discloses a system for extracting text content from all types of files in a unified and aggregated manner, which is used in the above-mentioned method for extracting text content from all types of files in a unified and aggregated manner, comprising: Signature verification module, used to verify timestamps, obtain SK, and reconstruct signatures; Extraction judgment module, used to parse file suffixes, judge extractability and obtain service addresses; An agent forwarding module, used to send task information to a target content extraction module; Object storage module, used to store result files and provide an interface to the content extraction module; The content extraction module is used to process one or several file types and extract file content using the corresponding parsing library or optical character recognition (OCR) technology.

[0013] In a third aspect, the present invention further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program implements the above-described method when executed by a processor.

[0014] In a fourth aspect, the present invention further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-described method when executing the computer program.

[0015] The beneficial effects of the present invention are: 1) Unified entry and standardization: providing a single, stable service entry point (Endpoint) and standardized API specifications for all upper-level businesses, greatly simplifying the integration complexity and development costs of business systems.

[0016] 2) High security and controllability: The standard S3 signature verification mechanism ensures the security and reliability of every call. Combined with the AK / SK system, unified permission control, traffic restriction, and post-audit can be easily implemented.

[0017] 3) High scalability and flexibility. Using a service-oriented architecture, the extraction capabilities for different file types are provided by independent microservice modules. When supporting new file types, simply develop and register a new content extraction module in the mapping table without modifying the main process or upper-level services, resulting in highly scalable systems.

[0018] 4) High efficiency and flexibility: Innovatively combining file size determination with a combination of synchronous and asynchronous file extraction logic. This provides fast responses for small files and avoids request blocking and timeouts for large files through asynchronous task queues, thereby optimizing system resource utilization, improving overall throughput, and enhancing user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The present invention is a flowchart of a method for extracting text content from all types of files in a unified and aggregated manner according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0021] The present invention discloses a method, system, medium and device for extracting text content from all types of files in a unified and aggregated manner. At present, file content extraction solutions are extremely complex, lack unified security management and control, have a rigid architecture and poor scalability, and do not differentiate between processing loads, resulting in resource waste and low efficiency. The present invention provides a standard, secure, efficient and easily scalable unified interface for upper-layer applications by constructing an aggregation service gateway that integrates S3 standard security authentication, dynamic routing distribution, and synchronous / asynchronous elastic processing logic, thereby providing a solid and reliable data processing foundation for enterprise-level content retrieval, big data analysis and artificial intelligence model training. It solves the problems of high development cost, poor compatibility, low security and efficiency in the prior art when extracting text content from different types of files. The flow chart of the method is as follows Figure 1 As shown, the specific steps include: S100. System initialization and configuration: On a server cluster, deploy and start the signature verification module, extraction judgment module, proxy forwarding module, object storage module or connect to an existing S3 service, and at least two specialized content extraction modules. Each content extraction module is responsible for processing one or several specific file types, for example: an Office document extraction module for processing .docx, .xlsx, and .pptx, and an OCR extraction module for processing .jpg, .png, and .pdf (scanned documents); in the system security database, generate a unique access key (AK) and secret key (SK) for each authorized external application or service. The AK is used to identify the caller, and the SK is used for signature calculation and is not exposed externally; configure the "file type-processing service" mapping table and the synchronous / asynchronous processing threshold; S200, Signature Verification: A unified API gateway is established to receive external requests for extracting file information. The gateway's built-in signature verification module uses the standard S3 object storage signature algorithm based on the preset access key (AK) and security key (SK) to verify the legitimacy, integrity, and timeliness of the request. S300, extraction judgment: After the signature verification passes, the extraction judgment module parses the file metadata (such as the file extension) in the request, queries the "file type-processing service" mapping table preset in the database or configuration center, determines whether the file type supports content extraction, and obtains the corresponding downstream (proxy forwarding module) processing service information; S400, proxy forwarding. After the extraction is judged to be passed, the proxy forwarding module forwards the request containing the file download address, file name, request ID, etc. to the corresponding downstream (content extraction module) through the internal service call mechanism according to the processing service information found in the mapping table; S500, content extraction. After receiving the request, the content extraction module intelligently determines whether to perform synchronous or asynchronous extraction based on the file size and processing complexity. It downloads the file through the interface provided by the object storage module and uses the corresponding parsing library or optical character recognition (OCR) technology to extract the text content. S600: Result Storage and Return: The content extraction module responds to the extracted text content or processing status according to pre-set return rules. For synchronous tasks, the extraction results can be returned directly to the requester. For asynchronous tasks, the results can be stored in the object storage module and a queryable task ID and result address can be returned to the upstream (proxy forwarding module).

[0022] Exemplarily, configuring the "file type-processing service" mapping table and the synchronous / asynchronous processing threshold in step S100 specifically includes the following steps: S101. Configure a "file type-processing service" mapping table. Create and maintain a mapping table in a distributed configuration center or database. This table should contain at least the following fields: file_extension (file extension, such as .pdf), handler_service (the service name or address of the content extraction module that handles files of this type, such as ocr-extraction-service), priority (priority, used for decision-making when extension conflicts occur), and status (status, enabled / disabled). S102: Configure synchronous / asynchronous processing threshold. In the configuration of the content extraction module, a file size threshold Sthreshold is preset, for example, Sthreshold = 20MB. This threshold is used to determine whether synchronous or asynchronous processing should be adopted.

[0023] Exemplarily, step S200 specifically includes the following steps: S201. The client constructs and sends a request. The external client (upper-layer business system) constructs an HTTPS POST request according to the API specification. The HTTPS POST request includes a request body and a request header. The request body includes in JSON format: the pre-signed download address file_url of the file to be processed in the S3 object storage, the original file name file_name, the unique request ID request_id for tracking, and a parameter response_mode specifying the return type ("direct" or "url"); The request header includes the client's AK, a Unix timestamp x-timestamp, and a signature Authorization generated according to the S3 signature algorithm V4; S202, the API gateway receives the request, and the unified API gateway of the system receives the extraction request; S203: Trigger the signature verification module. The gateway passes the request information to the signature verification module. The information includes the HTTP method, path, query parameters, request body, and request header. S204: Verify the timestamp. The signature verification module first checks the x-timestamp in the request header and compares it with the server's current time. If the time difference exceeds a preset window (e.g., 5 minutes), it is determined to be a replay attack and the request is rejected. S205: Obtain SK and reconstruct the signature. The module obtains the corresponding SK from the system security database based on the AK in the request header. If the AK is invalid, the request is rejected. Then, in strict accordance with the S3 signature algorithm specification, the server uses the same information to construct the string to be signed, String to Sign. S206: Calculate and compare the signature. Use the obtained SK to encrypt the string to be signed, generate the server signature, and compare it bit by bit with the signature "Authorization" in the client request header. S207. Process the verification result. If the signatures are inconsistent, it is determined to be an illegal request, and an HTTP 403 Forbidden error is returned to the client, and the audit log is recorded; if the signatures are consistent, the verification passes.

[0024] Exemplarily, step S300 specifically includes the following steps: S301: trigger the extraction judgment module. After the signature verification passes, the request information is passed to the extraction judgment module; S302, parsing the file suffix, the extraction judgment module parses the file suffix from the file_name field in the request body; for example, extracting ".pdf" from "financial report_Q2.pdf"; S303, query the mapping table, use the extracted suffix name as a keyword to query the configured "file type-processing service" mapping table; S304. Determine the retrievability and obtain the service address. Check the query results. If no corresponding record is found or the record status is disabled, it is determined that the file type is not supported and a specified error code (such as 415 Unsupported Media Type) is returned to the client. If the query is successful, the corresponding service name or address handler_service is obtained.

[0025] Exemplarily, step 400 specifically includes the following steps: S401. Trigger the proxy forwarding module and transfer the original request information together with the service name or address handler_service obtained in step S304 to the proxy forwarding module; S402: Execute internal forwarding. The proxy forwarding module acts as a router between services, constructs a new internal RPC or HTTP request, and sends the task information to the target content extraction module.

[0026] Exemplarily, step 500 specifically includes the following steps: S501: The content extraction module receives a task; for example, the OCR extraction module receives a task to process "Financial Report_Q2.pdf"; S502: Perform synchronous / asynchronous judgment. The extraction module first obtains the file metadata (especially Content-Length) from the S3 service through file_url, obtains the file size, and compares the file size with the file size threshold Sthreshold; S503: Asynchronous logic: If the file size exceeds the file size threshold Sthreshold, the extraction module returns a response to the proxy forwarding module, including the task ID task_id and the processing status. The task details, including the file_url, are serialized and pushed to a highly available message queue (such as RabbitMQ or Kafka). An independent background worker cluster consumes the tasks in the message queue. S504, synchronous logic / asynchronous worker execution. For small files (synchronous logic) or large file tasks consumed by background workers, the execution process downloads the file from the S3 object storage module to local temporary storage space through file_url; S505: Calling the core engine to perform extraction, performing content extraction based on the module's expertise; for example: the Office document module calls libraries such as Apache POI or python-docx to parse the document structure and extract plain text; the OCR extraction module performs page-by-page rendering, image processing, and text recognition on PDF or image files; S506. Format and store the results. Clean and organize all the extracted text content (e.g., remove extra line breaks) and merge them into a single text file (.txt). Upload this text file to a dedicated result storage bucket in the S3 object storage module. The file name is associated with the request_id or task_id.

[0027] Exemplarily, step 600 specifically includes the following steps: S601, synchronous return: For synchronously executed tasks and the response_mode requested by the client is "direct", after completing step S506, the extraction module directly puts the extracted text content into the HTTP response body and encapsulates it in JSON format, and finally returns it to the client through the proxy forwarding module; S602: Return the extracted content URL. For tasks executed synchronously but with response_mode set to "url" or all asynchronous tasks, after step S506 is completed, the system updates the task status (for example, in Redis or a database, the task_id status is updated to "completed" and the S3 address of the result file is recorded). For synchronous tasks, a response containing the S3 address is directly returned; for asynchronous tasks, the system waits for client queries. S603, asynchronous result query. For asynchronous tasks, after receiving the task_id returned in step S503, the client obtains the task progress by polling the status query API (GET / api / v1 / task_status?id={task_id}). When the task is completed, the API returns a success status and the pre-signed download address of the final result file in the S3 object storage.

[0028] Therefore, this application achieves text content extraction from all file types through a unified interface and process, improving extraction efficiency and accuracy, ensuring the security of the extraction process, and reducing development and maintenance costs. The method features unified access and standardized security authentication technology. By establishing a single API gateway with a built-in signature verification module based on the S3 standard, all extraction requests for different file types are consolidated and standardized AK / SK signature authentication is enforced. This not only greatly simplifies the integration of upper-layer applications but, more importantly, establishes a centralized and reliable security barrier, resolving security management challenges inherent in existing technologies caused by fragmented interfaces and varying authentication methods. Adaptive processing technology based on file size is introduced in the content extraction module, with synchronous / asynchronous processing decision logic based on file size thresholds. This intelligently selects the optimal processing mode based on the specific file load (measured by size): synchronous processing for small files ensures fast responses, while asynchronous processing for large files, via message queues, avoids request blocking and timeouts. This technology enables refined and intelligent management of system resources, significantly improving overall system throughput and stability.

[0029] Specifically, the present invention discloses a system for extracting text content from all types of files in a unified and aggregated manner, which is used in the above-mentioned method for extracting text content from all types of files in a unified and aggregated manner, including: Signature verification module, used to verify timestamps, obtain SK, and reconstruct signatures; Extraction judgment module, used to parse file suffixes, judge extractability and obtain service addresses; An agent forwarding module, used to send task information to a target content extraction module; Object storage module, used to store result files and provide an interface to the content extraction module; The content extraction module is used to process one or several file types and extract file content using the corresponding parsing library or optical character recognition (OCR) technology.

[0030] Illustratively, the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0031] Exemplarily, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0032] Illustratively, the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0033] Exemplarily, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0034] In summary, the current extraction tool processes for different file formats are isolated and complex to integrate; various services lack unified security authentication and management, posing a risk of data leakage; the system architecture is rigid and difficult to expand to support new file types; and the processing logic does not distinguish between file size and complexity, resulting in low processing efficiency and waste of resources. The present invention overcomes many of the shortcomings of file content extraction solutions in the prior art. By establishing a centralized service that integrates security, distribution, and flexible processing, it achieves standardized, efficient, secure, and highly scalable content extraction for all types of files, thereby providing stable and reliable data support for downstream big data analysis, content retrieval, and artificial intelligence applications.

[0035] The foregoing description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A method for extracting text content from all types of files in a unified aggregated manner, characterized in that: The following steps are involved: S100: System initialization and configuration. Deploy and start the signature verification module, extraction judgment module, proxy forwarding module, object storage module (or connect to an existing S3 service), and at least two specialized content extraction modules on a server cluster. Generate a unique access key (AK) and secret key (SK) pair in the system security database for each authorized external application or service. Configure the "file type-processing service" mapping table and synchronous / asynchronous processing thresholds. S200, signature verification: A unified API gateway is established to receive external requests for extracting file information. The signature verification module built into the gateway verifies the request using the standard S3 object storage signature algorithm based on the preset access key AK and security key SK. S300, extraction judgment: After the signature verification passes, the extraction judgment module parses the file metadata in the request, queries the "file type-processing service" mapping table preset in the database or configuration center, determines whether the file type supports content extraction, and obtains the corresponding downstream processing service information; S400, proxy forwarding: After the extraction is judged to be passed, the proxy forwarding module forwards the request containing the file download address, file name and request ID to the corresponding content extraction module through the internal service call mechanism according to the processing service information found in the mapping table; S500, content extraction. After receiving the request, the content extraction module intelligently determines whether to perform synchronous or asynchronous extraction based on the file size and processing complexity. It downloads the file through the interface provided by the object storage module and uses the corresponding parsing library or optical character recognition (OCR) technology to extract the text content. S600, result storage and return: the content extraction module responds to the extracted text content or processing status according to the preset return rules. For synchronous tasks, the extraction result is directly returned to the requester; For asynchronous tasks, the results are stored in the object storage module, and a queryable task ID and result address are returned to the proxy forwarding module.

2. A method for extracting text content from all types of files in a unified aggregated manner according to claim 1, characterized in that: The configuration of the "file type-processing service" mapping table and the synchronous / asynchronous processing threshold in step S100 specifically includes the following steps: S101. In a distributed configuration center or database, a mapping table is created and maintained. The mapping table includes a file extension, a service name or address of a content extraction module that processes files of this type, a priority, and a status. S102: Preset a file size threshold Sthreshold in the configuration of the content extraction module.

3. A method for extracting text content from all types of files in a unified aggregated manner according to claim 2, characterized in that: Step S200 specifically includes the following steps: S201. The client constructs and sends a request. The external client constructs an HTTPS POST request according to the API specification. The HTTPS POST request includes a request body and a request header. The request body includes in JSON format: the pre-signed download address file_url of the file to be processed in the S3 object storage, the original file name file_name, the unique request ID request_id for tracking, and a parameter response_mode that specifies the return type; The request header includes the client's AK, a Unix timestamp x-timestamp, and a signature Authorization generated according to the S3 signature algorithm V4; S202, the API gateway receives the request, and the unified API gateway of the system receives the extraction request; S203: Trigger the signature verification module. The gateway passes the request information to the signature verification module. The information includes the HTTP method, path, query parameters, request body, and request header. S204: Check the timestamp. The signature verification module first checks the x-timestamp in the request header and compares it with the server's current time. If the time difference exceeds the preset window, it is determined to be a replay attack and the request is rejected. S205: Obtain SK and reconstruct the signature. The module obtains the corresponding SK from the system security database based on the AK in the request header. If the AK is invalid, the request is rejected. Then, based on the S3 signature algorithm specification, the server uses the same information to construct the string to be signed, String to Sign. S206: Calculate and compare the signature. Use the obtained SK to encrypt the string to be signed, generate the server signature, and compare it bit by bit with the signature "Authorization" in the client request header. S207. Process the verification result. If the signatures are inconsistent, it is determined to be an illegal request, and an HTTP 403 Forbidden error is returned to the client, and the audit log is recorded; if the signatures are consistent, the verification passes.

4. A method for extracting text content from all types of files in a unified aggregated manner according to claim 3, characterized in that: Step S300 specifically includes the following steps: S301: trigger the extraction judgment module. After the signature verification passes, the request information is passed to the extraction judgment module; S302, parse the file suffix, and extract the file suffix from the file_name field in the request body by the extraction judgment module; S303, query the mapping table, use the extracted suffix name as a keyword to query the configured "file type-processing service" mapping table; S304. Determine the retrievability and obtain the service address. Check the query results. If no corresponding record is found or the status of the record is disabled, it is determined that the file type is not supported and a specified error code is returned to the client. If the query is successful, the corresponding service name or address handler_service is obtained.

5. A method for extracting text content from all types of files in a unified aggregated manner according to claim 4, characterized in that: Step 400 specifically includes the following steps: S401. Trigger the proxy forwarding module and transfer the original request information together with the service name or address handler_service obtained in step S304 to the proxy forwarding module; S402: Execute internal forwarding. The proxy forwarding module acts as a router between services, constructs a new internal RPC or HTTP request, and sends the task information to the target content extraction module.

6. A method for extracting text content from all types of files in a unified aggregated manner according to claim 5, characterized in that: Step 500 specifically includes the following steps: S501, the content extraction module receives the task; S502: Perform synchronous / asynchronous judgment. The extraction module first obtains the metadata of the file from the S3 service through file_url, obtains the file size, and compares the file size with the file size threshold Sthreshold; S503: Asynchronous logic: If the file size exceeds the file size threshold Sthreshold, the extraction module returns a response to the proxy forwarding module, including the task ID task_id and the processing status. The task details, including the file_url, are serialized and pushed to a highly available message queue. An independent background worker cluster consumes the tasks in the message queue. S504, synchronous logic / asynchronous Worker execution, for small files or large file tasks consumed by background workers, the execution process downloads the file from the S3 object storage module to the local temporary storage space through file_url; S505: Call the core engine to perform extraction and perform content extraction based on the module's expertise; S506: Format and store the results, clean and organize all the extracted text contents, and merge them into a single text file; upload this text file to a dedicated result storage bucket in the S3 object storage module, and associate the file name with the request_id or task_id.

7. A method for extracting text content from all types of files in a unified aggregated manner according to claim 6, characterized in that: Step 600 specifically includes the following steps: S601, synchronous return: For synchronously executed tasks and the response_mode requested by the client is "direct", after completing step S506, the extraction module directly puts the extracted text content into the HTTP response body and encapsulates it in JSON format, and finally returns it to the client through the proxy forwarding module; S602: Return the extracted content URL. For tasks that are executed synchronously but whose response_mode is "url" or all asynchronous tasks, the system updates the task status after step S506 is completed. For synchronous tasks, the system directly returns a response containing the S3 address. For asynchronous tasks, the system waits for client queries. S603, asynchronous result query. For asynchronous tasks, after receiving the task_id returned in step S503, the client obtains the task progress by polling the status query API. When the task is completed, the API returns a success status and the pre-signed download address of the final result file in the S3 object storage.

8. A system for extracting text content from all types of documents in a unified aggregated manner, used in a method for extracting text content from all types of documents in a unified aggregated manner as claimed in any one of claims 1 to 7, characterized in that: include: Signature verification module, used to verify timestamps, obtain SK, and reconstruct signatures; Extraction judgment module, used to parse the file suffix, judge the extractability and obtain the service address; An agent forwarding module, used to send task information to a target content extraction module; Object storage module, used to store result files and provide an interface to the content extraction module; Content extraction module, used to process one or several file types; Use the corresponding parsing library or optical character recognition (OCR) technology to extract file content.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images