Unstructured data processing method and device

Abstract

The invention discloses an unstructured data processing method and device. The method comprises the following steps: establishing a secure data channel between a bank external system and a bank internal unstructured data storage system; based on the security data channel, receiving an access request from a bank external system; processing unstructured data according to the type of the access request; when the access request is a query request, generating an unstructured data identifier based on a request parameter; calling a software toolkit of the unstructured data storage system to pull a corresponding file of the query request to local storage, and forming an unstructured data file and a corresponding return virtual path; receiving an access instruction for the unstructured data file; and when the access request is an uploading request, decoding the Base64 coding file, calling the software toolkit, and storing the software toolkit in an unstructured data storage system. The method and the device are used for improving the efficiency and the accuracy of querying, uploading and maintaining the non-structural data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to methods and apparatus for processing unstructured data. Background Technology

[0002] At present, banks mostly use credit review, collection and WeChat banking systems developed by external companies. However, the unstructured data generated or uploaded by these systems faces strict storage and network restrictions, namely, it is not allowed to be stored outside the bank, and the internal and external networks of the bank cannot be directly connected.

[0003] To address this issue, a new independent external image platform system needs to be built to transfer unstructured data to the bank's internal unstructured storage center via the external network. However, this solution has significant drawbacks. Firstly, it requires additional dedicated network lines, consuming substantial dedicated bandwidth. Secondly, the data is not stored locally but only used as a transaction intermediary, resulting in lengthy transaction times. Furthermore, subsequent maintenance and viewing of unstructured files necessitate retrieving data from the bank's internal P9USD storage system each time, leading to redundant consumption of internal network bandwidth. Summary of the Invention

[0004] This invention provides a method for processing unstructured data to improve the efficiency and accuracy of querying, uploading, and maintaining unstructured data. The method includes:

[0005] A secure data channel is established between the external system and the internal unstructured data storage system of the bank; the external system sends an access request containing request parameters and a Base64 encoded file to the external connection proxy server via the HTTP protocol; the external connection proxy server forwards the access request to the open area proxy server; the open area proxy server transmits the request to the internal backend application server of the bank;

[0006] Based on a secure data channel, it receives access requests from external systems.

[0007] According to the type of the access request, the unstructured data is processed; when the access request is a query request, an unstructured data identifier is generated based on the request parameters; the software toolkit of the unstructured data storage system is invoked to pull the corresponding file of the query request to local storage, forming an unstructured data file and a corresponding return virtual path; access instructions for the unstructured data file are received through the WEB server; the return virtual path is mapped to the actual file path in the local storage to provide access; when the access request is an upload request, the Base64 encoded file is decoded and then the software toolkit is invoked to store it in the unstructured data storage system.

[0008] This invention also provides an unstructured data processing apparatus to improve the efficiency and accuracy of querying, uploading, and maintaining unstructured data. The apparatus includes:

[0009] A secure data channel establishment module is used to establish a secure data channel between an external system and an internal unstructured data storage system. The external system sends an access request containing request parameters and a Base64 encoded file to an external proxy server via the HTTP protocol. The external proxy server forwards the access request to an open proxy server. The open proxy server then transmits the request to the internal backend application server.

[0010] The access request receiving module is used to receive access requests from external systems via a secure data channel.

[0011] The access request processing module is used to process access requests according to their type; when the access request is a query request, it generates an unstructured data identifier based on the request parameters; it calls the software toolkit of the unstructured data storage system to pull the corresponding file of the query request to local storage, forming an unstructured data file and a corresponding return virtual path; it receives access instructions for the unstructured data file through a web server; it maps the return virtual path to the actual file path in the local storage to provide access; when the access request is an upload request, it decodes the Base64 encoded file and calls the software toolkit to store it in the unstructured data storage system.

[0012] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described unstructured data processing method.

[0013] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described unstructured data processing method.

[0014] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described unstructured data processing method.

[0015] In this embodiment of the invention, a secure data channel is established between an external system and an internal unstructured data storage system. The external system sends an access request containing request parameters and a Base64 encoded file to an external proxy server via the HTTP protocol. The external proxy server forwards the access request to an open proxy server. The open proxy server transmits the request to the internal backend application server. Based on the secure data channel, access requests from the external system are received. Unstructured data is processed according to the type of the access request. When the access request is a query request, an unstructured data identifier is generated based on the request parameters. The software toolkit of the unstructured data storage system is invoked to retrieve the corresponding file of the query request to local storage, forming an unstructured data file and a corresponding return virtual path. Access instructions for the unstructured data file are received via a web server. The return virtual path is mapped to the actual file path in the local storage to provide access. When the access request is an upload request, the Base64 encoded file is decoded and stored in the unstructured data storage system using the software toolkit. This invention establishes a secure data channel between the external and open zone proxy servers, uses the HTTP protocol to receive access requests from external systems, and then intelligently processes them using the bank's internal backend application server. For query requests, an identifier is generated to retrieve files from central storage to the local machine and create a virtual path. The path is then mapped via a web server, allowing subsequent accesses to directly read from the local cache, avoiding repeated data retrieval and significantly reducing transaction latency and internal bandwidth consumption. For upload requests, Base64 decoding is performed before direct storage to the central system, ensuring data persistence. This invention establishes a secure data channel between the external and open zone proxy servers, receives and processes access requests from external systems based on the HTTP protocol, and intelligently manages unstructured data through local storage and virtual path mapping mechanisms, improving the efficiency and accuracy of data querying, uploading, and maintenance. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0017] Figure 1 This is a flowchart illustrating an unstructured data processing method according to an embodiment of the present invention;

[0018] Figure 2This is a specific example diagram illustrating the system architecture and data flow for unstructured data querying in an embodiment of the present invention;

[0019] Figure 3 This is a specific example diagram of the system interaction process for unstructured data upload processing in an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of the structure of an unstructured data processing device according to an embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0023] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] In the description of this specification, the terms "comprising," "including," "having," and "containing" are open-ended terms, meaning that they include but are not limited to. The terms "an embodiment," "a specific embodiment," "some embodiments," and "for example," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example that are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. The order of steps involved in the various embodiments is used to illustrate the implementation of this application, and the order of steps is not limited and can be adjusted appropriately as needed.

[0025] The acquisition, storage, use, and processing of data in this application comply with relevant regulations. The information collected in this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation interfaces are provided for users to choose to authorize or refuse.

[0026] It should be noted that in the embodiments of this application, certain existing solutions in the industry, such as software, components, and models, may be mentioned. For example, some existing software tools, components, algorithm models, or solutions well-known in other technical fields may be cited. These should be considered exemplary, and their purpose is only to illustrate the feasibility of implementing the technical solution of this application. These mentions should be understood as typical examples, and their core purpose is to illustrate and verify the rationality and feasibility of implementing the technical solution proposed in this application. However, this does not mean that the applicant has already used or necessarily used the solution. Such citations do not imply that the applicant has actually adopted these existing solutions, or that it will necessarily adopt these methods in its technical implementation process in the future. In other words, these mentions are only illustrative in nature, helping to understand the connection and transcendence of the innovation points of this application with the prior art, and do not constitute an endorsement or reliance statement on a specific prior art product.

[0027] To address the aforementioned problems, embodiments of the present invention provide an unstructured data processing method to improve the efficiency and accuracy of querying, uploading, and maintaining unstructured data. See [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart illustrating an unstructured data processing method according to an embodiment of the present invention. The method may include:

[0028] Step 101: Establish a secure data channel between the external system and the internal unstructured data storage system of the bank; the external system sends an access request containing request parameters and a Base64 encoded file to the external connection proxy server via the HTTP protocol; the external connection proxy server forwards the access request to the open area proxy server; the open area proxy server transmits the request to the internal backend application server of the bank;

[0029] Step 102: Receive access requests from external systems via a secure data channel;

[0030] Step 103: Process the unstructured data according to the type of the access request; when the access request is a query request, generate an unstructured data identifier based on the request parameters; call the software toolkit of the unstructured data storage system to pull the corresponding file of the query request to local storage, forming an unstructured data file and a corresponding return virtual path; receive the access instruction for the unstructured data file through the WEB server; map the return virtual path to the actual file path in the local storage to provide access; when the access request is an upload request, decode the Base64 encoded file and call the software toolkit to store it in the unstructured data storage system.

[0031] In this embodiment of the invention, a secure data channel is established between an external system and an internal unstructured data storage system. The external system sends an access request containing request parameters and a Base64 encoded file to an external proxy server via the HTTP protocol. The external proxy server forwards the access request to an open proxy server. The open proxy server transmits the request to the internal backend application server. Based on the secure data channel, access requests from the external system are received. Unstructured data is processed according to the type of the access request. When the access request is a query request, an unstructured data identifier is generated based on the request parameters. The software toolkit of the unstructured data storage system is invoked to retrieve the corresponding file of the query request to local storage, forming an unstructured data file and a corresponding return virtual path. Access instructions for the unstructured data file are received via a web server. The return virtual path is mapped to the actual file path in the local storage to provide access. When the access request is an upload request, the Base64 encoded file is decoded and stored in the unstructured data storage system using the software toolkit. This invention establishes a secure data channel between the external and open zone proxy servers, uses the HTTP protocol to receive access requests from external systems, and then intelligently processes them using the bank's internal backend application server. For query requests, an identifier is generated to retrieve files from central storage to the local machine and create a virtual path. The path is then mapped via a web server, allowing subsequent accesses to directly read from the local cache, avoiding repeated data retrieval and significantly reducing transaction latency and internal bandwidth consumption. For upload requests, Base64 decoding is performed before direct storage to the central system, ensuring data persistence. This invention establishes a secure data channel between the external and open zone proxy servers, receives and processes access requests from external systems based on the HTTP protocol, and intelligently manages unstructured data through local storage and virtual path mapping mechanisms, improving the efficiency and accuracy of data querying, uploading, and maintenance.

[0032] In specific implementation, the first step is to establish a secure data channel between the external system and the internal unstructured data storage system of the bank. The external system sends an access request containing request parameters and a Base64 encoded file to the external connection proxy server via the HTTP protocol. The external connection proxy server forwards the access request to the open area proxy server. The open area proxy server transmits the request to the internal backend application server of the bank.

[0033] In this embodiment, a secure data channel is established between an external system and an internal unstructured data storage system. This process begins with an access request initiated by the external system. The external system encapsulates information including the transaction serial number, creation date, and Base64-encoded file data into a request message conforming to the HTTP protocol. This request message is first transmitted via a dedicated network line to a proxy server deployed in the external network area of ​​the bank. As the first line of defense at the bank's network boundary, the external proxy server receives and inspects the request before forwarding it to a proxy server in the open area within the bank's internal network. The open area proxy server then ultimately passes the received request to a backend application server deployed within the bank's internal network environment. The backend application server then begins parsing and processing the request, thereby completing the establishment of the secure data channel and data transmission.

[0034] In specific implementation, after step 101: establishing a secure data channel between the external system and the internal unstructured data storage system of the bank, step 102: receiving access requests from the external system based on the secure data channel.

[0035] In this embodiment, based on the established secure data channel, access requests from external systems are received and processed. The bank's internal backend application server parses the received HTTP request messages to identify their specific operation types. These access requests are mainly of two types: requests to query unstructured data and requests to upload unstructured data. For query requests, the content includes business parameters for locating files, such as the transaction number and file creation date. For upload requests, in addition to necessary business parameters, the content also includes the unstructured file data itself, converted using Base64 encoding. The bank's internal backend application server distinguishes between these two different types of requests by parsing specific fields or structures in the request messages, and extracts key business parameters and data content to prepare for subsequent targeted business processing logic.

[0036] In specific implementation, after step 102: receiving access requests from external systems via a secure data channel, step 103: processing unstructured data according to the type of the access request; when the access request is a query request, generating an unstructured data identifier based on the request parameters; calling the software toolkit of the unstructured data storage system to pull the corresponding file of the query request to local storage, forming an unstructured data file and a corresponding return virtual path; receiving access instructions for the unstructured data file through a web server; mapping the return virtual path to the actual file path in the local storage to provide access; when the access request is an upload request, decoding the Base64 encoded file and calling the software toolkit to store it in the unstructured data storage system.

[0037] In one embodiment, when the access request is a query request, an unstructured data identifier is generated based on the request parameters;

[0038] When the access request is a query request, the creation date and business serial number parameters of the query request are extracted, and combined with the predefined business type to generate an unstructured data identifier.

[0039] In this embodiment, when an access request is identified as a query request, the process of generating an unstructured data identifier is initiated. First, key parameter information is extracted from the query request, primarily including the creation date recorded in the file and a unique transaction serial number. These parameters are then combined with a predefined business type from the bank's internal unstructured data storage system. Business types are standardized according to different application scenarios; for example, credit review and debt collection each have their corresponding specific type codes. The extracted creation date, transaction serial number, and determined business type code are combined and assembled according to a predetermined format and order to form a complete unstructured data identifier that conforms to the bank's internal unstructured data storage system specifications. This identifier will serve as the sole basis for accessing and locating the target unstructured data file.

[0040] In one embodiment, the software toolkit of the unstructured data storage system is invoked to retrieve the corresponding file of the query request to local storage, forming an unstructured data file and a corresponding returned virtual path, including:

[0041] Based on the unstructured data identifier, the software toolkit of the unstructured data storage system is invoked to retrieve the unstructured file corresponding to the unstructured data identifier to local storage.

[0042] The virtual path corresponding to the generated unstructured data file is returned as the result by the interface.

[0043] In the above embodiment, after generating the unstructured data identifier, a dedicated software toolkit provided by the bank's unstructured data storage system is invoked. Using the functional interface provided by this toolkit, the generated unstructured data identifier is passed as a key input parameter. The software toolkit searches and locates the target file within the bank's unstructured data storage system based on this identifier, finding one or more completely corresponding unstructured files. After location is complete, the toolkit performs a file retrieval operation, transferring the target file from the central storage system to the local environment where the bank's backend application server resides, typically on a local NAS storage disk. Once the file is successfully retrieved locally, it forms a directly accessible local unstructured data file for this query request. Subsequently, the system generates a specific virtual access path for these local files. This path does not directly point to the physical location of the file on the storage disk, but rather is a logical address that needs to be mapped. Finally, this generated virtual path is returned as the processing result to the request initiator through an interface.

[0044] In one embodiment, when the access request is an upload request, decoding the Base64 encoded file and then using the software toolkit to store it in an unstructured data storage system includes:

[0045] When the access request is an upload request, the Base64 encoded file carried in the access request is decoded;

[0046] The software toolkit of the unstructured data storage system is invoked to store the decoded file into the unstructured data storage system.

[0047] In the above embodiment, when an access request is identified as an upload request, the processing flow begins to decode the Base64-encoded file data carried in the request. This decoding process converts the encoded string data back into the original unstructured file format, generating a temporary file that can be stored. Subsequently, a dedicated software toolkit provided by the bank's unstructured data storage system, namely the P9USD system, is invoked. By calling the file storage function interface in this software toolkit, the decoded complete file data is used as input. The toolkit is responsible for establishing a connection with the P9USD storage center and performing the write operation, ultimately securely and persistently storing the file in the designated unstructured data storage system.

[0048] In one embodiment, receiving an access instruction for the unstructured data file via a web server; mapping the returned virtual path to the actual file path in the local storage to provide access includes:

[0049] Receive an access instruction from an internal platform web page; the access instruction points to the returned virtual path;

[0050] The virtual path is resolved and redirected to the actual unstructured data file in the local storage via a web server.

[0051] In this embodiment, the system receives an access command from the in-house imaging platform's web page via a web server. This access command contains virtual path information returned by a previous interface. The web server, acting as a bridging component, processes this access command. Its core function is to resolve the logical virtual path and convert it into a physical path pointing to the actual unstructured data file in the local storage system. When a user accesses this virtual path through a browser, the request is received by the web server. Internally, the server matches the virtual path with the local storage directory structure according to preset mapping rules to find the corresponding real file. After path resolution, the web server performs a redirection operation, seamlessly guiding the user's access request to the actual file in local storage, thus achieving final access to the unstructured data file. The entire process is transparent to the user; the user only perceives access to the virtual path, while the actual file is securely hosted in a controlled local storage environment.

[0052] In one embodiment, the method further includes: periodically cleaning up unstructured data files generated by query or upload operations in the local storage by configuring a scheduled task; the scheduled task is used to clean up files whose modification time is earlier than a certain threshold based on the file modification time characteristics.

[0053] In one embodiment, the system also includes an automated file lifecycle management mechanism. A scheduled task is set to periodically clean up unstructured data files temporarily stored in the local storage area due to query or upload operations. This scheduled task is automatically triggered based on a predefined time period. Its core logic is to scan all relevant files in the local storage area and check the last modified time attribute of each file. The system compares the current time with the file's last modified time and calculates the time difference. If a file's last modified time is earlier than the current time and reaches or exceeds a set threshold period, the file is determined to be an expired file that can be cleaned up. The scheduled task automatically identifies and deletes all files that meet this condition, thereby freeing up local storage space. This automated cleanup process runs continuously in a loop, ensuring the effective utilization of system storage resources and avoiding the long-term accumulation of invalid data.

[0054] Two specific embodiments are given below to illustrate the specific application of the method of the present invention.

[0055] First specific embodiment:

[0056] In this embodiment, the process begins by establishing a secure data channel. An external system sends an access request, containing the transaction serial number, creation date, and Base64-encoded file data, to the bank's external access proxy server via a dedicated network line. The external access proxy server forwards the request to the bank's open access proxy server. The open access proxy server then securely transmits the request to the bank's internal backend application server.

[0057] The bank's backend application server parses received access requests to determine their type. When the request is a query, the server extracts the creation date and transaction serial number parameters from the request, combines them with the predefined business type in the P9USD system, and generates a standard unstructured data identifier. Based on this identifier, the system calls the software development kit provided by the P9USD system to pull all the unstructured files corresponding to the identifier from the P9USD storage center to the local NAS storage disk. After the files are successfully pulled, the system generates the corresponding virtual path as the interface return result.

[0058] When the access request is an upload request, the bank's backend application server decodes the Base64 encoded file carried in the request and generates a temporary file. Then, it calls the P9USD system's software development kit to persistently store the decoded file in the P9USD storage center.

[0059] For access initiated through the in-house imaging platform's web page, the system receives the access command containing the virtual path via the web server. The web server, acting as a bridging and mapping component, resolves the virtual path and redirects it to the actual file path on the local NAS storage disk, thus enabling access to unstructured data files. For query requests initiated through the web page, the system, after retrieving the files, will also generate thumbnails of all image data and return them to the front end; for upload or delete requests, operator permission verification will be performed first.

[0060] Furthermore, the bank's backend application servers implement automatic storage space management through a scheduled task. This task executes every 30 minutes, automatically cleaning up unstructured data files on the local NAS storage disk whose modification time is more than 30 minutes earlier than the current time, ensuring efficient utilization of storage resources. The entire implementation, through a secure channel via a hierarchical proxy, local caching and virtual path mapping, and an automated cleanup mechanism, achieves the secure and efficient flow and maintenance of unstructured data within and outside the bank's network environment.

[0061] Second specific embodiment:

[0062] like Figure 2 As shown, Figure 2This diagram illustrates a specific example of the system architecture and data flow for unstructured data querying in an embodiment of the present invention. An external system initiates a query request, which passes sequentially through an external connection proxy server and an open connection proxy server via a dedicated network, ultimately reaching the bank's internal backend application server. The application server generates an unstructured data identifier based on the request parameters. By calling the P9USD system's software development kit, it queries and retrieves the corresponding unstructured data file from the P9USD storage center to its local NAS storage. Simultaneously, a virtual access path is generated and returned. Internal users access this virtual path via a web server, which then performs path mapping to access the actual file in local storage, completing the entire query process.

[0063] The system can upload unstructured data from external systems (credit review, collection, and WeChat banking developed by UnionPay Data), and transfer this data to the internal P9USD storage center through the external network. It can also jump from external systems to the internal image platform to query and manage non-result data.

[0064] Figure 3 This diagram illustrates a specific example of the system interaction flow for unstructured data upload processing in an embodiment of the present invention. The external system transmits Base64-encoded file data via a dedicated line, forwarding it to the internal backend application server through external connection and open area proxy servers. After decoding the encoded file, the application server calls the P9USD system's software development kit to store the file in the P9USD storage center. For upload requests initiated through the bank's internal web page, the system first verifies operator permissions; only after successful verification does it execute subsequent file storage operations. The entire process ensures secure transmission and reliable storage of data between the bank's internal and external network environments.

[0065] The CISWL_NG external connectivity unit is a virtual machine cluster located in the bank's external connectivity zone, serving as a critical hub for interaction between the bank's internal and external networks. This unit uses two virtual machines running in an active-active mode to ensure high service availability. Nginx software is deployed on it, primarily acting as a reverse proxy server to provide a unified access point. All network requests originating from external systems first reach this unit, where initial security filtering and request forwarding are performed, directing compliant requests to the internal open zone.

[0066] The Open Zone Deployment Unit comprises two components. The CISWL_WEB unit, deployed on two nodes using containerization technology, primarily runs an Nginx server. It is responsible for receiving and forwarding network requests from the external network zone, while also hosting the front-end page of the in-house imaging platform. Furthermore, this unit undertakes the critical functions of a web server, providing the ability to directly access and view unstructured data files via IP address, port number, and path. The Open Zone Deployment Unit uses containerization, deploying two Nginx nodes to receive and forward network requests from the external network zone, deploy the front-end page of the in-house imaging platform, and provide web server functionality, allowing files to be viewed via IP:port and path.

[0067] The CISWL_AP unit is also containerized and deployed on two nodes, using Tomcat as the application server to run the backend Java application. Notably, the CISWL_WEB frontend unit and the CISWL_AP backend application unit share a 500GB NAS storage disk. This design facilitates temporary file storage and fast access. The open zone deployment unit CISWL_AP is also containerized, deployed on two nodes, using Tomcat to deploy the backend application. The open zone WEB and AP share a 500GB NAS disk.

[0068] At the technical implementation level, the CISWL_WEB front-end unit is developed using the Vue technology stack to build the user interface for the imaging platform. A key feature is the provision of a secure redirect link from the UnionPay data system to the internal imaging platform, embedding the identity and permission information of the currently logged-in operator. This design allows the imaging platform to precisely control user access permissions for operations such as uploading, deleting, querying, and downloading based on the input permission parameters. Simultaneously, the front-end interface also supports direct access to peripheral hardware such as printers and scanners, enhancing the system's functionality.

[0069] The CISWL_AP backend unit is developed using the Java technology stack, building a complete backend service system. This unit exposes five HTTP interfaces, covering core operations for unstructured data management, including data querying, file uploading, and data deletion. These interfaces receive frontend requests, process business logic, and interact with the P9USD storage system, collectively forming a complete unstructured data processing platform.

[0070] Among them, CISWL_AP: uses the JAVA technology stack to develop backend services, and exposes a total of 5 HTTP interfaces:

[0071] The specific implementation process of the UnionPay direct query interface is as follows: The UnionPay data-side system, acting as the request initiator, uses a dedicated interface to transmit the creation date and transaction serial number related to the unstructured data storage as query parameters via the bank's dedicated network line. This request first reaches the proxy server deployed in the bank's external network area, undergoes security verification, and is then forwarded to the open network proxy server, ultimately reaching the backend application service. Upon receiving these parameters, the backend application standardizes and assembles them with the predefined business types of the P9USD system to generate a complete unstructured data identifier. Subsequently, the application calls the software development kit provided by the P9USD system, using this identifier to retrieve and pull all corresponding unstructured files from the P9USD storage center to the local NAS storage disk. After processing, the system returns the virtual access path corresponding to the file. The UnionPay data side can then access the file via the IP address and port number combined with this path, where the mapping from the virtual path to the actual file path is handled by the web server.

[0072] The processing flow of the UnionPay direct upload interface is as follows: The UnionPay data side transmits the creation date, business transaction number, and Base64 encoded file data via a dedicated line. The request is forwarded sequentially through the external connection zone proxy and the open zone proxy, finally reaching the backend application service. The backend application first decodes and restores the Base64 encoded file data, generating a temporary file. Then, following the same logic as the query interface, it generates an unstructured data identifier and calls the P9USD system's software development kit to persistently store the file in the P9USD storage center.

[0073] The open zone web page query interface has been enhanced with improved security and user experience compared to the direct connection query interface of the bank's data platform. When a query request is initiated from the web page of the bank's imaging platform, the system first performs an operator permission check to verify whether the current user has query privileges. After successful verification, the system processes the request according to the query interface's workflow, but before returning the results, it processes all image data files in a loop, automatically generates corresponding thumbnails, and synchronously returns this thumbnail data to the front-end interface for users to quickly preview.

[0074] The open zone web page upload interface has also added a security control mechanism. When processing upload requests from the in-house imaging platform web page, the system first verifies the operator's upload permissions. After the permission verification is successful, the subsequent processing flow is consistent with the direct upload interface of the bank's financial data platform, ensuring that the file is securely stored in the P9USD system.

[0075] The open zone web page's delete interface is specifically designed for file deletion operations. After the user selects the image data to be deleted on the front-end interface and confirms the deletion, the front-end sends the unstructured data identifier and the specific file name to the back-end service. The back-end service first verifies whether the current operator has deletion permissions. If the verification is successful, it calls the P9USD system's software development kit to execute the deletion operation and returns the processing result to the front-end interface to notify the user.

[0076] In addition, the CISWL_AP server introduces an automated storage management mechanism, which uses a cron job to periodically clean up temporary unstructured files. This task is executed cyclically at fixed time intervals, specifically set to automatically trigger the cleanup process every thirty minutes.

[0077] Once the scheduled task starts, the system automatically scans the local NAS storage disk on the server for all unstructured data files generated by upload and query operations. The core criterion for the cleanup process is the file modification time attribute. The system obtains the last modification timestamp of each file and compares it with the current system time. By calculating the time difference, it accurately identifies files whose last modification time is 30 minutes or more earlier than the current time. These files are judged by the system as expired temporary files and no longer have immediate use value.

[0078] Once files matching the criteria are identified, a scheduled task automatically performs deletion operations, completely removing these expired files from the local storage disk. This process is fully automated, requiring no manual intervention, ensuring the continuous and effective use of storage space resources. Through this periodic cleanup mechanism, the system effectively prevents disk space exhaustion caused by the continuous accumulation of temporary files, while also helping to maintain stable system performance. This design improves the lifecycle management of unstructured data, providing an important guarantee for the long-term stable operation of the system.

[0079] This application establishes a secure data channel via proxy servers in the external and open zones, enabling the secure transmission of unstructured data generated by external systems to the bank's internal P9USD storage center for centralized and persistent storage. This effectively solves the data storage limitations caused by network isolation between internal and external systems, while adhering to bank data security standards. By designing query and caching mechanisms in the backend application server, files are pulled from P9USD to generate virtual paths locally when processing query requests, and path mapping is performed using a web server. This allows unstructured data stored internally to be accessed and displayed securely and efficiently, solving the problem of data not being directly previewable and accessible under centralized storage. Furthermore, by developing an internal image platform webpage integrating operator permission verification, a unified graphical interface is provided to bank users, supporting convenient query and maintenance functions including thumbnail generation, file upload, and deletion, greatly improving the operational efficiency and user experience of unstructured data management.

[0080] The core of this application lies in the complete internalization of the unstructured data processing workflow. Specifically, all relevant business systems, such as credit review and collection, are developed and deployed by the bank itself within its internal network environment, fundamentally avoiding the issue of data needing to be transmitted or transmitted from the bank's external network. Since all system components reside within the bank's internal network, data flows within the internal security domain throughout its entire lifecycle, eliminating the need to consider the challenges of complex security verification, dedicated line transmission, and data anonymization associated with crossing network boundaries. Under this architecture, the responsibility for storing and displaying unstructured data is delegated to each independent business system component. Each business system needs to independently implement its interface with the P9USD storage center and is independently responsible for handling the display logic of unstructured files generated by its business, such as developing its own file preview and download functions. While this avoids the need to build a unified transmission platform, it also leads to problems such as redundant functionalities, inconsistent user experience, and high maintenance costs. This solution reflects a decentralized management approach, which contrasts sharply with the unified, efficient, and secure data channel platform advocated by this invention.

[0081] In this application, the image platform is the core system for realizing the full lifecycle management of unstructured data in this solution. As a unified integrated platform, it is responsible for handling a full set of operations, including uploading, storing, retrieving, downloading, and online viewing of various unstructured data, including images, audio, video, and documents. It provides a centralized data management portal and a visual operation interface for internal and external users of the bank.

[0082] Unstructured data specifically refers to information carriers that do not have a predefined data model or fixed format, such as ID card photos, contract signing recordings, authentication videos, and various scanned documents generated during business transactions. Unlike structured data in databases, this type of data has diverse formats and irregular structures, and its storage, indexing, and querying require specific technical processing methods.

[0083] P9USD is the specific name of the bank's internal dedicated unstructured data storage system, responsible for the reliable, secure storage and efficient management of massive amounts of unstructured data. This system provides the underlying storage capabilities for this solution, and all final data processed through the imaging platform is persistently stored in this system.

[0084] External connectivity refers to the complete process of data exchange and business interaction between the bank's internal network and external networks (such as third-party service provider networks). This process involves strict security controls. This application utilizes technologies such as dedicated line connections and proxy server forwarding to achieve secure and controllable data flow between the bank's internal and external environments while ensuring network security.

[0085] Of course, it is understood that there may be other variations of the above detailed process, and all such variations should fall within the protection scope of this invention.

[0086] In this embodiment of the invention, a secure data channel is established between an external system and an internal unstructured data storage system. The external system sends an access request containing request parameters and a Base64 encoded file to an external proxy server via the HTTP protocol. The external proxy server forwards the access request to an open proxy server. The open proxy server transmits the request to the internal backend application server. Based on the secure data channel, access requests from the external system are received. Unstructured data is processed according to the type of the access request. When the access request is a query request, an unstructured data identifier is generated based on the request parameters. The software toolkit of the unstructured data storage system is invoked to retrieve the corresponding file of the query request to local storage, forming an unstructured data file and a corresponding return virtual path. Access instructions for the unstructured data file are received via a web server. The return virtual path is mapped to the actual file path in the local storage to provide access. When the access request is an upload request, the Base64 encoded file is decoded and stored in the unstructured data storage system using the software toolkit. This invention establishes a secure data channel between the external and open zone proxy servers, uses the HTTP protocol to receive access requests from external systems, and then intelligently processes them using the bank's internal backend application server. For query requests, an identifier is generated to retrieve files from central storage to the local machine and create a virtual path. The path is then mapped via a web server, allowing subsequent accesses to directly read from the local cache, avoiding repeated data retrieval and significantly reducing transaction latency and internal bandwidth consumption. For upload requests, Base64 decoding is performed before direct storage to the central system, ensuring data persistence. This invention establishes a secure data channel between the external and open zone proxy servers, receives and processes access requests from external systems based on the HTTP protocol, and intelligently manages unstructured data through local storage and virtual path mapping mechanisms, improving the efficiency and accuracy of data querying, uploading, and maintenance.

[0087] This invention also provides an unstructured data processing apparatus, as described in the following embodiments. Since the principle by which this apparatus solves the problem is similar to that of the unstructured data processing method, its implementation can be found in the implementation of the unstructured data processing method, and repeated details will not be elaborated further.

[0088] This invention also provides an unstructured data processing device to improve the efficiency and accuracy of querying, uploading, and maintaining unstructured data. Figure 4 This is a schematic diagram of the structure of an unstructured data processing device according to an embodiment of the present invention, as shown below. Figure 4 As shown, the device includes:

[0089] The secure data channel establishment module 401 is used to establish a secure data channel between the external system and the internal unstructured data storage system of the bank; the external system sends an access request containing request parameters and a Base64 encoded file to the external connection zone proxy server via the HTTP protocol; the external connection zone proxy server forwards the access request to the open zone proxy server; the open zone proxy server transmits the request to the internal backend application server of the bank;

[0090] Access request receiving module 402 is used to receive access requests from external systems via a secure data channel;

[0091] The access request processing module 403 is used to process the access request according to the type of the access request; when the access request is a query request, it generates an unstructured data identifier based on the request parameters; it calls the software toolkit of the unstructured data storage system to pull the corresponding file of the query request to local storage, forming an unstructured data file and a corresponding return virtual path; it receives the access instruction for the unstructured data file through the WEB server; it maps the return virtual path to the actual file path in the local storage to provide access; when the access request is an upload request, it decodes the Base64 encoded file and calls the software toolkit to store it in the unstructured data storage system.

[0092] In one embodiment, when the access request is a query request, an unstructured data identifier is generated based on the request parameters;

[0093] When the access request is a query request, the creation date and business serial number parameters of the query request are extracted, and combined with the predefined business type to generate an unstructured data identifier.

[0094] In one embodiment, the software toolkit of the unstructured data storage system is invoked to retrieve the corresponding file of the query request to local storage, forming an unstructured data file and a corresponding returned virtual path, including:

[0095] Based on the unstructured data identifier, the software toolkit of the unstructured data storage system is invoked to retrieve the unstructured file corresponding to the unstructured data identifier to local storage.

[0096] The virtual path corresponding to the generated unstructured data file is returned as the result by the interface.

[0097] In one embodiment, when the access request is an upload request, decoding the Base64 encoded file and then using the software toolkit to store it in an unstructured data storage system includes:

[0098] When the access request is an upload request, the Base64 encoded file carried in the access request is decoded;

[0099] The software toolkit of the unstructured data storage system is invoked to store the decoded file into the unstructured data storage system.

[0100] In one embodiment, receiving an access instruction for the unstructured data file via a web server; mapping the returned virtual path to the actual file path in the local storage to provide access includes:

[0101] Receive an access instruction from an internal platform web page; the access instruction points to the returned virtual path;

[0102] The virtual path is resolved and redirected to the actual unstructured data file in the local storage via a web server.

[0103] In one embodiment, it also includes:

[0104] By configuring a scheduled task, unstructured data files generated by query or upload operations in the local storage are cleaned up periodically; the scheduled task is used to clean up files whose modification time is earlier than a certain threshold based on the file modification time characteristics.

[0105] This invention provides an embodiment of a computer device for implementing all or part of the above-described unstructured data processing methods. The computer device specifically includes the following components:

[0106] The computer device comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between related devices; the computer device can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the computer device can be implemented with reference to the embodiments for implementing unstructured data processing methods and unstructured data processing devices, the contents of which are incorporated herein by reference, and repeated details will not be described again.

[0107] Figure 5 This is a schematic diagram of a computer device provided in an embodiment of the present invention, which discloses a schematic block diagram of the system configuration of a computer device 1000 according to an embodiment of this application. Figure 5 As shown, the computer device 1000 may include a central processing unit 1001 and a memory 1002; the memory 1002 is coupled to the central processing unit 1001. It is worth noting that... Figure 5 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0108] In one embodiment, unstructured data processing functionality can be integrated into the central processing unit 1001. The central processing unit 1001 can be configured to perform the following control:

[0109] A secure data channel is established between the external system and the internal unstructured data storage system of the bank; the external system sends an access request containing request parameters and a Base64 encoded file to the external connection proxy server via the HTTP protocol; the external connection proxy server forwards the access request to the open area proxy server; the open area proxy server transmits the request to the internal backend application server of the bank;

[0110] Based on a secure data channel, it receives access requests from external systems.

[0111] According to the type of the access request, the unstructured data is processed; when the access request is a query request, an unstructured data identifier is generated based on the request parameters; the software toolkit of the unstructured data storage system is invoked to pull the corresponding file of the query request to local storage, forming an unstructured data file and a corresponding return virtual path; access instructions for the unstructured data file are received through the WEB server; the return virtual path is mapped to the actual file path in the local storage to provide access; when the access request is an upload request, the Base64 encoded file is decoded and then the software toolkit is invoked to store it in the unstructured data storage system.

[0112] In another embodiment, the unstructured data processing device can be configured separately from the central processing unit 1001. For example, the unstructured data processing device can be configured as a chip connected to the central processing unit 1001, and the unstructured data processing function can be implemented through the control of the central processing unit.

[0113] like Figure 5 As shown, the computer device 1000 may further include: a communication module 1003, an input unit 1004, an audio processor 1005, a display 1006, and a power supply 1007. It is worth noting that the computer device 1000 does not necessarily need to include... Figure 5 All components shown; in addition, the computer device 1000 may also include Figure 5 For components not shown, please refer to existing technologies.

[0114] like Figure 5As shown, the central processing unit 1001, sometimes also referred to as a controller or operation control, may include a microprocessor or other processor device and / or logic device. The central processing unit 1001 receives input and controls the operation of various components of the computer device 1000.

[0115] The memory 1002 may be, for example, one or more of a cache, flash memory, hard drive, removable medium, volatile memory, non-volatile memory, or other suitable device. It can store the aforementioned device-related information, and may also store programs for executing that information. The central processing unit 1001 can execute the program stored in the memory 1002 to perform information storage or processing, etc.

[0116] Input unit 1004 provides input to central processing unit 1001. This input unit 1004 may be, for example, a keypad or touch input device. Power supply 1007 provides power to computer device 1000. Display 1006 displays images, text, and other display objects. This display may be, for example, an LCD display, but is not limited to this.

[0117] The memory 1002 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 1002 can also be some other type of device. The memory 1002 includes a buffer memory 1021 (sometimes referred to as a buffer). The memory 1002 may include an application / function storage unit 1022 for storing application programs and function programs or processes for executing operations of the computer device 1000 via the central processing unit 1001.

[0118] The memory 1002 may also include a data storage unit 1023 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the computer device. The driver storage unit 1024 of the memory 1002 may include various drivers for the computer device for communication functions and / or for performing other functions of the computer device (such as messaging applications, address book applications, etc.).

[0119] The communication module 1003 is a transmitter / receiver that transmits and receives signals via the antenna 1008. The communication module (transmitter / receiver) 1003 is coupled to the central processing unit 1001 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0120] Based on different communication technologies, multiple communication modules 1003 can be configured in the same computer device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 1003 is also coupled to a speaker 1009 and a microphone 1010 via an audio processor 1005 to provide audio output via the speaker 1009 and receive audio input from the microphone 1010, thereby realizing typical telecommunications functions. The audio processor 1005 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 1005 is also coupled to a central processing unit 1001, enabling on-device recording via the microphone 1010 and on-device playback of stored sound via the speaker 1009.

[0121] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described unstructured data processing method.

[0122] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described unstructured data processing method.

[0123] In this embodiment of the invention, a secure data channel is established between an external system and an internal unstructured data storage system. The external system sends an access request containing request parameters and a Base64 encoded file to an external proxy server via the HTTP protocol. The external proxy server forwards the access request to an open proxy server. The open proxy server transmits the request to the internal backend application server. Based on the secure data channel, access requests from the external system are received. Unstructured data is processed according to the type of the access request. When the access request is a query request, an unstructured data identifier is generated based on the request parameters. The software toolkit of the unstructured data storage system is invoked to retrieve the corresponding file of the query request to local storage, forming an unstructured data file and a corresponding return virtual path. Access instructions for the unstructured data file are received via a web server. The return virtual path is mapped to the actual file path in the local storage to provide access. When the access request is an upload request, the Base64 encoded file is decoded and stored in the unstructured data storage system using the software toolkit. This invention establishes a secure data channel between the external and open zone proxy servers, uses the HTTP protocol to receive access requests from external systems, and then intelligently processes them using the bank's internal backend application server. For query requests, an identifier is generated to retrieve files from central storage to the local machine and create a virtual path. The path is then mapped via a web server, allowing subsequent accesses to directly read from the local cache, avoiding repeated data retrieval and significantly reducing transaction latency and internal bandwidth consumption. For upload requests, Base64 decoding is performed before direct storage to the central system, ensuring data persistence. This invention establishes a secure data channel between the external and open zone proxy servers, receives and processes access requests from external systems based on the HTTP protocol, and intelligently manages unstructured data through local storage and virtual path mapping mechanisms, improving the efficiency and accuracy of data querying, uploading, and maintenance.

[0124] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0126] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0128] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of processing unstructured data, characterized by, include: Establish a secure data channel between external systems and internal unstructured data storage systems within the bank; The external system of the bank sends an access request containing request parameters and a Base64 encoded file to the external connection area proxy server via the HTTP protocol; the external connection area proxy server forwards the access request to the open area proxy server; the open area proxy server transmits the request to the bank's internal backend application server; Based on a secure data channel, it receives access requests from external systems. The unstructured data is processed according to the type of the access request; when the access request is a query request, an unstructured data identifier is generated based on the request parameters. The software toolkit of the unstructured data storage system is invoked to pull the corresponding file of the query request to the local storage, forming an unstructured data file and the corresponding return virtual path; The system receives access instructions for the unstructured data file via a web server; maps the returned virtual path to the actual file path in the local storage to provide access; and when the access request is an upload request, decodes the Base64 encoded file and calls the software toolkit to store it in the unstructured data storage system.

2. The method as described in claim 1, characterized in that, When the access request is a query request, an unstructured data identifier is generated based on the request parameters; When the access request is a query request, the creation date and business serial number parameters of the query request are extracted, and combined with the predefined business type to generate an unstructured data identifier.

3. The method as described in claim 1, characterized in that, The software toolkit of the unstructured data storage system is invoked to retrieve the corresponding file for the query request to local storage, forming an unstructured data file and a corresponding virtual path, including: Based on the unstructured data identifier, the software toolkit of the unstructured data storage system is invoked to retrieve the unstructured file corresponding to the unstructured data identifier to local storage. The virtual path corresponding to the generated unstructured data file is returned as the result by the interface.

4. The method as described in claim 1, characterized in that, When the access request is an upload request, the Base64 encoded file is decoded and then stored in an unstructured data storage system using the software toolkit, including: When the access request is an upload request, the Base64 encoded file carried in the access request is decoded; The software toolkit of the unstructured data storage system is invoked to store the decoded file into the unstructured data storage system.

5. The method as described in claim 1, characterized in that, The system receives access instructions for the unstructured data file via a web server; maps the returned virtual path to the actual file path in the local storage to provide access, including: Receive an access instruction from an internal platform web page; the access instruction points to the returned virtual path; The virtual path is resolved and redirected to the actual unstructured data file in the local storage via a web server.

6. The method as described in claim 1, characterized in that, Also includes: By configuring scheduled tasks, unstructured data files generated by query or upload operations in the local storage can be cleaned up periodically. The scheduled task is used to clean up files whose modification time is earlier than a certain threshold based on the file's modification time characteristics.

7. An unstructured data processing device, characterized in that, include: The secure data channel establishment module is used to establish a secure data channel between external systems and internal unstructured data storage systems of the bank. The external system of the bank sends an access request containing request parameters and a Base64 encoded file to the external connection area proxy server via the HTTP protocol; the external connection area proxy server forwards the access request to the open area proxy server; the open area proxy server transmits the request to the bank's internal backend application server; The access request receiving module is used to receive access requests from external systems via a secure data channel. An access request processing module is used to process access requests according to their type. When the access request is a query request, an unstructured data identifier is generated based on the request parameters; The software toolkit of the unstructured data storage system is invoked to pull the corresponding file of the query request to the local storage, forming an unstructured data file and the corresponding return virtual path; The system receives access instructions for the unstructured data file via a web server; maps the returned virtual path to the actual file path in the local storage to provide access; and when the access request is an upload request, decodes the Base64 encoded file and calls the software toolkit to store it in the unstructured data storage system.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

Images