Memory process management method applied to customs intelligent supervision and related equipment

Abstract

This application relates to the field of smart customs technology, and provides a memory process management method and related equipment for intelligent customs supervision. By deriving worker processes from the main process and executing initialization routines after deriving, the feature parameter files and model files for customs declaration risk identification are preloaded into the memory space of each worker process. A grouped index is pre-built based on the core customs business keys. A listening thread monitors changes to program files and triggers a full restart of the worker processes. An independent cache hot update listening thread is established within each worker process to ensure uninterrupted business operations during risk control parameter updates. The complexity of matching each customs declaration with feature parameters is reduced from a full traversal to an index lookup. The cache query interface checks the source file modification time and triggers synchronous reload before returning data, enabling the model service to balance processing efficiency, data consistency, and service continuity when processing customs declaration data.

Description

Technical Field

[0001] This application relates to the field of smart customs technology, and more specifically, to a memory process management method and related equipment for smart customs supervision. Background Technology

[0002] With the continuous growth of international trade, customs supervision faces enormous pressure in processing massive amounts of customs declaration data. Utilizing artificial intelligence models to automate risk identification of customs declarations has become a key means to improve regulatory efficiency. Existing model service systems, in real-time processing, require rereading a large number of feature parameter files and large model files from the disk for each customs declaration.

[0003] These frequent disk I / O operations became the main time-consuming nodes, severely restricting the overall processing efficiency. At the same time, the data volume of some feature parameters was huge, and the traditional loop traversal method used in the matching process resulted in a huge amount of computation, which further increased the identification time of a single shipment, resulting in low identification efficiency of customs declaration risks. Summary of the Invention

[0004] This application provides a memory process management method and related equipment for customs intelligent supervision, which can at least partially solve the problem of low efficiency in identifying customs declaration risks.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to one aspect of this application, a memory process management method for customs intelligent supervision is provided, comprising: starting a main process for a model service; creating an independent file system listening thread, wherein the file system listening thread continuously monitors change events of program files; the main process spawns at least one worker process, each worker process executing an initialization routine after being spawned, the initialization routine comprising: loading the required feature parameter files and machine learning model files from disk into the memory space of the worker process, constructing a global cache object, and pre-constructing a grouped index based on the key fields of the feature parameter files; when the file system listening thread detects a change in the program files, sending a restart command to the main process, the main process responding to the restart command restarting all worker processes, and the restarted worker processes re-executing the initialization routine to achieve a full update of the program. Cache refresh; In each of the worker processes, a cache hot update monitoring thread is established independently of the file system monitoring thread. The cache hot update monitoring thread periodically scans the disk directory of the feature parameter file and the model file in a polling manner to monitor the file change status under the disk directory. When the cache hot update monitoring thread detects that a file under the disk directory has changed, it triggers a cache refresh, reloads the changed file and updates the corresponding cache area in the global cache object, or cleans up the cache entries corresponding to deleted files from the global cache object. After receiving a customs declaration data request, the worker process accesses the memory data in the global cache object, completes data matching based on the key fields of the current customs declaration and the grouping index, and calls the corresponding executable model object from the model file to perform risk identification calculation, and returns the calculation result.

[0007] In this application, based on the aforementioned scheme, the step of loading the required feature parameter files and machine learning model files from the disk into the memory space of the working process, constructing a global cache object, and pre-constructing a grouping index based on the key fields of the feature parameter files includes: submitting a structured query request to the big data platform through the library interface provided by the big data platform to obtain the full set of feature parameter files and machine learning model files; converting the feature parameter files into a Pandas DataFrame format dataset in the memory space; performing a grouping operation on the Pandas DataFrame format dataset according to the preset core business key, grouping and extracting the parameter records in the feature parameter files according to the import / export port code and trade method to generate multiple lightweight quantum sets, and constructing corresponding grouping indexes for the lightweight quantum sets in the memory space; storing the lightweight quantum sets formed after grouping and their corresponding grouping indexes as grouping parameter sets in the dictionary members of the global cache object.

[0008] In this application, based on the aforementioned scheme, loading the machine learning model file from the disk into the memory space of the working process includes: sending a model file read request to the resource storage of the big data platform to obtain a serialized model file byte stream; converting the model file byte stream into an executable model object in the memory space through a deserialization operation; and storing the executable model object in the dictionary member of the global cache object.

[0009] In this application, based on the aforementioned scheme, the cache hot update monitoring thread periodically scans the disk directory of the feature parameter file and the model file in a polling manner to monitor the file change status under the disk directory, including: the cache hot update monitoring thread is started independently within the working process, with a preset time interval as the polling cycle; in each polling cycle, the last modification timestamp or MD5 value of a specific pattern file under the disk directory is read; the read last modification timestamp or MD5 value is compared with the loading timestamp or MD5 value recorded in the global cache object to determine whether the file under the disk directory has been added, modified, or deleted.

[0010] In this application, based on the aforementioned scheme, the step of triggering a cache refresh when the cache hot update monitoring thread detects a change in a file under the disk directory, reloading the changed file and updating the corresponding cache area in the global cache object, or clearing the cache entries corresponding to deleted files from the global cache object, includes: triggering a cache refresh when the cache hot update monitoring thread detects a new file or a change in the last modification timestamp of an existing file under the disk directory; reloading the changed file from the disk into memory and updating the corresponding cache area in the global cache object; and triggering the cache refresh when the cache hot update monitoring thread determines that a file under the disk directory has been deleted, clearing the cache entries corresponding to the deleted file from the global cache object.

[0011] In this application, based on the aforementioned scheme, the step of accessing the memory data in the global cache object and completing data matching based on the key fields of the current customs declaration combined with the grouping index includes: accessing the memory data in the global cache object and obtaining the key fields of the current customs declaration, the key fields including import / export method, port code, and tax number; locating a pre-constructed lightweight quantum set corresponding to the scenario of the customs declaration from the global cache object based on the obtained import / export method, port code, and tax number of the current customs declaration; and using the merge vectorization calculation method of the Pandas library to associate and match the current customs declaration data with the located lightweight quantum set.

[0012] In this application, based on the aforementioned scheme, before performing risk identification calculation after receiving the customs declaration data request, the workflow further includes: when the externally provided cache query interface is called by a business request, checking the last modification timestamp of the corresponding source file under the source file path before returning data; comparing the checked last modification timestamp of the source file with the cache loading timestamp recorded in the global cache object; if it is determined that the source file has been modified after the cache loading, triggering the synchronous reload operation of the cache item, reloading the source file from the disk and updating the global cache object; after completing the synchronous reload operation, returning the updated cache data through the cache query interface for the risk identification calculation.

[0013] According to one aspect of this application, a memory process management device for intelligent customs supervision is provided, comprising: A module is created to start the main process of the model service and to create an independent file system listening thread, which continuously monitors change events of program files. An initialization module is used for the main process to spawn at least one worker process. Each worker process executes an initialization routine after being spawned. The initialization routine includes: loading the required feature parameter files and machine learning model files from the disk into the memory space of the worker process, constructing them as a global cache object, and pre-constructing a grouped index based on the key fields of the feature parameter files. The refresh module is used to send a restart command to the main process when the file system listening thread detects that the program file has been changed. The main process responds to the restart command and restarts all working processes. The restarted working processes re-execute the initialization routine to achieve a full cache refresh after the program is updated. The monitoring module is used to establish a cache hot update monitoring thread independently of the file system monitoring thread in each of the working processes. The cache hot update monitoring thread periodically scans the disk directory of the feature parameter file and the model file in a polling manner to monitor the file change status under the disk directory. The caching module is used to trigger a cache refresh when the cache hot update monitoring thread detects that a file in the disk directory has changed. This refresh will reload the changed file and update the corresponding cache area in the global cache object, or clean up the cache entries corresponding to deleted files from the global cache object. The identification module is used by the working process to access the memory data in the global cache object after receiving the customs declaration data request, complete the data matching based on the key fields of the current customs declaration and the group index, and call the corresponding executable model object from the model file to perform risk identification calculation and return the calculation result.

[0014] According to one aspect of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the memory process management method for customs intelligent supervision as described in the above embodiments.

[0015] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the memory process management method for customs intelligent supervision as described in the above embodiments.

[0016] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the memory process management method for customs intelligent supervision provided in the various optional implementations described above.

[0017] The main differences and technical effects of the technical solution of this application compared with the prior art are as follows: This technical solution derives worker processes from the main process and executes initialization routines after deriving. It preloads the feature parameter files and model files for customs declaration risk identification into the memory space of each worker process and constructs a global cache object. Simultaneously, it pre-builds grouped indexes based on core customs business keys such as import / export port codes and trade methods, ensuring that the processing path for each customs declaration data request is entirely in-memory, eliminating frequent disk I / O as the main source of time consumption. A file system monitoring thread monitors program file changes and triggers a full restart of worker processes, ensuring consistent cache status across worker processes after rule code updates. An independent cache hot update monitoring thread is established within each worker process to perform incremental loading or cleanup of individual changed files, ensuring uninterrupted business operations during risk control parameter updates. The pre-built grouped indexes, combined with vectorized calculation methods, reduce the matching complexity between each customs declaration and feature parameters from a full traversal to an index lookup. The cache query interface checks the source file modification time and triggers synchronous reload before returning data, further narrowing the data inconsistency time window, enabling the model service to balance processing efficiency, data consistency, and service continuity when processing customs declaration data.

[0018] Compared with the prior art, the beneficial effects of the technical solution of this application are: 1. Significant Performance Improvement, Achieving "Process Reengineering": The technical solution in this application shifts the main time-consuming steps of model screening from disk I / O to memory access, achieving a fundamental optimization of "reducing multiple reads to a single read." According to actual application data, the screening time for a single customs declaration form is reduced from 4.12 seconds to 1.26 seconds, a performance improvement of 69%, and the proportion of documents taking more than 5 seconds decreases from 4.5% to 0%. This technological breakthrough enables the compression and reconstruction of the originally sequential and time-consuming review process, directly reflecting "process reengineering" at the technical level.

[0019] 2. Strong Technological Innovation: The technical solution in this application creatively combines the "preloading" and "listening and refreshing" mechanisms, resolving the contradiction between data real-time performance and resource loading efficiency. This is not a simple accumulation of technologies, but a "golden idea"-style application innovation for a specific business scenario, similar to the concept of "distillation," extracting the core bottlenecks and solving them precisely.

[0020] 3. Enhance regulatory efficiency and create "new-quality regulatory power": Increased efficiency means that more customs declarations can be processed in the same amount of time, and risk assessment is more timely and accurate. This directly promotes the transformation of customs supervision from relying on manpower and experience to relying on data-driven and intelligent algorithms. It is the embodiment of "new-quality productivity" in the field of supervision, that is, through the application of new technologies (this optimized system), new elements (efficient data utilization) and new facilities (high-performance model services) are brought about, ultimately achieving a qualitative change in the regulatory system (process).

[0021] 4. High system stability and maintainability: The hot update mechanism ensures that business is not interrupted during parameter updates, and the system can apply the latest rules without restarting, which improves system stability and operation and maintenance efficiency. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0023] Figure 1 The flowchart illustrating a memory process management method applied to intelligent customs supervision in one embodiment of this application is shown.

[0024] Figure 2 The flowchart illustrating the generation of a grouping index is shown in one embodiment of this application.

[0025] Figure 3The diagram illustrates a line graph comparing the average response latency under different data sizes in one embodiment of this application.

[0026] Figure 4 The diagram illustrates a line graph comparing the average response latency under different concurrent requests in one embodiment of this application.

[0027] Figure 5 The diagram illustrates a line graph showing the monitoring results of hot updates and resource usage in one embodiment of this application.

[0028] Figure 6 The diagram illustrates a bar chart of monitoring results for hot updates and resource usage in one embodiment of this application.

[0029] Figure 7 The illustration shows a schematic diagram of a memory process management device applied to intelligent customs supervision in one embodiment of this application.

[0030] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0031] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0032] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0033] It should be noted that the data acquisition or information collection in this embodiment is performed after authorization by the user or the object of collection, and its process and purpose strictly follow the relevant regulations.

[0034] The block diagrams shown in the attached figures are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more hardware modules composed of smart chips, smart integrated circuits, or application-specific integrated circuits (ASICs), or in different network and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0036] This system is deployed on Elastic Compute Service (ECS), a cloud server that can scalably adjust computing resource configurations according to business needs. The instance specification is selected as compute-optimized (e.g., ecs.c6.xlarge or higher) to ensure sufficient memory resources for model and parameter file preloading. The operating system is CentOS 7.9 or Alibaba Cloud Linux 3. All feature parameter data (such as customs declaration history, risk parameter rules, etc.) is stored in the cloud big data computing service (MaxCompute, formerly ODPS). MaxCompute is a fast, fully managed TB / PB-level data warehouse solution specifically designed for storing massive amounts of structured data and offline batch computing. ODPS is responsible for the storage, batch processing, and scheduling of massive business data. The final feature parameter table after preprocessing and processing, as well as the serialized files output from model training, are the data sources directly used by the system.

[0037] The model service is developed using Python 3.8+ and built on the high-performance web framework FastAPI. FastAPI is a modern Python web framework based on the ASGI (Asynchronous Server Gateway Interface) standard, which natively supports asynchronous request processing and can effectively improve concurrent processing performance in I / O-intensive scenarios. The service provides a RESTful API interface for customs declaration business systems to call. The service runs behind the Gunicorn WSGI server, a Python WSGI / ASGI server that uses a pre-fork multi-process model to achieve request isolation and support high-concurrency request processing. The service interacts with ODPS through the officially provided pyodps library, a Python SDK for big data computing services provided by the cloud provider, which encapsulates operations such as establishing connections with ODPS, submitting query commands, and retrieving result sets.

[0038] The implementation details of the technical solution of this application are described below: Figure 1 A flowchart illustrating a memory process management method for customs intelligent supervision according to an embodiment of this application is shown. (Refer to...) Figure 1 As shown, the memory process management method applied to intelligent customs supervision includes at least steps S110 to S160, which are described in detail below: S110, Start the main process of the model service and create an independent file system listening thread, which continuously monitors the change events of program files.

[0039] In one embodiment of this application, the model service adopts a multi-process architecture developed using the Python language. Upon system startup, the operating system first loads the executable program of the model service, creating and running the main process. The main process is the control core of the entire model service, responsible for the spawning, scheduling, and management of worker processes, as well as the coordination and allocation of global resources. After the main process starts, it calls the thread creation interface provided by the operating system (such as Python's `threading` module) to create an independent file system monitoring thread. This thread runs in parallel with the main business thread of the main process, without blocking each other, and is specifically responsible for monitoring changes to program files in the project's source code directory.

[0040] In one embodiment of this application, the file system monitoring thread employs a polling mechanism to detect file changes. This thread traverses a specified source code directory at preset time intervals (e.g., every 5 seconds), reading the last modified timestamp (lastModified) of all .py files in the directory, and comparing the currently read timestamp with the previously recorded timestamp. If the last modified timestamp of any file is updated, or a new .py file is added to the directory, or an existing .py file is deleted, a file change event is determined. It should be noted that, to improve monitoring efficiency and avoid unnecessary restarts, the monitoring thread can use regular expression matching rules for filenames or file paths to exclude test files (such as .py files starting with "test_" or containing "_test") from the monitoring scope, monitoring only business logic code files in the production environment. Furthermore, the monitoring thread can also use a notification mechanism based on the operating system's native file system events (such as Linux's inotify or Windows' ReadDirectoryChangesW) instead of polling to achieve more efficient and lower-latency change detection.

[0041] In one embodiment of this application, the main process and the file system monitoring thread interact through an inter-process communication mechanism. Upon detecting a change in a program file, the file system monitoring thread encapsulates the change information (including change type, changed file path, etc.) into a specific message and sends it to the main process through a pre-established pipe or queue between the main process and the monitoring thread. The main process continuously listens for messages on this communication channel, and upon receiving a file change notification, triggers the subsequent worker process restart process. This communication mechanism ensures the real-time transmission of change events, enabling the main process to respond promptly to code updates and achieve hot deployment of services. Simultaneously, to avoid frequent restarts within a short period (e.g., during continuous code editing), the monitoring thread can be configured with debounce logic, sending a restart command only during a stable period (e.g., 2 seconds) after the file change event stops, thereby avoiding unnecessary repeated restart operations.

[0042] In one embodiment, the file system listening thread uses a polling mechanism to monitor program files. Its scanning interval is 5 seconds, and a 5-second debouncing window is provided to merge consecutive file saving actions within a short period of time, avoiding resource waste caused by frequent restarts.

[0043] In another embodiment, the cache hot update monitoring thread scans the disk directories of feature parameter files and model files with a polling period of 2 seconds. The 2-second interval is set according to the daily update frequency of the business files, ensuring the timeliness of hot updates without placing additional burden on the server.

[0044] When a file encounters a deserialization or read error during the refresh process, the system logs the error and retains the old version cache in the global cache object. In subsequent scan cycles, it continues to attempt to reload the file to ensure service stability.

[0045] The above process achieves asynchronous, real-time monitoring of program file change events by creating a file system monitoring thread independent of the main business logic within the main process. Because this monitoring thread runs in parallel with the main business thread, its file monitoring operations do not block or interfere with the main process's management and scheduling of worker processes or its normal response to business requests. Simultaneously, by continuously monitoring program file change events, this step provides accurate event-driven signals for subsequently triggering worker process restarts, enabling the system to automatically detect code updates without manual intervention. This lays the foundation for hot deployment and non-stop upgrades of services.

[0046] S120, the main process spawns at least one worker process, and each worker process executes an initialization routine after being spawned. The initialization routine includes: loading the required feature parameter file and machine learning model file from the disk into the memory space of the worker process, constructing a global cache object, and pre-constructing a grouped index based on the key fields of the feature parameter file.

[0047] In this embodiment, the main process creates and starts at least one worker process by calling the operating system's process spawning interface (such as Python's os.fork() or the Process startup method of the multiprocessing module). After each worker process is successfully spawned by the main process, it does not start processing business requests, but first forcibly executes a predefined initialization routine. The core tasks of this initialization routine include: firstly, submitting query requests to the remote big data storage system (such as ODPS) through the standard file reading API or through the library interface provided by the big data platform (such as pyodps); and loading the required feature parameter files (such as .csv or .pkl files containing risk rules, historical statistical information, etc.) and machine learning model files (such as trained .pkl format serialized files) from the disk or remote storage into the worker process's independent memory space at once. This data is organized into a global cache object (e.g., a singleton cache manager instance). This global cache object resides in memory throughout the entire lifecycle of the worker process, allowing all subsequent customs declaration processing requests to share access. Simultaneously, in this initialization routine, the system also performs grouping operations on the data records in the loaded feature parameter file based on key business fields (such as import / export port codes and trade methods) (e.g., using the Pandas library's `groupby` method). The full set of feature parameters is split into multiple lightweight subsets according to the different values ​​of these key fields, and a corresponding grouping index (e.g., a dictionary-structured key-value mapping) is built in memory for each subset. This allows subsequent real-time matching to quickly locate the corresponding parameter subset based on the specific scenario of the current customs declaration, avoiding traversing the entire dataset for each match. This lays the foundation for zero disk I / O access and efficient data matching in subsequent business processing.

[0048] like Figure 2 As shown, in one embodiment of this application, the required feature parameter file and machine learning model file are loaded from disk into the memory space of the working process, constructed as a global cache object, and a grouping index is pre-built based on the key fields of the feature parameter file, including: S210 submits a structured query request to the big data platform through the library interface provided by the big data platform to obtain the full set of feature parameter files and machine learning model files; S220, convert the feature parameter file into a Pandas DataFrame format dataset in memory space; S230, perform grouping operation on the Pandas DataFrame format dataset according to the preset core business key, group and extract parameter records in the feature parameter file according to import and export port codes and trade methods, generate multiple light quantum sets, and build corresponding grouping indexes for the light quantum sets in memory space; S240, the light quantum set formed after grouping and its corresponding grouping index are stored as a grouping parameter set in the dictionary member of the global cache object.

[0049] In one embodiment of this application, when loading the required feature parameter files and machine learning model files from disk into the memory space of the worker process, constructing them as global cache objects, and pre-building grouping indexes based on the key fields of the feature parameter files, the worker process first establishes a secure connection with the remote big data platform through the library interface provided by the big data platform, submits a structured SQL query request to the platform, and obtains the full content of the feature parameter files and the serialized machine learning model files in batches. After receiving the data returned by the platform, the worker process directly converts and loads the data records in the feature parameter files into a Pandas DataFrame format dataset in the memory space. This format provides an efficient data operation interface and vectorized computing capabilities, facilitating subsequent fast filtering, grouping, and merging operations. As for the machine learning model files, they are converted from byte streams into executable model objects that can be directly called in memory through deserialization operations (such as pickle.loads). The entire process does not involve writing to or reading from the local disk, avoiding additional I / O overhead.

[0050] Specifically, the data records in the obtained feature parameter files are directly converted into PandasDataFrame format datasets in memory space. DataFrame is the core two-dimensional tabular data structure in the Pandas library. It organizes data in the form of rows and columns, and provides efficient interfaces for data filtering, grouping, merging, and vectorized operations. It can make full use of the SIMD instruction set of modern CPUs to perform batch data parallel processing, and has an order-of-magnitude advantage in performance compared to the native Python list or dictionary structure.

[0051] Furthermore, after the feature parameter data is successfully loaded into a DataFrame format dataset, the workflow performs grouping operations on the dataset according to preset core business keys (including but not limited to import / export port codes and trade methods). Specifically, by calling the Pandas library's groupby method, the full set of parameter records is grouped and selected according to the combined values ​​of import / export port codes and trade methods, splitting the originally single large-scale parameter table into multiple independent lightweight quantum sets. Each lightweight quantum set corresponds to a specific combination scenario of port and trade methods. At the same time, a corresponding grouping index is built in memory for each lightweight quantum set (e.g., a dictionary structure with the combined values ​​of port codes and trade methods as keys and the corresponding lightweight quantum set data as values). This allows subsequent real-time matching to directly locate the corresponding parameter subset by key-value lookup based on the actual scenario of the current customs declaration, without traversing the full data. After the grouping operation is completed, the workflow stores all lightweight quantum sets and their corresponding grouping indexes as a complete grouping parameter set in the dictionary member of the global cache object. This enables the required feature parameter data to be quickly obtained in O(1) time complexity during subsequent business request processing, significantly improving data matching efficiency.

[0052] In this embodiment, the dictionary members are key-value data structures built into Python based on hash tables. They directly calculate the storage location using the hash value of the key, and can complete the insertion, search and deletion operations of data in O(1) time complexity. They are particularly suitable as a cache storage structure for high-frequency access.

[0053] In one embodiment of this application, loading a machine learning model file from disk into the memory space of the worker process includes: Send a model file read request to the resource storage of the big data platform to obtain the serialized model file byte stream; The model file byte stream is converted into an executable model object in memory through a deserialization operation; The executable model object is stored in a dictionary member of the global cache object.

[0054] In one embodiment of this application, when loading the machine learning model file from disk into the memory space of the worker process, the worker process first sends a model file read request to the resource storage of the big data platform (such as MaxCompute) through the library interface (such as pyodps) provided by the big data platform, specifying the name and storage path of the model file to be obtained, in order to obtain the serialized byte stream of the model file; the model file is usually a .pkl format file, that is, the trained machine learning model object is converted into a byte stream form that can be stored or transmitted through Python's pickle serialization tool; after receiving the model file byte stream returned by the platform, the worker process directly calls... The deserialization method (such as pickle.loads) restores the byte stream into an executable model object in memory. The entire process is completed in memory without any local disk write or read operations, thus avoiding disk I / O overhead. Finally, the worker process stores the executable model object as key-value pairs in the dictionary member of the global cache object (e.g., using "model" or "cb_model" as the key and the model object instance as the value). In subsequent risk identification calculations, each business request can directly retrieve and call the prediction method of the model object from memory through the dictionary key name without having to repeatedly load the model file, thus realizing global sharing and efficient reuse of model resources.

[0055] Optionally, when loading parameter files and model files into memory, in addition to storing them directly in the global cache object within the process, the data can also be read into an external distributed cache system (such as Redis) depending on the actual deployment architecture to support cache sharing across processes or nodes; in the preferred embodiment of this application, for the sake of minimum latency, the data is stored directly in the global cache object in the local memory of each working process.

[0056] Optionally, the model file is not limited to the .pkl format, but can also be the .model format or other serialized format files; the feature parameter file is not limited to the .csv format, but can also be a columnar storage format file such as .parquet or .orc.

[0057] In one embodiment, the global cache object is implemented using a module-level singleton and a global dictionary. Specifically, the global cache object includes at least a model cache dictionary and a parameter cache dictionary. The model cache dictionary uses the absolute path of the feature parameter file or model file as the key and "deserialized model object or DataFrame format data + the file's last modification timestamp" as the value. The parameter cache dictionary uses the absolute path as the key and "DataFrame or Polars object + the file's last modification timestamp" as the value. The worker process also maintains a set of loaded filenames in memory for quickly identifying file additions, deletions, and modifications during hot updates. When a single file loading fails, the global cache object retains a valid object from the previous version of that file to avoid affecting ongoing business requests.

[0058] In one specific embodiment, the offline defragmentation phase of the pre-built grouped index includes: First, obtain the full feature parameter table and standardize it, unifying field names, data types, and null value rules; Secondly, construct composite business keys based on preset core business keys, such as import and export port codes, trade methods, and tariff codes; Then, the groupby operation is performed on the standardized full parameter table to split the full data into multiple lightweight quantum sets corresponding to specific business scenarios; For high-density groups with still large data volumes, further split them into sub-shard files such as bgdqz_d07004_cs_01.pkl and bgdqz_d07004_cs_02.pkl according to hash buckets or secondary business keys.

[0059] The grouped index is presented in memory as a two-layer structure of "grouped index dictionary + sub-sharded DataFrame": the first layer is used to quickly locate the target subset based on the business key, and the second layer is used to perform local matching within the subset.

[0060] The above process, through the spawning of worker processes from the main process and the execution of initialization routines after spawning, achieves the pre-binding and readiness of computing and data resources. Specifically, feature parameter files and model files are loaded from disk into the memory space of worker processes and constructed as global cache objects. This ensures that each worker process has all the necessary computing resources fully residing in its memory before it begins receiving and processing business requests. Subsequent processing of any business request does not require reading any parameters or model files from disk, fundamentally eliminating the most significant time-consuming node: frequent disk I / O operations. Simultaneously, a grouping index is pre-built based on the key fields of the feature parameter files, advancing the traversal and grouping operations that would normally be performed in real-time with each request to the initialization phase. This allows the data matching operation for each request after initialization to be reduced from a full traversal to an index lookup, significantly reducing the computational overhead of a single request.

[0061] S130, when the file system listening thread detects that the program file has been changed, it sends a restart command to the main process. The main process responds to the restart command and restarts all working processes. The restarted working processes re-execute the initialization routine to achieve a full cache refresh after the program is updated.

[0062] In this embodiment, after detecting a change in a program file, the file system listening thread does not directly perform any cache refresh or process management operations. Instead, it encapsulates the change information into a restart instruction and sends it to the main process via inter-process communication. The main process is then responsible for the lifecycle management of the worker processes. When the main process receives the restart instruction from the file system listening thread through the communication channel, it first marks the current state as "restarting." In this mode, the main process suspends receiving new business requests or temporarily suspends new requests to ensure that no requests are assigned to worker processes that are terminating during the restart process. Subsequently, the main process sends termination signals (such as SIGTERM) to all worker processes under its management one by one, waiting for each worker process to complete the currently processed business request and automatically exit. For worker processes that do not respond to the termination signal within a specified time, the main process will forcibly terminate their execution to avoid process freezes and resource leaks.

[0063] In one embodiment of this application, after all worker processes have successfully terminated, the main process calls the process spawning interface again to create the same number of worker processes as before the restart. Each newly spawned worker process, upon startup, no longer uses any old memory state but executes the exact same initialization routine as during system initialization. This initialization routine includes: reloading the latest feature parameter files and machine learning model files from disk or a remote big data platform into the worker process's memory space, constructing a global cache object, and rebuilding the grouped index based on the key fields of the latest feature parameter files. Since this process is forcibly executed after the worker processes are spawned, any old version data loaded into memory before the program files were changed is completely discarded, replaced by entirely new cached content built based on the latest code logic and data files. This full cache refresh mechanism ensures that all worker processes run on a unified, latest version state after the code update, avoiding inconsistencies in business logic caused by some processes not being updated in time.

[0064] In one embodiment of this application, the main process, when restarting all worker processes, does not simply terminate and then restart all worker processes simultaneously. Instead, it adopts a rolling restart approach, terminating and restarting only one worker process at a time. Once that worker process completes its initialization routine and enters a normal service state, it then proceeds to process the next worker process. During this rolling restart, the main process assigns newly arriving business requests only to worker processes that have already restarted, while worker processes that have not yet restarted continue processing existing requests. This rolling restart strategy avoids all worker processes being unavailable simultaneously, effectively ensuring the continuous availability of the model service during code updates and achieving smooth upgrades under a high-availability architecture. After all worker processes have completed restarting and successfully executed their initialization routines, the main process exits the "restarting" mode and resumes normal request scheduling, thus completing the full cache refresh process after the program update.

[0065] In the aforementioned communication mechanism, the file system listening thread and the main process can interact in any of the following ways: asynchronous notification based on operating system signals (such as sending a SIGHUP signal), streaming data transmission based on pipes, or message passing based on queues. In the preferred embodiment of this application, the SIGHUP signal method is used to achieve fast notification of change events, which does not require the establishment of an additional communication channel, is simple to implement, and has the lowest latency.

[0066] The above process, through the signal communication mechanism between the file system listening thread and the main process, triggers a full restart of the worker processes when program files are modified. This forces all worker processes to re-execute the initialization routine after the code update, reload the latest feature parameter files and model files from disk, and rebuild the global cache objects and grouped indexes. Since the restarted worker processes completely discard the old version's memory state and rebuild the cache based on the latest code and data, this step effectively avoids code version inconsistencies and data inconsistencies caused by some worker processes not updating in time. It ensures that all worker processes run on a unified, up-to-date state after the update, providing a guarantee for the accuracy and consistency of risk identification calculations.

[0067] S140, In each of the working processes, a cache hot update monitoring thread is established independently of the file system monitoring thread. The cache hot update monitoring thread periodically scans the disk directory of the feature parameter file and the model file in a polling manner to monitor the file change status under the disk directory.

[0068] In this embodiment, after each worker process completes its initialization routine and enters normal service mode, it also establishes an additional cache hot update monitoring thread within its process, independent of the file system monitoring thread created by the main process. This cache hot update monitoring thread and the file system monitoring thread are isolated from each other and do not interfere with each other. The former is specifically responsible for monitoring the change status of the disk directory where the business data files (i.e., feature parameter files and machine learning model files) are located, while the latter is responsible for monitoring the changes of the source code program files. The cache hot update monitoring thread uses a preset time interval (e.g., every 30 seconds) as its polling cycle. In each polling cycle, it actively scans the file list under the specified disk directory, reads the last modified timestamp (lastModified) of each file or calculates the MD5 checksum of the file, and compares the read timestamp or MD5 value with the cache loading timestamp or historical MD5 value recorded inside the worker process to determine whether the files in the directory have undergone changes such as addition, modification or deletion, thereby realizing continuous monitoring of the data file change status.

[0069] In one embodiment of this application, the cache hot update monitoring thread periodically scans the disk directory of the feature parameter file and the model file in a polling manner to monitor the file change status in the disk directory, including: The cache hot update monitoring thread is started independently within the working process, with a preset time interval as the polling period; In each polling cycle, read the last modification timestamp or MD5 value of a specific pattern file in the disk directory; The last modification timestamp or MD5 value read is compared with the load timestamp or MD5 value recorded in the global cache object to determine whether the files in the disk directory have been added, modified, or deleted.

[0070] In one embodiment of this application, the cache hot update monitoring thread starts independently within the worker process and uses a preset time interval (e.g., 30 seconds) as the polling cycle. This time interval can be flexibly configured according to the business's requirements for data real-time performance. For example, for scenarios with high frequency of risk rule updates, it can be shortened to 10 seconds to improve response speed, while for scenarios with relatively stable rules, it can be extended to 60 seconds to reduce unnecessary system overhead. At the end of each polling cycle, the cache hot update monitoring thread first obtains a list of files in a specified disk directory, and filters out specific pattern files to be monitored according to a preset filename matching pattern (such as regular expressions like label_*.pkl, cb_model_*.pkl, etc.). Then, it reads the last modified timestamp (lastModified) of these files one by one or calculates the MD5 value of the files using a hash algorithm as the basis for determining whether the files have been changed.

[0071] Furthermore, the cache hot update monitoring thread compares the last modification timestamp or MD5 value of each file currently read with the corresponding load timestamp or MD5 value calculated during loading in the global cache object. If a file's last modification timestamp is later than the load timestamp recorded in the global cache object, or if the calculated MD5 value is inconsistent with the MD5 value recorded during loading, the file is determined to have been modified. If a file in the disk directory is not found in the global cache object's record list, it is determined to be a newly added file. If a file recorded in the global cache object is no longer in the current disk directory, it is determined to have been deleted. Through this comparison logic, the cache hot update monitoring thread can accurately identify the addition, modification, or deletion status of files in the disk directory within each polling cycle, thus providing accurate change information for subsequent targeted cache refresh operations.

[0072] The above process decouples code file monitoring from data file monitoring by establishing a dedicated data file hot update monitoring thread within each worker process, separate from the file system monitoring thread created by the main process. Since the cache hot update monitoring thread and the file system monitoring thread run independently and do not interfere with each other—the former focusing on monitoring changes to feature parameters and model data files, and the latter focusing on monitoring changes to source code files—there is no mutual blocking or competition between them. This avoids the complexity and performance bottlenecks that occur when a single monitoring thread needs to handle multiple change logics simultaneously. Furthermore, each worker process has its own independent hot update monitoring thread, allowing each process to independently detect data file changes and autonomously determine the refresh timing without cross-process coordination and synchronization, thus improving system response speed.

[0073] S150, when the cache hot update monitoring thread detects that the files in the disk directory have changed, it triggers a cache refresh, reloads the changed files and updates the corresponding cache area in the global cache object, or cleans up the cache entries corresponding to the deleted files from the global cache object.

[0074] In this embodiment, when the cache hot update monitoring thread detects changes such as additions, modifications, or deletions to feature parameter files or machine learning model files in the disk directory by comparing the last modification timestamp or MD5 value, the thread does not simply clear the entire global cache object and reload all data. Instead, it automatically triggers a targeted incremental cache refresh mechanism. Specifically, for newly added or modified files, the cache hot update monitoring thread calls an independent cache loading module to reread and load only the changed single file from the disk into memory. Then, it precisely replaces the cache area corresponding to that file in the global cache object with the newly loaded data (e.g., updating the value corresponding to a specific key in a dictionary member). The cache areas corresponding to other unchanged files in the global cache object remain unchanged and are completely unaffected. For deleted files, the cache hot update monitoring thread searches for and cleans up the cache entries corresponding to the deleted file in the global cache object (e.g., deleting the corresponding key-value pairs in a dictionary member). This ensures that the data in the memory cache is always strictly consistent with the actual file state in the disk directory, achieving hot updates without interrupting business operations.

[0075] In one embodiment of this application, when the cache hot update monitoring thread detects a change in a file under the disk directory, it triggers a cache refresh, reloading the changed file and updating the corresponding cache area in the global cache object, or clearing the cache entries corresponding to deleted files from the global cache object, including: When the cache hot update monitoring thread detects that a new file has been added or the last modification timestamp of an existing file has changed in the disk directory, a cache refresh is triggered. The modified files are reloaded from disk into memory, and the corresponding cache area in the global cache object is updated. When the cache hot update monitoring thread determines that a file in the disk directory has been deleted, it triggers the cache refresh to clean up the cache entry corresponding to the deleted file from the global cache object.

[0076] In one embodiment of this application, when the cache hot update monitoring thread determines that a new file exists in the disk directory or that the last modification timestamp of an existing file has changed by comparing the last modification timestamp or MD5 value, the thread triggers an incremental cache refresh process for that file. Specifically, the cache hot update monitoring thread first calls an independent cache loading module based on the complete path and filename of the changed file, and re-reads the file from the disk or remote storage through a standard file reading API or a library interface provided by a big data platform. For feature parameter files, the read data is converted into a Pandas DataFrame format dataset in memory, and grouped according to its corresponding business key (such as import / export port codes and trade methods) to generate corresponding lightweight quantum sets and grouping indexes. For model files, the byte stream is restored to an executable model object in memory through deserialization. After loading, the system uses the newly loaded data to precisely replace the cache area corresponding to that file in the global cache object (i.e., update the value pointed to by the corresponding key in the dictionary member), while the cache areas corresponding to other unchanged files remain unchanged. Only the single file that has changed is reloaded and updated, thus achieving synchronization between the memory cache and the disk file.

[0077] Furthermore, when the cache hot update listener thread scans the disk directory during its polling cycle and finds that a file previously recorded in the global cache object no longer exists in the current disk directory, it determines that the file has been deleted. At this point, the cache hot update listener thread also triggers the cache refresh mechanism. However, unlike the processing logic for file additions or modifications, for deleted files, the system does not perform a loading operation. Instead, it directly searches for and cleans up the cache entry corresponding to the deleted file in the global cache object. That is, it removes the corresponding key-value pair from the dictionary members of the global cache object and releases the memory resources occupied by the cache entry. If the deleted file is the sole source of a lightweight quantum set, the system will also simultaneously clean up the corresponding group index entries to ensure that subsequent matching logic does not reference expired data. Through these two complementary processing mechanisms—loading and updating during additions / modifications and cleaning up and releasing during deletions—the cache hot update listener thread can maintain the content of the global cache object incrementally, ensuring data consistency while minimizing resource overhead and impact on business processing performance during cache refresh.

[0078] In one embodiment, the worker process provides a cache query interface. When this interface is invoked by a business request, its execution flow includes: First, based on the file identifier in the request, check the last modification timestamp of the corresponding file under the source file path; Then, the last modification timestamp read is compared with the cache load timestamp recorded in the global cache object; If it is determined that the source file has been modified after the cache has been loaded, then the synchronous reload operation of the cache item is triggered, that is, the source file is reloaded from the disk and the global cache object is updated; Finally, after completing the synchronous reload, the updated cache data is returned through the cache query interface for subsequent risk identification calculations.

[0079] The above process, upon detecting a data file change, does not clear the entire cache and reload all data. Instead, it reloads only the single file that has changed, precisely replacing the corresponding cache area in the global cache object, or clearing the cache entry corresponding to the deleted file. Because the incremental update strategy avoids the large-scale memory allocation, data copying, and garbage collection overhead of a full refresh, this step minimizes the time overhead and CPU resource consumption of cache refresh operations while ensuring consistency between the memory cache and disk file data. Furthermore, since the worker process can still handle business requests normally during hot updates (the old version data remains in the unupdated cache area), this step achieves parameter hot updates without business interruption, applying the latest rules without stopping the service.

[0080] S160, after receiving the customs declaration data request, the working process accesses the memory data in the global cache object, completes data matching based on the key fields of the current customs declaration and the grouping index, and calls the corresponding executable model object from the model file to perform risk identification calculation and returns the calculation result.

[0081] In this embodiment, when the worker process receives a single customs declaration data request submitted by the customs declaration business system through the RESTful API interface, the worker process does not trigger any disk I / O operations to read the feature parameter file or model file. Instead, it directly accesses the global cache object already residing in its memory space to obtain the pre-loaded feature parameter data and executable model object. Specifically, the worker process first extracts key fields (such as import / export method, port code, and tariff number) from the current customs declaration data, and then searches in the group index of the global cache object based on the values ​​of these key fields. Since the group index is a dictionary structure pre-built based on core business keys such as import / export port codes and trade methods, the system can... The process quickly locates the light quantum set corresponding to the customs declaration scenario with a time complexity of O(1). Then, the merge vectorization calculation method of the Pandas library is used to quickly associate and match the current customs declaration data with the located light quantum set to complete the matching calculation of feature parameters. After obtaining the matching result, the worker process directly calls the prediction method of the corresponding executable model object from the dictionary members of the global cache object, combines the matching result features with the original customs declaration features to construct a feature vector that meets the model input requirements, and passes it into the model object to perform risk identification calculation. Finally, the risk score or risk level of the customs declaration is obtained, and the calculation result is returned to the customs declaration business system through the RESTful API interface. In the entire process, the worker process is completely based on memory data for calculation, without reading any files from the disk, realizing efficient risk identification processing with zero disk I / O.

[0082] In one embodiment of this application, accessing the memory data in the global cache object and performing data matching based on the key fields of the current customs declaration combined with the grouping index includes: Access the memory data in the global cache object to obtain the key fields of the current customs declaration, including import / export method, port code and HS code; Based on the import / export method, port code, and HS code of the current customs declaration, locate the pre-built light quantum set corresponding to the scenario of the customs declaration from the global cache object; The merge vectorization computation method of the Pandas library is used to associate and match the current customs declaration data with the located light quantum set.

[0083] In one embodiment of this application, after receiving a customs declaration data request, the worker process first parses the request message and extracts the key fields of the current customs declaration. The key fields include, but are not limited to, import / export methods (such as import, export, transit, etc.), port codes (i.e., the port codes of the declared goods that actually enter or leave the country), and tariff codes (i.e., the classification codes of commodities in the customs tariff, such as the first few digits of the HS code). The worker process directly obtains the values ​​of these fields from the memory data of the global cache object without any disk access. Then, it uses the combined values ​​of these key fields as the query key to search in the pre-built grouped index (i.e., dictionary structure) in the global cache object. Since the keys of the grouped index are organized according to the combination of core business keys such as import / export methods and port codes, the system can quickly locate the lightweight quantum set corresponding to the customs declaration scenario through the dictionary key-value lookup operation.

[0084] Furthermore, after locating the corresponding lightweight quantum set, the workflow employs the merge vectorized computation method provided by the Pandas library. The core data of the current customs declaration (such as the declaration item number, commodity code, declared quantity, and declared unit price) is used as the left table, and the parameter records in the located lightweight quantum set are used as the right table. The associated key field (such as tax number or commodity code) is specified, and a one-time inner join or left join operation is performed to complete the association and matching between the customs declaration data and the feature parameters. Vectorized computation is one of the core performance advantages of the Pandas library. Its underlying architecture is based on NumPy array operations, which can transform the inefficient operation of iterating row by row and matching record by record using multiple for loops into batch parallel computation of the entire data column. This fully utilizes the SIMD (Single Instruction Multiple Data) instruction set of modern CPUs for acceleration, increasing the matching speed by tens or even hundreds of times in matching scenarios with millions of data points. By combining the rapid location based on grouped indexes with the association matching based on vectorized calculations, the system significantly reduces the matching time for a single customs declaration while ensuring matching accuracy, providing efficient data support for subsequent risk identification calculations.

[0085] In one embodiment, after the merge vectorization calculation completes the association matching, the system performs a one-time summary and statistical analysis of the hit results using the groupby and agg methods. The statistical results include at least: the number of hit rules, the number of unique rules, the hit ratio, the mean / maximum / minimum values ​​of the numerical fields, the 0 / 1 count of the flag fields, and the default compensation flag. Subsequently, the above statistical results are concatenated with the original customs declaration features and the label features returned from the pylabel_main module, aligned according to the preset online_desired_order field order, to form the standardized input feature vector required by the CatBoost model. Finally, runtime information such as score, status, and total_seconds are appended to the end of the feature vector to form a complete data structure for database storage and data return.

[0086] In one embodiment of this application, before performing risk identification calculation after receiving the customs declaration data request, the workflow further includes: When the cache query interface provided to the outside world is called by a business request, it checks the last modification timestamp of the corresponding source file under the source file path before returning data; The last modification timestamp of the detected source file is compared with the cache loading timestamp recorded in the global cache object. If it is determined that the source file has been modified after the cache is loaded, the synchronous reload operation of the cache item is triggered, the source file is reloaded from the disk and the global cache object is updated. After the synchronization reload operation is completed, the updated cached data is returned through the cache query interface for the risk identification calculation.

[0087] In one embodiment of this application, after receiving a customs declaration data request and before formally executing the risk identification calculation, the workflow performs an additional data consistency check through the externally provided cache query interface to ensure that the cached data used in this calculation is always consistent with the disk source file. Specifically, when a business request calls the cache query interface to try to obtain the required feature parameter data, the cache query interface does not directly return the data already stored in the global cache object. Instead, it first accesses the corresponding source file in the disk directory and reads its last modification timestamp based on the file identifier or path information specified in the request. Then, it reads the last modification timestamp of the source file. The timestamp is compared with the loading timestamp of the cached item recorded in the global cache object. If the comparison result indicates that the last modification timestamp of the source file is later than the cache loading timestamp, it means that the source file has been modified by an external program or manual operation after the cache is loaded. At this time, the cache query interface triggers a synchronous reload operation for the cached item, that is, rereads the complete content of the source file from the disk and updates the corresponding cache area in the global cache object. After the reload operation is successfully completed, the updated cache data is returned to the business requester through the cache query interface for risk identification calculation. If the comparison result indicates that the source file has not been modified, the existing cache data in the global cache object is returned directly. This synchronous reload mechanism, as an important supplement to the cache hot update listening thread polling refresh mechanism, can realize on-demand synchronous updates when file changes occur during the polling interval of the listening thread (i.e., between two polls) through real-time checks triggered by business requests. This further narrows the time window of data inconsistency and effectively avoids the problem of temporary data inconsistency caused by polling delay, providing double protection for the accuracy of risk identification calculation.

[0088] The above process, based on the pre-loaded global cache object and the pre-built grouped index, ensures that during the processing of each business request, the access paths of the worker process to feature parameters and model files are all pure memory operations, completely bypassing the disk file system, and therefore generating no disk I / O latency. The introduction of the grouped index reduces the operational complexity of the data matching stage from O(n) of full traversal to O(1) of index lookup, and the call to the model object is a method call in memory rather than an initialization call after disk loading. This step significantly shortens the total time from matching to completion of calculation for a single ticket, achieving an order-of-magnitude improvement in risk identification performance.

[0089] It should be noted that the technical solution of this application is not a simple accumulation of technologies, but an application for a specific business scenario that extracts and solves the core bottlenecks. Through memory preloading and listening refresh mechanism, the regulatory process is compressed and reconstructed, which promotes the transformation of customs supervision from relying on manpower and experience to relying on data-driven and intelligent algorithms. That is, the application of new technologies brings new elements for efficient data utilization and new facilities based on high-performance model services, ultimately achieving a qualitative change in the regulatory system.

[0090] This technical solution derives worker processes from the main process and executes initialization routines after deriving. It preloads the feature parameter files and model files for customs declaration risk identification into the memory space of each worker process and constructs a global cache object. Simultaneously, it pre-builds grouped indexes based on core customs business keys such as import / export port codes and trade methods, ensuring that the processing path for each customs declaration data request is entirely in-memory, eliminating frequent disk I / O as the main source of time consumption. A file system monitoring thread monitors program file changes and triggers a full restart of worker processes, ensuring consistent cache status across worker processes after rule code updates. An independent cache hot update monitoring thread is established within each worker process to perform incremental loading or cleanup of individual changed files, ensuring uninterrupted business operations during risk control parameter updates. The pre-built grouped indexes, combined with vectorized calculation methods, reduce the matching complexity between each customs declaration and feature parameters from a full traversal to an index lookup. The cache query interface checks the source file modification time and triggers synchronous reload before returning data, further narrowing the data inconsistency time window, enabling the model service to balance processing efficiency, data consistency, and service continuity when processing customs declaration data.

[0091] In this embodiment, the test environment consisted of 36 Linux servers, each with an 8-core CPU and 32GB of memory, deploying Flask + Gunicorn services, with 8 workers configured per machine. During service startup, the model and parameter files were pre-loaded uniformly, with an average memory usage of approximately 2GB per worker. The statistical criteria used were continuous load test results under the same business message sample and deployment strategy, with the average of multiple repeated tests used as the comparison value.

[0092] Specifically, Figure 3 (Corresponding to Table 1) and Figure 4 (Table 2) shows the response latency changes under different data sizes and concurrency levels. It can be seen that when the parameter file is expanded from 100MB to 600MB, the average response time before optimization increases significantly, while after optimization it remains relatively stable at around 1 second. Furthermore, when concurrent requests increase, the increase in latency after optimization is significantly slower, indicating that cache hits and vectorized matching effectively alleviate I / O and loop computation pressure.

[0093] Table 1 Comparison of average response latency under different data sizes

[0095] Table 2 Comparison of average response latency under different concurrent requests

[0097] In hot update scenarios, when the parameter file changes, the service can be refreshed without restarting, and business requests will only experience slight jitter. Figure 5 (Corresponding to Table 3) shows that the latency near the update event only rises briefly before quickly falling back; Figure 6 The results showed that after cache preloading, CPU utilization increased from about 31% to about 60%, disk I / O decreased significantly, and memory usage increased only slightly, indicating that the performance improvement mainly came from compute path optimization rather than resource stacking.

[0098] Table 3 Monitoring Results of Hot Updates and Resource Usage

[0100] The following describes embodiments of the memory process management device for customs intelligent supervision according to this application, which can be used to execute the memory process management method for customs intelligent supervision described in the above embodiments of this application. It is understood that the memory process management device for customs intelligent supervision can be a computer program (including program code) running on a computer device. For example, the memory process management device for customs intelligent supervision can install customs management application software to realize cloud computing of the big data generated during the customs declaration process through its platform; the memory process management device for customs intelligent supervision can be used to execute the corresponding steps in the method provided in the embodiments of this application. For details not disclosed in the embodiments of the memory process management device for customs intelligent supervision of this application, please refer to the embodiments of the memory process management method for customs intelligent supervision described above in this application.

[0101] Figure 7 A block diagram of a memory process management device for smart customs supervision according to an embodiment of this application is shown.

[0102] Reference Figure 7 As shown, a memory process management device for customs intelligent supervision according to an embodiment of this application includes: Module 310 is created to start the main process of the model service and to create an independent file system listening thread, which continuously monitors the change events of program files. An initialization module 320 is used to derive at least one worker process from the main process. Each worker process executes an initialization routine after being derived. The initialization routine includes: loading the required feature parameter file and machine learning model file from the disk into the memory space of the worker process, constructing a global cache object, and pre-constructing a grouped index based on the key fields of the feature parameter file. The refresh module 330 is used to send a restart command to the main process when the file system listening thread detects that the program file has been changed. The main process responds to the restart command and restarts all working processes. The restarted working processes re-execute the initialization routine to achieve a full cache refresh after the program is updated. The monitoring module 340 is used to establish a cache hot update monitoring thread independently of the file system monitoring thread in each working process. The cache hot update monitoring thread periodically scans the disk directory of the feature parameter file and the model file in a polling manner to monitor the file change status under the disk directory. The cache module 350 is used to trigger a cache refresh when the cache hot update monitoring thread detects that the files under the disk directory have changed, to reload the changed files and update the corresponding cache area in the global cache object, or to clean up the cache entries corresponding to the deleted files from the global cache object. The identification module 360 ​​is used to access the memory data in the global cache object after the working process receives the customs declaration data request, complete the data matching based on the key fields of the current customs declaration and the group index, and call the corresponding executable model object from the model file to perform risk identification calculation and return the calculation result.

[0103] In this application, based on the aforementioned scheme, the step of loading the required feature parameter files and machine learning model files from the disk into the memory space of the working process, constructing a global cache object, and pre-constructing a grouping index based on the key fields of the feature parameter files includes: submitting a structured query request to the big data platform through the library interface provided by the big data platform to obtain the full set of feature parameter files and machine learning model files; converting the feature parameter files into a Pandas DataFrame format dataset in the memory space; performing a grouping operation on the Pandas DataFrame format dataset according to the preset core business key, grouping and extracting the parameter records in the feature parameter files according to the import / export port code and trade method to generate multiple lightweight quantum sets, and constructing corresponding grouping indexes for the lightweight quantum sets in the memory space; storing the lightweight quantum sets formed after grouping and their corresponding grouping indexes as grouping parameter sets in the dictionary members of the global cache object.

[0104] In this application, based on the aforementioned scheme, loading the machine learning model file from the disk into the memory space of the working process includes: sending a model file read request to the resource storage of the big data platform to obtain a serialized model file byte stream; converting the model file byte stream into an executable model object in the memory space through a deserialization operation; and storing the executable model object in the dictionary member of the global cache object.

[0105] In this application, based on the aforementioned scheme, the cache hot update monitoring thread periodically scans the disk directory of the feature parameter file and the model file in a polling manner to monitor the file change status under the disk directory, including: the cache hot update monitoring thread is started independently within the working process, with a preset time interval as the polling cycle; in each polling cycle, the last modification timestamp or MD5 value of a specific pattern file under the disk directory is read; the read last modification timestamp or MD5 value is compared with the loading timestamp or MD5 value recorded in the global cache object to determine whether the file under the disk directory has been added, modified, or deleted.

[0106] In this application, based on the aforementioned scheme, the step of triggering a cache refresh when the cache hot update monitoring thread detects a change in a file under the disk directory, reloading the changed file and updating the corresponding cache area in the global cache object, or clearing the cache entries corresponding to deleted files from the global cache object, includes: triggering a cache refresh when the cache hot update monitoring thread detects a new file or a change in the last modification timestamp of an existing file under the disk directory; reloading the changed file from the disk into memory and updating the corresponding cache area in the global cache object; and triggering the cache refresh when the cache hot update monitoring thread determines that a file under the disk directory has been deleted, clearing the cache entries corresponding to the deleted file from the global cache object.

[0107] In this application, based on the aforementioned scheme, the step of accessing the memory data in the global cache object and completing data matching based on the key fields of the current customs declaration combined with the grouping index includes: accessing the memory data in the global cache object and obtaining the key fields of the current customs declaration, the key fields including import / export method, port code, and tax number; locating a pre-constructed lightweight quantum set corresponding to the scenario of the customs declaration from the global cache object based on the obtained import / export method, port code, and tax number of the current customs declaration; and using the merge vectorization calculation method of the Pandas library to associate and match the current customs declaration data with the located lightweight quantum set.

[0108] In this application, based on the aforementioned scheme, before performing risk identification calculation after receiving the customs declaration data request, the workflow further includes: when the externally provided cache query interface is called by a business request, checking the last modification timestamp of the corresponding source file under the source file path before returning data; comparing the checked last modification timestamp of the source file with the cache loading timestamp recorded in the global cache object; if it is determined that the source file has been modified after the cache loading, triggering the synchronous reload operation of the cache item, reloading the source file from the disk and updating the global cache object; after completing the synchronous reload operation, returning the updated cache data through the cache query interface for the risk identification calculation.

[0109] This technical solution derives worker processes from the main process and executes initialization routines after deriving. It preloads the feature parameter files and model files for customs declaration risk identification into the memory space of each worker process and constructs a global cache object. Simultaneously, it pre-builds grouped indexes based on core customs business keys such as import / export port codes and trade methods, ensuring that the processing path for each customs declaration data request is entirely in-memory, eliminating frequent disk I / O as the main source of time consumption. A file system monitoring thread monitors program file changes and triggers a full restart of worker processes, ensuring consistent cache status across worker processes after rule code updates. An independent cache hot update monitoring thread is established within each worker process to perform incremental loading or cleanup of individual changed files, ensuring uninterrupted business operations during risk control parameter updates. The pre-built grouped indexes, combined with vectorized calculation methods, reduce the matching complexity between each customs declaration and feature parameters from a full traversal to an index lookup. The cache query interface checks the source file modification time and triggers synchronous reload before returning data, further narrowing the data inconsistency time window, enabling the model service to balance processing efficiency, data consistency, and service continuity when processing customs declaration data.

[0110] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0111] It should be noted that the computer system of the electronic device in this embodiment is only an example and should not impose any limitations on the function and scope of use of the embodiments of this application.

[0112] In this embodiment, the computer system includes a central processing unit 401, which can perform various appropriate actions and processes based on programs stored in read-only memory 402 or programs loaded from storage section 408 into random access memory 403, such as executing the memory process management method for customs intelligent supervision described in the above embodiment. The random access memory 403 also stores various programs and data required for system operation, thereby realizing big data storage and big data management. The central processing unit 401, read-only memory 402, and random access memory 403 are interconnected via bus 404. Input / output interface 405 is also connected to bus 404.

[0113] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0114] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit 401, it performs various functions defined in the system of this application.

[0115] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0118] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0119] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the memory process management method for customs intelligent supervision described in the above embodiments.

[0120] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0121] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0122] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0123] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A memory process management method applied to intelligent customs supervision, characterized in that, include: Start the main process of the model service and create an independent file system listening thread, which continuously monitors the change events of program files. The main process spawns at least one worker process. Each worker process executes an initialization routine after being spawned. The initialization routine includes: loading the required feature parameter file and machine learning model file from the disk into the memory space of the worker process, constructing a global cache object, and pre-constructing a grouped index based on the key fields of the feature parameter file. When the file system listening thread detects that the program file has been changed, it sends a restart command to the main process. The main process responds to the restart command and restarts all worker processes. The restarted worker processes re-execute the initialization routine to achieve a full cache refresh after the program is updated. In each of the aforementioned working processes, a cache hot update monitoring thread is established independently of the file system monitoring thread. The cache hot update monitoring thread periodically scans the disk directory of the feature parameter file and the model file in a polling manner to monitor the file change status under the disk directory. When the cache hot update monitoring thread detects that a file in the disk directory has changed, it triggers a cache refresh, reloads the changed file and updates the corresponding cache area in the global cache object, or cleans up the cache entries corresponding to deleted files from the global cache object. After receiving a customs declaration data request, the process accesses the memory data in the global cache object, performs data matching based on the key fields of the current customs declaration and the grouping index, and calls the corresponding executable model object from the model file to perform risk identification calculation and returns the calculation result.

2. The memory process management method applied to intelligent customs supervision according to claim 1, characterized in that, The required feature parameter files and machine learning model files are loaded from disk into the memory space of the worker process, constructed as a global cache object, and a grouped index is pre-built based on the key fields of the feature parameter files, including: By submitting a structured query request to the big data platform through the library interface provided by the platform, you can obtain the full set of feature parameter files and machine learning model files. Convert the feature parameter file into a Pandas DataFrame format dataset in memory space; The Pandas DataFrame format dataset is grouped according to the preset core business key. The parameter records in the feature parameter file are grouped and extracted according to the import and export port codes and trade methods to generate multiple light quantum sets. A corresponding grouping index is built for the light quantum sets in the memory space. The light quantum set formed after grouping and its corresponding grouping index are stored as a grouping parameter set in the dictionary member of the global cache object.

3. The memory process management method applied to intelligent customs supervision according to claim 1, characterized in that, Loading the machine learning model file from disk into the worker process's memory space includes: Send a model file read request to the resource storage of the big data platform to obtain the serialized model file byte stream; The model file byte stream is converted into an executable model object in memory through a deserialization operation; The executable model object is stored in a dictionary member of the global cache object.

4. The memory process management method applied to intelligent customs supervision according to claim 1, characterized in that, The cache hot update monitoring thread periodically scans the disk directories of the feature parameter files and the model files in a polling manner to monitor the file change status under the disk directories, including: The cache hot update monitoring thread is started independently within the working process, with a preset time interval as the polling period; In each polling cycle, read the last modification timestamp or MD5 value of a specific pattern file in the disk directory; The last modification timestamp or MD5 value read is compared with the load timestamp or MD5 value recorded in the global cache object to determine whether the files in the disk directory have been added, modified, or deleted.

5. The memory process management method applied to intelligent customs supervision according to claim 1, characterized in that, When the cache hot update monitoring thread detects changes to files in the disk directory, it triggers a cache refresh, reloading the changed files and updating the corresponding cache area in the global cache object, or clearing cache entries corresponding to deleted files from the global cache object, including: When the cache hot update monitoring thread detects that a new file has been added or the last modification timestamp of an existing file has changed in the disk directory, a cache refresh is triggered. The modified files are reloaded from disk into memory, and the corresponding cache area in the global cache object is updated. When the cache hot update monitoring thread determines that a file in the disk directory has been deleted, it triggers the cache refresh to clean up the cache entry corresponding to the deleted file from the global cache object.

6. The memory process management method applied to intelligent customs supervision according to claim 1, characterized in that, Access the memory data in the global cache object, and perform data matching based on the key fields of the current customs declaration and the grouping index, including: Access the memory data in the global cache object to obtain the key fields of the current customs declaration, including import / export method, port code and HS code; Based on the import / export method, port code, and HS code of the current customs declaration, locate the pre-built light quantum set corresponding to the scenario of the customs declaration from the global cache object; The merge vectorization computation method of the Pandas library is used to associate and match the current customs declaration data with the located light quantum set.

7. The memory process management method applied to intelligent customs supervision according to claim 1, characterized in that, The workflow, after receiving the customs declaration data request but before performing risk identification calculations, also includes: When the cache query interface provided to the outside world is called by a business request, it checks the last modification timestamp of the corresponding source file under the source file path before returning data; The last modification timestamp of the detected source file is compared with the cache loading timestamp recorded in the global cache object. If it is determined that the source file has been modified after the cache is loaded, the synchronous reload operation of the cache item is triggered, the source file is reloaded from the disk and the global cache object is updated. After the synchronization reload operation is completed, the updated cached data is returned through the cache query interface for the risk identification calculation.

8. A memory process management device for intelligent customs supervision, characterized in that, include: A module is created to start the main process of the model service and to create an independent file system listening thread, which continuously monitors change events of program files. An initialization module is used for the main process to spawn at least one worker process. Each worker process executes an initialization routine after being spawned. The initialization routine includes: loading the required feature parameter files and machine learning model files from the disk into the memory space of the worker process, constructing them as a global cache object, and pre-constructing a grouped index based on the key fields of the feature parameter files. The refresh module is used to send a restart command to the main process when the file system listening thread detects that the program file has been changed. The main process responds to the restart command and restarts all working processes. The restarted working processes re-execute the initialization routine to achieve a full cache refresh after the program is updated. The monitoring module is used to establish a cache hot update monitoring thread independently of the file system monitoring thread in each of the working processes. The cache hot update monitoring thread periodically scans the disk directory of the feature parameter file and the model file in a polling manner to monitor the file change status under the disk directory. The caching module is used to trigger a cache refresh when the cache hot update monitoring thread detects that a file in the disk directory has changed. This refresh will reload the changed file and update the corresponding cache area in the global cache object, or clean up the cache entries corresponding to deleted files from the global cache object. The identification module is used by the working process to access the memory data in the global cache object after receiving the customs declaration data request, complete the data matching based on the key fields of the current customs declaration and the group index, and call the corresponding executable model object from the model file to perform risk identification calculation and return the calculation result.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the memory process management method for intelligent customs supervision as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the memory process management method for customs intelligent supervision as described in any one of claims 1 to 7.

Images