Batch file import system and method for mapping memory queue

Abstract

The application relates to the technical field of computer control, and discloses a batch file import system of a mapping memory queue and an import method thereof, which comprises a multi-channel access module to connect various access modes of a Web interface and an API interface; a metafile extraction module receives and stores metafiles through a data import layer and takes the metafiles as file structures, automatically identifies different file structures through AI technology to automatically fragment, and performs cloud native asynchronous extraction and transmission; a dynamic mapping module establishes a distributed task scheduling engine through a self-defined function, performs elastic resource scheduling on the basis of Kubernetes to realize resource dynamic scaling of a file structure; and a mapping management module performs compliance verification on multidimensional data, verifies a syntax layer and a semantic layer in the file structure, and intelligently identifies and marks abnormal data of the file structure.

Description

Technical Field

[0001] This invention relates to the field of computer control technology, and more specifically to a batch file import system and method using a mapped memory queue. Background Technology

[0002] Queuing is a first-in, first-out (FIFO) data structure. It acts as a container for messages, with messages enqueued from the rear and dequeued from the front; enqueuing is the process of sending a message, and dequeuing is the process of receiving a message. It is primarily used to decouple computing units, achieve redundancy, horizontal scaling, flexible configuration, and peak / valley smoothing. For any IT architecture or application software, queuing is a crucial component.

[0003] However, in queuing technology, the enqueue rate often exceeds the dequeue rate, causing the data in the queue to gradually increase and eventually lead to backlog. This can result in millions of data entries piling up in the queue, making it impossible to provide timely storage services. Furthermore, the backlog of millions of data entries increases the risk of data loss. Currently, to address these technical issues, enterprises typically purchase additional machines to expand capacity and alleviate queue backlog. However, machine expansion is very costly. Moreover, even after hardware expansion, while it can increase the upload speed of data in the queue to some extent, it still presents challenges.

[0004] However, when dealing with large-scale enterprise-level data, the throughput of the import process in the queue data is often low. Therefore, the company designed a batch file import system and method for a mapping memory queue that achieves seamless integration of large-scale enterprise-level data through cloud-native architecture and asynchronous processing mechanism. Summary of the Invention

[0005] This invention aims to provide an efficient, reliable, and scalable batch data import system for DMS (Document Management System). Through cloud-native architecture and asynchronous processing mechanisms, it realizes a batch file import system and method for seamless integration of enterprise-level large-scale data using a mapping memory queue.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: 1) A batch file import system with a mapped memory queue, comprising: The multi-channel access module is configured to connect to the data import layer of the API gateway to enable various access modes, including web interfaces and API interfaces. The metafile extraction module is configured as a data storage layer with OSS-based distributed object storage and structured data storage. It receives and stores metafiles through the data import layer and uses metafiles as the file structure. It automatically identifies different file structures through AI technology and automatically segments and extracts and transmits them in a cloud-native asynchronous manner. The dynamic mapping module is configured with custom functions that define mapping rules based on JSON Schema. It uses these custom functions to build a distributed task scheduling engine and uses file structures to perform elastic resource scheduling based on Kubernetes for dynamic scaling of resources. The mapping management module is configured to perform compliance verification based on the dynamic mapping module, using multi-dimensional data to verify the syntax and semantic layers within the file structure, and to intelligently identify and mark abnormal data in the file structure.

[0007] This invention adopts a layered and decoupled design, with a multi-channel access module, a metafile extraction module, a dynamic mapping module, and a mapping management module working together to perform operations such as integrated object storage, data queues, and distributed task scheduling. At the same time, the multi-channel access module imports data from the API gateway into the metafile extraction module through multiple access modes, including connecting to the Web interface and API interface.

[0008] In this process, the metafile extraction module and the dynamic mapping module are combined. The file structure received by the metafile extraction multi-channel access module is stored in the data storage layer, and distributed object storage and structured data storage based on OSS are used. AI technology is combined to automatically identify different file results and transmit them with automatic sharding cloud-native asynchronous extraction. During the transmission, the designed dynamic mapping module acts on the automatic sharding cloud-native asynchronous extraction and transmission in the metafile extraction module. The dynamic mapping module defines the mapping rules based on JSON Schema and establishes a distributed task scheduling engine with the custom function. The file structure is based on Kubernetes to dynamically scale resources and perform elastic scheduling of imported files to adapt to the batch import status of different environments, thereby greatly reducing the data backlog of files in the queue.

[0009] Meanwhile, based on the dynamic mapping module combined with the metafile extraction module in the queue data, the mapping management module acts on the dynamic mapping module, the metafile extraction module and the multi-channel access module to perform compliance verification with multi-dimensional data, verify the syntax and semantic layers within the file structure, and intelligently identify and mark abnormal data in the file structure, thereby ensuring high throughput and strong consistency in the data file import process in the queue, while providing comprehensive data governance and compliance assurance.

[0010] 2) A batch file import system based on a mapped memory queue as described in 1), wherein the data import layer that interfaces with the API gateway, and the multi-channel access module that interfaces with multiple access modes such as Web interface and API interface, include: The system includes a host channel interface module, a storage device interface module, a network interface module, and a control module. The host channel interface module, storage device interface module, and network interface module are connected to the control module via a PCI bus to form a multi-channel storage device with host channels, storage device channels, and network channels. The server accesses the host channel through its configured peripheral adapter; the storage devices connect to the same or different networks as the server through their respective network interfaces.

[0011] This invention increases storage capacity by adding multi-channel storage devices, while also increasing the number of data channels, achieving simultaneous expansion and speed increase. The operation of multiple storage devices is highly parallel, enabling not only parallel access operations but also parallel network data transmission. At the same time, it achieves centralized file system management and distributed data access, ensuring high management efficiency and high storage performance.

[0012] 3) A batch file import system based on a mapped memory queue as described in 1), wherein: The metafile extraction module has an Alibaba Cloud OSS / MinIO+ distributed relational database. The database has an Excel file structure that can be set. The Excel file structure has different fields. The AI ​​technology identifies the fields in the Excel file structure and, together with the dynamic mapping module, establishes a field mapping relationship between the fields and the data transmitted by the multi-channel access module.

[0013] This invention uses Alibaba Cloud OSS / MinIO+ distributed relational database as a data quality measurement system. It uses AI technology to identify different fields of the Excel file structure set in the database and establish a field mapping relationship between the fields and the data transmitted by the multi-channel access module. This can reduce the missing rate of required fields and maintain the integrity of the recorded Excel file structure.

[0014] 4) A batch file import system based on a mapped memory queue as described in 3), wherein: The field mapping relationship is set within the dynamic mapping module. The metafile extraction module uses AI technology to analyze and organize the common network element configuration from the configuration data of various device files. Combined with the custom function in the dynamic mapping module that defines mapping rules based on JSONSchema, a field mapping table is generated. The field mapping table is then combined with an automatically established distributed task scheduling engine using functions.

[0015] 5) A batch file import system based on a mapped memory queue as described in 1), wherein the file extraction module for automatic sharding cloud-native asynchronous extraction and transmission includes: It utilizes a cloud-native asynchronous transmission system to achieve automatic fragmentation and asynchronous extraction transmission of different file structures. Within this system, it features a distributed architecture for cloud-native asynchronous upload. The fragmented upload mechanism enables automatic fragmentation and parallel upload of file structures. Based on OSS's Multipart Upload feature, it achieves automatic recovery after network interruption. Simultaneously, OSS's Multipart Upload feature is applied to the distributed construction of the cloud-native asynchronous upload to monitor key indicators such as upload speed and success rate, dynamically adjust transmission strategies, ensure end-to-end encrypted transmission, and support STS temporary authorization and signature verification.

[0016] This invention utilizes a cloud-native asynchronous upload distributed architecture as a prior art. This architecture decouples data transmission and processing into independent components. In this invention, a multi-channel access module is used to upload data to the metafile extraction module for storage. After storage, a message queue is used to persist the uploaded data. Based on a fragmented upload mechanism, the file structure is automatically fragmented and uploaded in parallel, breaking through the size limit of a single file structure. Based on the Multipart Upload feature of OSS, automatic recovery after network interruption is achieved, thereby providing a guarantee for network interrupted transmission and reducing the phenomenon of data loss during upload.

[0017] By leveraging OSS's Multipart Upload feature to implement a distributed architecture for native asynchronous uploads, key metrics such as upload speed and success rate can be monitored, and transmission strategies can be dynamically adjusted to achieve transmission quality monitoring. At the same time, encrypted transmission is performed throughout the upload process, and STS temporary authorization and signature verification are supported, which ensures the security of the transmission channel and guarantees the security of the data.

[0018] 6) A batch file import system based on a mapping memory queue as described in 1), which includes a custom function that defines mapping rules based on JSONSchema, and uses the custom function to build a distributed task scheduling engine, including the following steps: AI technology is used to automatically identify the file structure and combine it with a custom function in the dynamic mapping module that defines mapping rules based on JSON Schema to generate a field mapping table. The fields in the field mapping table correspond to the data of the matching multi-channel access mode. The location of the data storage of the current multi-channel access is obtained from the field mapping table, and the mapping relationship of field name, field type and field value in the data is extracted and stored. The custom function, which defines mapping rules based on JSON Schema, is parsed to obtain the verification parameters in the current custom function that match the field mapping table. The verification parameters are then bound to the relevant field names in the tracking dataset. The field mapping table is combined with the function to automatically build a distributed task scheduling engine for uploading.

[0019] 7) A batch file import system based on a mapped memory queue as described in 6), wherein: The distributed task scheduling engine includes intelligent task decomposition, elastic resource scheduling, priority queue management, and task status tracking. The intelligent task decomposition automatically splits and processes tasks based on file data size and data load of each access mode. The elastic resource scheduling uses Kubernetes to dynamically scale computing resources. The priority queue management is designed with multiple levels of task priorities and combined with an SLA guarantee mechanism. The task status tracking has full lifecycle status management.

[0020] 8) A batch file import system for a mapping memory queue as described in 1), wherein the dynamic mapping module for elastic resource scheduling based on Kubernetes for dynamic resource scaling using file structure includes: The dynamic mapping module is used to obtain the upload volume of different access modes within the multi-channel access module; The load information uploaded to the metadata extraction module in each access mode is obtained. Based on the data load information of each access mode in the metadata extraction module, the threshold for the data that can be processed in the multi-channel access mode is determined. When the data that can be processed reaches the set threshold, Kubernetes is triggered to perform elastic resource scheduling for dynamic scaling of resources based on Kubernetes.

[0021] 9) A batch file import system based on a mapping memory queue as described in 1), wherein a mapping management module performs compliance verification using multi-dimensional data, verifies the syntax and semantic layers within the file structure, and intelligently identifies and marks abnormal data in the file structure, including: The uploaded data is formed into a dataset, which is then subjected to multi-dimensional verification with a set of preset analysis strategies. The preset analysis strategies perform syntactic layer verification on basic rules of data type, format, and length, semantic layer verification on business rules, dependencies, and data consistency detection, and compliance verification on GDPR and industry norms and regulations. Meanwhile, intelligent anomaly detection is guided by abnormal data and identifies and marks abnormal data based on machine learning.

[0022] 10) A batch file import method for a mapped memory queue, applied to the batch file import system for a mapped memory queue described in 1)-9) above, comprising the following steps: Step 1: Connect to the data import layer of the API gateway to support various access modes, including web interfaces and API interfaces; Step 2: The data storage layer based on OSS distributed object storage and structured data storage receives and stores meta files through the data import layer and uses meta files as the file structure. AI technology is used to automatically identify different file structures and automatically shard them for cloud-native asynchronous extraction and transmission. Step 3: Define a custom function based on JSON Schema to define mapping rules, build a distributed task scheduling engine with the custom function, and use the file structure to perform elastic resource scheduling based on Kubernetes for dynamic scaling of resources; Step 4: Perform compliance verification using multi-dimensional data, verify the syntax and semantic layers within the file structure, and intelligently identify and mark abnormal data in the file structure.

[0023] This invention constructs a data quality measurement system through a multi-channel access module, a meta-file extraction module, a dynamic mapping module, and a mapping management module. The data quality measurement system achieves completeness indicators through the missing rate of required fields and record completeness scores, accuracy indicators through format error rate and business rule violation rate, consistency indicators through cross-table association consistency and time-series data continuity, and timeliness indicators through data latency and processing cycle analysis.

[0024] This invention features a multi-level fault tolerance mechanism: task-level fault tolerance is achieved through automatic retry and degradation processing of failed tasks; service-level fault tolerance is achieved by using a circuit breaker to prevent cascading failures; and data center-level fault tolerance is achieved through cross-availability zone deployment and automatic failover. Simultaneously, this invention also provides data consistency guarantees: an eventual consistency model is implemented by ensuring eventual consistency through event tracing; a compensation transaction mechanism is implemented through automatic data repair in abnormal scenarios; and periodic reconciliation is achieved through automatic comparison and discrepancy repair of source and target data. Attached Figure Description

[0025] Figure 1 This is a framework diagram of a batch file import system with a mapping memory queue according to the present invention; Figure 2 This is a flowchart illustrating a batch file import method using a mapping memory queue according to the present invention. Detailed Implementation

[0026] The following detailed description illustrates the specific implementation method: See the example. Figure 1As shown, a batch file import system with a mapping memory queue includes a multi-channel access module, a metafile extraction module, a dynamic mapping module, and a mapping management module. The multi-channel access module is configured to connect to the data import layer of an API gateway to support various access modes such as web interfaces and API interfaces. The metafile extraction module is configured as a data storage layer with OSS-based distributed object storage and structured data storage. It receives and stores metafiles through the data import layer and uses metafiles as the file structure. It automatically identifies different file structures using AI technology to automatically shard and extract and transmit cloud-native asynchronously.

[0027] The dynamic mapping module is configured to have custom functions that define mapping rules based on JSON Schema. These custom functions are used to build a distributed task scheduling engine, and the file structure is used to perform elastic resource scheduling based on Kubernetes for dynamic scaling of resources. The mapping management module is configured to perform compliance verification based on the dynamic mapping module, verify the syntax and semantic layers within the file structure, and intelligently identify and mark abnormal data in the file structure.

[0028] This embodiment adopts a layered decoupled design, with a multi-channel access module, a metafile extraction module, a dynamic mapping module, and a mapping management module working together to perform operations such as fusion object storage, data queues, and distributed task scheduling. At the same time, the multi-channel access module imports data from the API gateway into the metafile extraction module through multiple access modes, including connecting to the Web interface and API interface.

[0029] In this process, the metafile extraction module and the dynamic mapping module are combined. The file structure received by the metafile extraction multi-channel access module is stored in the data storage layer, and distributed object storage and structured data storage based on OSS are used. AI technology is combined to automatically identify different file results and transmit them with automatic sharding cloud-native asynchronous extraction. During the transmission, the designed dynamic mapping module acts on the automatic sharding cloud-native asynchronous extraction and transmission in the metafile extraction module. The dynamic mapping module defines the mapping rules based on JSON Schema and establishes a distributed task scheduling engine with the custom function. The file structure is based on Kubernetes to dynamically scale resources and perform elastic scheduling of imported files to adapt to the batch import status of different environments, thereby greatly reducing the data backlog of files in the queue.

[0030] Furthermore, in this embodiment, the dynamic mapping module for elastic resource scheduling based on Kubernetes using a file structure for dynamic resource scaling includes: The dynamic mapping module is used to obtain the upload volume of different access modes within the multi-channel access module; The load information uploaded to the metadata extraction module in each access mode is obtained. Based on the data load information of each access mode in the metadata extraction module, the threshold for the data that can be processed in the multi-channel access mode is determined. When the data that can be processed reaches the set threshold, Kubernetes is triggered to perform elastic resource scheduling for dynamic scaling of resources based on Kubernetes.

[0031] Simultaneously, a mapping management module that performs compliance verification using multi-dimensional data, verifies the syntax and semantic layers within the file structure, and intelligently identifies and marks abnormal data within the file structure includes: The uploaded data is formed into a dataset, which is then subjected to multi-dimensional verification against a set of preset analysis strategies. The preset analysis strategies perform syntactic layer verification on basic rules for data type, format, and length, semantic layer verification on business rules, dependencies, and data consistency detection, and compliance verification on GDPR and industry norms and regulations. Meanwhile, intelligent anomaly detection is guided by abnormal data and identifies and marks abnormal data based on machine learning.

[0032] Meanwhile, based on the dynamic mapping module combined with the metafile extraction module in the queue data, the mapping management module acts on the dynamic mapping module, the metafile extraction module and the multi-channel access module to perform compliance verification with multi-dimensional data, verify the syntax and semantic layers within the file structure, and intelligently identify and mark abnormal data in the file structure, thereby ensuring high throughput and strong consistency in the data file import process in the queue, while providing comprehensive data governance and compliance assurance.

[0033] Furthermore, in this embodiment, the data import layer that interfaces with the API gateway, and the multi-channel access module that interfaces with various access modes such as the Web interface and API interface, include: The host channel interface module, storage device interface module, network interface module, and control module are connected to the control module via a PCI bus to form a multi-channel storage device with host channel, storage device channel, and network channel. The server connects to the host channel through its configured peripheral adapter; the storage devices connect to the same or different networks as the server through their respective network interfaces.

[0034] This embodiment increases storage capacity by adding multi-channel storage devices, while also increasing the number of data channels, achieving simultaneous expansion and speed increase. The operation of multiple storage devices is highly parallel, enabling not only parallel access operations but also parallel network data transmission. At the same time, it achieves centralized file system management and distributed data access, ensuring high management efficiency and high storage performance.

[0035] Meanwhile, the metafile extraction module has an Alibaba Cloud OSS / MinIO+ distributed relational database. The database has an option to set the Excel file structure, with different fields in the Excel file structure. AI technology is used to identify the fields in the Excel file structure, and combined with the dynamic mapping module, a field mapping relationship is established between the fields and the data transmitted by the multi-channel access module.

[0036] This embodiment uses Alibaba Cloud OSS / MinIO+ distributed relational database as the data quality measurement system. It uses AI technology to identify different fields of the Excel file structure set in the database and establish a field mapping relationship between the fields and the data transmitted by the multi-channel access module. This can reduce the missing rate of required fields and maintain the integrity of the recorded Excel file structure.

[0037] The field mapping relationship is set within the dynamic mapping module. The metafile extraction module uses AI technology to analyze and organize the common network element configuration from the configuration data of various device files. Combined with the custom function in the dynamic mapping module that defines mapping rules based on JSONSchema, a field mapping table is generated. The field mapping table is then combined with an automatically established distributed task scheduling engine using functions.

[0038] Furthermore, in this embodiment, the file extraction module for automatic fragmented cloud-native asynchronous extraction transmission includes: It utilizes a cloud-native asynchronous transmission system to achieve automatic fragmentation and asynchronous extraction transmission of different file structures. Within this system, it features a distributed architecture for cloud-native asynchronous upload. The fragmented upload mechanism enables automatic fragmentation and parallel upload of file structures. Based on OSS's Multipart Upload feature, it achieves automatic recovery after network interruption. Simultaneously, OSS's Multipart Upload feature is applied to the distributed construction of the cloud-native asynchronous upload to monitor key indicators such as upload speed and success rate, dynamically adjust transmission strategies, ensure end-to-end encrypted transmission, and support STS temporary authorization and signature verification.

[0039] This embodiment uses a cloud-native asynchronous upload distributed architecture as an existing technology. The cloud-native asynchronous upload distributed architecture decouples data transmission and processing into independent components. In this embodiment, a multi-channel access module is used to upload data to the metafile extraction module for storage. After storage, a message queue is used to persist the uploaded data. Based on the fragmented upload mechanism, the file structure is automatically fragmented and uploaded in parallel, breaking through the limitation of the size of a single file structure. Based on the Multipart Upload feature of OSS, automatic recovery after network interruption is realized, thereby providing network breakpoint resume guarantee and reducing the phenomenon of data loss during upload.

[0040] By leveraging OSS's Multipart Upload feature to implement a distributed architecture for native asynchronous uploads, key metrics such as upload speed and success rate can be monitored, and transmission strategies can be dynamically adjusted to achieve transmission quality monitoring. At the same time, encrypted transmission is performed throughout the upload process, and STS temporary authorization and signature verification are supported, which ensures the security of the transmission channel and guarantees the security of the data.

[0041] There are custom functions that define mapping rules based on JSON Schema. A distributed task scheduling engine is built using these custom functions, including the following steps: AI technology is used to automatically identify the file structure and combine it with a custom function in the dynamic mapping module that defines mapping rules based on JSON Schema to generate a field mapping table. The fields in the field mapping table correspond to the data of the matching multi-channel access mode. The location of the data storage of the current multi-channel access is obtained from the field mapping table, and the mapping relationship of field name, field type and field value in the data is extracted and stored. The system parses custom functions that define mapping rules based on JSON Schema to obtain the validation parameters that match the field mapping table in the current custom function. It then binds the validation parameters to the relevant field names in the tracking dataset and uses the field mapping table to automatically build a distributed task scheduling engine for uploading.

[0042] The distributed task scheduling engine includes intelligent task decomposition, elastic resource scheduling, priority queue management, and task status tracking. Intelligent task decomposition automatically splits and processes tasks based on file data size and data load of each access mode. Elastic resource scheduling uses Kubernetes to dynamically scale computing resources. Priority queue management is designed with multiple levels of task priorities and combined with an SLA guarantee mechanism. Task status tracking has full lifecycle status management.

[0043] See Figure 2 A batch file import method using a mapped memory queue, applied to the aforementioned batch file import system using a mapped memory queue, includes the following steps: Step 1: Connect to the data import layer of the API gateway to support various access modes, including web interfaces and API interfaces; Step 2: The data storage layer based on OSS distributed object storage and structured data storage receives and stores meta files through the data import layer and uses meta files as the file structure. AI technology is used to automatically identify different file structures and automatically shard them for cloud-native asynchronous extraction and transmission. Step 3: Define a custom function based on JSON Schema to define mapping rules, build a distributed task scheduling engine with the custom function, and use the file structure to perform elastic resource scheduling based on Kubernetes for dynamic scaling of resources; Step 4: Perform compliance verification using multi-dimensional data, verify the syntax and semantic layers within the file structure, and intelligently identify and mark abnormal data in the file structure.

[0044] This embodiment uses a multi-channel access module, a meta-file extraction module, a dynamic mapping module, and a mapping management module to jointly construct a data quality measurement system. The data quality measurement system achieves completeness indicators through the missing rate of required fields and record completeness scores; accuracy indicators through format error rate and business rule violation rate; consistency indicators through cross-table association consistency and time-series data continuity; and timeliness indicators through data latency and processing cycle analysis.

[0045] This embodiment features a multi-level fault tolerance mechanism: automatic retry and degradation handling for failed tasks achieves task-level fault tolerance; circuit breaker mode prevents cascading failures to achieve service-level fault tolerance; and cross-availability zone deployment and automatic failover achieve data center-level fault tolerance. Simultaneously, this embodiment also provides data consistency guarantees: eventual consistency is ensured through event tracing to achieve an eventual consistency model; automatic data repair in abnormal scenarios achieves a compensation transaction mechanism; and automatic comparison and discrepancy repair between source and target data achieves periodic reconciliation.

[0046] The above are merely embodiments of this solution. Commonly known technical solutions and / or characteristics are not described in detail here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the technical solution of this solution. These modifications and improvements should also be considered within the scope of protection of this solution, and will not affect the effectiveness of the implementation of this solution or the practicality of the patent. The scope of protection claimed in this application shall be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A batch file import system using a mapped memory queue, characterized in that, include: The multi-channel access module is configured to connect to the data import layer of the API gateway to enable various access modes, including web interfaces and API interfaces. The metafile extraction module is configured as a data storage layer with OSS-based distributed object storage and structured data storage. It receives and stores metafiles through the data import layer and uses metafiles as the file structure. It automatically identifies different file structures through AI technology and automatically segments and extracts and transmits them in a cloud-native asynchronous manner. The dynamic mapping module is configured with custom functions that define mapping rules based on JSON Schema. It uses these custom functions to build a distributed task scheduling engine and uses file structures to perform elastic resource scheduling based on Kubernetes for dynamic scaling of resources. The mapping management module is configured to perform compliance verification based on the dynamic mapping module, using multi-dimensional data to verify the syntax and semantic layers within the file structure, and to intelligently identify and mark abnormal data in the file structure.

2. The batch file import system with a mapping memory queue according to claim 1, characterized in that: The data import layer, which interfaces with the API gateway, is a multi-channel access module that supports various access modes, including web interfaces and API interfaces. The system includes a host channel interface module, a storage device interface module, a network interface module, and a control module. The host channel interface module, storage device interface module, and network interface module are connected to the control module via a PCI bus to form a multi-channel storage device with host channels, storage device channels, and network channels. The server accesses the host channel through its configured peripheral adapter; the storage devices connect to the same or different networks as the server through their respective network interfaces.

3. The batch file import system with a mapping memory queue according to claim 1, characterized in that: The metafile extraction module has an Alibaba Cloud OSS / MinIO+ distributed relational database. The database has an Excel file structure that can be set. The Excel file structure has different fields. The AI ​​technology identifies the fields in the Excel file structure and, together with the dynamic mapping module, establishes a field mapping relationship between the fields and the data transmitted by the multi-channel access module.

4. The batch file import system with a mapping memory queue according to claim 3, characterized in that: The field mapping relationship is set within the dynamic mapping module. The metafile extraction module uses AI technology to analyze and organize the common network element configuration from the configuration data of various device files. Combined with the custom function in the dynamic mapping module that defines mapping rules based on JSON Schema, a field mapping table is generated. The field mapping table is then combined with an automatically established distributed task scheduling engine using functions.

5. A batch file import system with a mapping memory queue according to claim 1, characterized in that: The file extraction module, which uses automatic fragmentation and cloud-native asynchronous extraction for transmission, includes: It utilizes a cloud-native asynchronous transmission system to achieve automatic fragmentation and asynchronous extraction transmission of different file structures. Within this system, it features a distributed architecture for cloud-native asynchronous upload. The fragmented upload mechanism enables automatic fragmentation and parallel upload of file structures. Based on OSS's Multipart Upload feature, it achieves automatic recovery after network interruption. Simultaneously, OSS's Multipart Upload feature is applied to the distributed construction of the cloud-native asynchronous upload to monitor key indicators such as upload speed and success rate, dynamically adjust transmission strategies, ensure end-to-end encrypted transmission, and support STS temporary authorization and signature verification.

6. The batch file import system with a mapping memory queue according to claim 1, characterized in that: This includes custom functions that define mapping rules based on JSON Schema. A distributed task scheduling engine is built using these custom functions, and the process involves the following steps: AI technology is used to automatically identify the file structure and combine it with a custom function in the dynamic mapping module that defines mapping rules based on JSON Schema to generate a field mapping table. The fields in the field mapping table correspond to the data of the matching multi-channel access mode. The location of the data storage of the current multi-channel access is obtained from the field mapping table, and the mapping relationship of field name, field type and field value in the data is extracted and stored. The custom function, which defines mapping rules based on JSON Schema, is parsed to obtain the verification parameters in the current custom function that match the field mapping table. The verification parameters are then bound to the relevant field names in the tracking dataset. The field mapping table is combined with the function to automatically build a distributed task scheduling engine for uploading.

7. A batch file import system with a mapping memory queue according to claim 6, characterized in that: The distributed task scheduling engine includes intelligent task decomposition, elastic resource scheduling, priority queue management, and task status tracking. The intelligent task decomposition automatically splits and processes tasks based on file data size and data load of each access mode. The elastic resource scheduling uses Kubernetes to dynamically scale computing resources. The priority queue management is designed with multiple levels of task priorities and combined with an SLA guarantee mechanism. The task status tracking has full lifecycle status management.

8. A batch file import system with a mapping memory queue according to claim 1, characterized in that: The dynamic mapping module for elastic resource scheduling based on Kubernetes and using a file structure for dynamic resource scaling includes: The dynamic mapping module is used to obtain the upload volume of different access modes within the multi-channel access module; The load information uploaded to the metadata extraction module in each access mode is obtained. Based on the data load information of each access mode in the metadata extraction module, the threshold for the data that can be processed in the multi-channel access mode is determined. When the data that can be processed reaches the set threshold, Kubernetes is triggered to perform elastic resource scheduling for dynamic scaling of resources based on Kubernetes.

9. A batch file import system with a mapping memory queue according to claim 1, characterized in that: The compliance verification module uses multi-dimensional data to verify the syntax and semantic layers within the file structure, and performs intelligent anomaly data identification and tagging within the file structure. This mapping management module includes: The uploaded data is formed into a dataset, which is then subjected to multi-dimensional verification with a set of preset analysis strategies. The preset analysis strategies perform syntactic layer verification on basic rules of data type, format, and length, semantic layer verification on business rules, dependencies, and data consistency detection, and compliance verification on GDPR and industry norms and regulations. Meanwhile, intelligent anomaly detection is guided by abnormal data and identifies and marks abnormal data based on machine learning.

10. A batch file import method for a mapped memory queue, applied to a batch file import system for a mapped memory queue as described in any one of claims 1-9, characterized in that: Includes the following steps: Step 1: Connect to the data import layer of the API gateway to support various access modes, including web interfaces and API interfaces; Step 2: The data storage layer based on OSS distributed object storage and structured data storage receives and stores meta files through the data import layer and uses meta files as the file structure. AI technology is used to automatically identify different file structures and automatically shard them for cloud-native asynchronous extraction and transmission. Step 3: Define a custom function based on JSON Schema to define mapping rules, build a distributed task scheduling engine with the custom function, and use the file structure to perform elastic resource scheduling based on Kubernetes for dynamic scaling of resources; Step 4: Perform compliance verification using multi-dimensional data, verify the syntax and semantic layers within the file structure, and intelligently identify and mark abnormal data in the file structure.

Images