Multimodal data archiving and processing system based on cloud computing

Abstract

The present invention provides a multimodal data archiving processing system based on cloud computing, which relates to the field of data processing technology. The system is used to: obtain archiving requests and archiving tasks, and determine the archiving type, processing priority and task number based on them; record the memory identification, execution path and cache address of the processing node; extract the structured field group and unstructured content block of the archiving task; generate a structure reference table by mapping the structured field group according to the fields; segment the unstructured content block in chronological order to generate a content time stream; record each field mapping and time segmentation operation and determine its reference relationship with the processing node; determine the task number, resource status and data fragment of each processing node; obtain state anchor data; combine the structure reference table and content time stream with the state anchor data into a two-layer archiving component, and archive the component. The present invention can perform content backtracking and operation backtracking in the multimodal data archiving process.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal data archiving and processing system based on cloud computing. Background Art

[0002] In existing multimodal data management systems, structured data, unstructured documents, and metadata are scattered and disparate, making unified packaging and archiving difficult. Some archiving processes rely on local tools and manual rule configuration, resulting in complex implementations and poor environmental adaptability. Inconsistent data semantics and unstandardized structural models often lead to a lack of universal descriptions for archived data, limiting the portability and repeatability of data packages. Although some systems support data extraction and dedomaining, the archiving process requires simultaneous processing of multimodal content, and existing methods may not be able to form logically complete, semantically self-describing archive packages.

[0003] For example, there is a lack of unified identification and reference mechanism between PDF files and XML structured forms, and the data content is disconnected from the process information. It may be impossible to fully reproduce the original business logic, affecting the authenticity of the archived data. Summary of the Invention

[0004] The purpose of the present invention is to provide a multimodal data archiving and processing system based on cloud computing, aiming to solve the problems mentioned in the background technology.

[0005] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0006] A multimodal data archiving and processing system based on cloud computing, the system comprising:

[0007] The configuration module is used to obtain the archiving request and the archiving task, and determine the archiving type, processing priority and task number according to the archiving request and the task number, and obtain the task configuration data;

[0008] The resource module is used to record the memory identification, execution path and cache address of the processing node before the archiving task is started, and obtain the resource status data set;

[0009] A processing module is used to extract structured field groups and unstructured content blocks based on task configuration data, generate a structure reference table by mapping structured field groups by fields, and generate a content time stream by segmenting unstructured content blocks in chronological order;

[0010] The recording module is used to record each field mapping and time segmentation operation, obtain operation event data, and determine its reference relationship with the processing node to obtain process link data;

[0011] The synchronization module is used to match the resource status data set with the processing nodes in the process link data, determine the task number, resource status and data fragment of each processing node, and obtain the state anchor data;

[0012] an execution module for combining the structure reference table and the content time stream with the state anchor data into a two-layer archive component, wherein the two-layer archive component includes a data component stream and an operation link set;

[0013] The archiving module is used to archive the double-layer archiving components and generate process index data.

[0014] Furthermore, the configuration module includes:

[0015] A type determination unit is configured to receive an archiving request, match the type field of the archiving task with the archiving type list, obtain a matching item, and determine the archiving type when the matching item is greater than a preset matching threshold;

[0016] The path judgment unit is used to perform hierarchical expansion on the path field of the archiving task, divide the path characters into multiple path segments using the hierarchical separator, and judge whether each path segment meets the requirements according to the path pattern list. If the result is yes, it marks it as a valid path segment and combines the valid path segments into an archiving path structure;

[0017] The merging unit is used to merge the archiving type and the archiving path structure to obtain the archiving source data identification item and assign an archiving task number to it;

[0018] The priority unit is used to generate a processing priority according to the archiving task and the task queue it is in, and bind it with the archiving task number to obtain task configuration data.

[0019] Furthermore, the resource module includes:

[0020] The node screening unit is used to call the cloud platform resource management interface according to the archived task number, obtain the list of currently available processing nodes, and screen the processing nodes whose idleness and computing power meet the minimum execution standards to obtain the node candidate set;

[0021] The resource status unit is used to request the resource status of each processing node in the node candidate set, call the node interface to extract the memory identification, execution path and cache address of the processing node, and obtain the resource status characteristics;

[0022] The binding unit is used to group the resource status features into a triple structure and bind it with the corresponding archive task number to obtain a resource status mapping item;

[0023] The number grouping unit is used to group the resource status mapping items according to the archiving task numbers and assign resource snapshot numbers to obtain a resource status data set.

[0024] Furthermore, the processing module includes:

[0025] The path parsing unit is used to extract the structured field group from the archiving task, and perform path parsing on each structured field according to the field path, extract the path label of each level, obtain the field path stack, and set the last level field value as the field end value to obtain a path value pair;

[0026] A sub-segment mapping unit is used to detect the path-value pair, determine the sub-segment type of the field value, and call the corresponding field mapping structure according to the sub-segment type to obtain a structure reference table, wherein the structure reference table includes a path hierarchical structure, a field type label, and a standard field code;

[0027] The time partitioning unit is used to extract unstructured content blocks from the archiving task, sort them in chronological order, and then partition them into fixed time segments and insert anchor points at the segmentation points to obtain content slices;

[0028] The sorting unit is used to arrange each content slice in sequence to obtain a content time stream, wherein the content time stream includes the start and end time of the paragraph, the number of words in the paragraph, the anchor field path and the paragraph index value.

[0029] Furthermore, the recording module includes:

[0030] The field operation unit is used to monitor each field mapping operation in the generation of the structure reference table, seal the path label, field type and sub-segment mapping structure during path resolution, and record the task number of the operation to obtain the field operation data;

[0031] The paragraph operation unit is used to track each time slice operation in the content time stream generation, classify and identify the anchor point of each content slice, and record the paragraph index, paragraph word count and segmentation number to obtain paragraph operation data;

[0032] A combining unit is used to combine and sort the field operation data and the paragraph operation data in the order of their recording time, and add the operation number to them to obtain the operation event data set;

[0033] The node mapping unit is used to query the processing node number of each operation event in the operation event data set in the resource status data set, determine the mapping relationship, and obtain process link data.

[0034] Furthermore, the synchronization module includes:

[0035] The resource matching unit is used to extract the processing node number of each operation event in the process link data, and query the resource status data set for a triple structure with the same processing node number and task number to obtain a matching resource group;

[0036] A state binding unit is used to use the memory identifier, execution path, and cache address of the matching resource group as state parameters, and associate them with the path label or paragraph index in the operation event to obtain a state binding item;

[0037] The annotation merging unit is used to annotate the task number and data fragment of the state-bound item, generate the state-anchored sub-item, and merge all the state-anchored sub-items into state-anchored data according to the task number.

[0038] Furthermore, the execution module includes:

[0039] A field anchoring unit is used to retrieve the processing node number and memory address corresponding to each field path label in the structure reference table from the state anchoring data according to the path label of the path label, and to construct a field anchor pair;

[0040] The field binding unit is used to locate the co-occurrence position of the paragraph index and the field path in the field anchor pair in the content time stream, determine the logical relationship between the paragraph index and the field path, and construct the field binding item;

[0041] a combining unit, configured to combine the processing node number and memory address in the field anchor pair with the field path and paragraph index in the field binding item into a four-element index block, and append a task number thereto to obtain a data component unit;

[0042] The sorting and grouping unit is used to sort the data component units according to the hierarchical structure of the field path, and to archive and group them according to the task number to obtain the data component flow.

[0043] Furthermore, the execution module further includes:

[0044] The operation node unit is used to extract the operation number, processing node number and task number of each operation event from the state anchor data to obtain the operation node set;

[0045] An operation sequence unit is used to sort the processing nodes in the operation node set in ascending order according to the event number, and group them based on the task number to obtain a task operation sequence;

[0046] The event linked list unit is used to read the event number and data fragment one by one in the task operation sequence, and establish a one-way event connection pointer between two adjacent operation events to obtain the event linked list data;

[0047] The identifier merging unit is used to add a chain head anchor identifier and a chain tail end identifier to each event chain list, construct a task operation link entry, and combine all task operation link entries to obtain an operation link set.

[0048] Furthermore, the archiving type list includes a plurality of archiving type items, each archiving type item includes a type number field, a keyword set field, and an archiving template identification field, wherein the keyword set field includes a number of matching keywords related to the archiving type. When determining the archiving type, keywords are extracted from the type field in the archiving task and compared with the keyword set field. When the number of matching keywords exceeds a preset matching threshold, the archiving type item is determined to be the archiving type of the archiving task.

[0049] The path pattern list includes multiple path template items, each path template item includes a template number field, a path format field, a maximum level field and a wildcard allowed field, wherein the path format field is a legal path structure format, containing one or more levels of path segments, and the path segments are divided by separators. When the path field in the archiving task is expanded into multiple path segments, the number and content of the path segments are judged item by item to determine whether they comply with the path format field and the maximum level field. When they comply with the path template item, the path segment is marked as a valid path segment.

[0050] Furthermore, the data component stream is used to perform content backtracking of the archiving task during the archiving process; and the operation link set is used to perform operation backtracking of the archiving task during the archiving process.

[0051] The above solution of the present invention includes at least the following beneficial effects:

[0052] The present invention determines the task type and priority, and classifies, extracts and processes structured and unstructured data, and uniformly constructs the two types of data into a two-layer archiving component. The collaborative processing of structured field groups and unstructured content blocks no longer relies on traditional local archiving tools for manual matching and integration, but instead completes the integration operation through a standardized data interface, so that the archived data has a unified structure, clear path hierarchy and attribution information, can support the subsequent rapid decoding and access of different data, improve the packaging standardization of multimodal data archiving, and solve the problems of data dispersion and non-uniform packaging.

[0053] The present invention establishes a structure reference table and content time stream during the processing process and records each processing node and task number, thereby retaining the correspondence between content, structure and process in the final archiving component, so that the archived data is not only a collection of existing information, but also a compressed expression of the business processing path. The structure reference table describes the field path and label, and the content time stream reflects the time process of unstructured data. This information together constitutes the contextual semantics of the data, so that the semantics of the archived data can be restored when the archive package is retrieved or reviewed.

[0054] The present invention realizes the perception and adaptation of the resource status in the current cloud computing environment by recording the memory identification, execution path and cache address of the processing node. The binding relationship between the resource status and the task number is used for the path positioning and execution scheduling of the subsequent archiving process. In scenarios such as dynamic node allocation and task migration that may occur in the cloud platform, the archiving task can be continued and completed stably based on the recorded resource status data, ensuring the recovery capability after task interruption, thereby improving the environmental robustness and stability of the entire system.

[0055] The present invention implements logically parallel processing by performing field mapping and time segmentation operations on structured field groups and unstructured content blocks respectively, and then normalizes them and uniformly constructs them into a two-layer archiving component. This parallel-first, fusion-later processing flow has higher processing efficiency than the traditional serial data integration method, and avoids the problem of mutual interference between structured data and text content in traditional methods.

[0056] The present invention integrates the structure reference table and content time flow with the state anchor data to generate a two-layer archiving component including a data component flow and an operation link set, breaking the design limitation of the traditional archiving system that only archives data but not the process, and realizes the integration of content archiving and process archiving. The archiving component can be stored as a standard data package in the cloud and supports parsing, verification and reconstruction by other archiving management platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flowchart of a multimodal data archiving and processing system based on cloud computing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] like Figure 1As shown, an embodiment of the present invention provides a multimodal data archiving and processing system based on cloud computing, the system comprising:

[0060] The configuration module is used to obtain the archiving request and the archiving task, and determine the archiving type, processing priority and task number according to the archiving request and the task number, and obtain the task configuration data;

[0061] The resource module is used to record the memory identification, execution path and cache address of the processing node before the archiving task is started, and obtain the resource status data set;

[0062] A processing module is used to extract structured field groups and unstructured content blocks based on task configuration data, generate a structure reference table by mapping structured field groups by fields, and generate a content time stream by segmenting unstructured content blocks in chronological order;

[0063] The recording module is used to record each field mapping and time segmentation operation, obtain operation event data, and determine its reference relationship with the processing node to obtain process link data;

[0064] The synchronization module is used to match the resource status data set with the processing nodes in the process link data, determine the task number, resource status and data fragment of each processing node, and obtain the state anchor data;

[0065] an execution module for combining the structure reference table and the content time stream with the state anchor data into a two-layer archive component, wherein the two-layer archive component includes a data component stream and an operation link set;

[0066] The archiving module is used to archive the double-layer archiving components and generate process index data.

[0067] In an embodiment of the present invention, a configuration module is used to obtain archiving requests and archiving tasks, and determine the archiving type, processing priority and task number based on them, obtain task configuration data, clarify task execution strategy and priority, avoid resource competition and processing delays, and provide a basis for subsequent task scheduling; a resource module is used to record the memory identification, execution path and cache address of the processing node before the archiving task is started, obtain a resource status data set, establish a resource mapping between the archiving task and the operating environment, and provide a basis for subsequent data processing operations; a processing module is used to extract structured field groups and unstructured content blocks according to task configuration data, and generate a structure reference table by mapping the structured field groups according to the fields, divide the unstructured content blocks in chronological order to generate a content time stream, and achieve regular format and clear logic within the data type through field hierarchical mapping and time series organization, thereby solving the problem of mixed structured content and unstructured content and lack of unified processing in the prior art.

[0068] The recording module is used to record each field mapping and time segmentation operation, obtain operation event data, and determine its reference relationship with the processing node to obtain process link data, thereby realizing the traceability of each step of the operation in the archiving task processing process and enhancing the traceability of data processing; the synchronization module is used to match the resource status data set with the processing node in the process link data, determine the task number, resource status and data fragment of each processing node, obtain state anchor data, ensure the correspondence between the archive content and the execution status, and provide a basis for subsequent reconstruction tasks; the execution module is used to combine the structure reference table and the content time flow with the state anchor data into a double-layer archiving component. The double-layer archiving component includes a data component flow and an operation link set, which not only retains the structural content, but also retains the generation process, and has dual evidence storage capabilities of content and process; the archiving module is used to archive the double-layer archiving component and generate process index data to ensure that the archived data can be completely restored, verified and analyzed.

[0069] The archiving module is used to archive the double-layer archiving components and generate process index data, specifically including:

[0070] First, the system receives the two-layer archive component output by the execution module, which includes a data component stream and a set of operation links. The system uses this two-layer archive component as the archive input unit and binds each component to a corresponding task number, data component type identifier, and generation timestamp. This binding operation is completed through the archive configuration table, which is initialized based on the archive type and task number recorded in the task configuration data and is used to manage the metadata structure of the archive output. Each data component unit in the data component stream contains a four-tuple of field path, paragraph index, processing node number, and memory address. The system generates a unique storage reference identifier for each unit through a hash pointer mechanism and writes it to the archive record index table.

[0071] After completing the index writing of the data component stream, the system proceeds to process the operation link set. The operation link set consists of a set of linked lists sorted by operation number. Each link represents the operation execution sequence of an archived task and includes fields such as the chain head identifier, event number, processing node number, task number, and chain tail identifier. After receiving the operation link set, the archiving module first categorizes and archives the linked lists by task number and assigns a unique sequence identifier based on the linked list structure. The system then compares the event number in each link with the operation event data table, extracting the associated field values, field paths, and slice anchor information to form an operation tracking node set. In this node set, each node contains the operation source path, node resource location, corresponding data component unit index value, and operation timestamp. The system combines this node set with the original task number to generate a process index entry, which is stored in the process index table as the process-level index information for the archive package.

[0072] The process index data is ultimately exported by the archiving module to the archive metadata repository, where it forms a complete archive package along with the two-layer archive components. The system supports unified identification, compression, encryption, and upload of archive packages to a designated cloud storage destination. The archive index, including the task number, archive type, archive component summary, and generation time, is registered in the metadata management platform.

[0073] In a preferred embodiment of the present invention, the configuration module includes:

[0074] A type determination unit is configured to receive an archiving request, match the type field of the archiving task with the archiving type list, obtain a matching item, and determine the archiving type when the matching item is greater than a preset matching threshold;

[0075] The path judgment unit is used to perform hierarchical expansion on the path field of the archiving task, divide the path characters into multiple path segments using the hierarchical separator, and judge whether each path segment meets the requirements according to the path pattern list. If the result is yes, it marks it as a valid path segment and combines the valid path segments into an archiving path structure;

[0076] The merging unit is used to merge the archiving type and the archiving path structure to obtain the archiving source data identification item and assign an archiving task number to it;

[0077] The priority unit is used to generate a processing priority according to the archiving task and the task queue it is in, and bind it with the archiving task number to obtain task configuration data.

[0078] In an embodiment of the present invention, a type judgment unit is used to receive an archiving request, match and compare the type field of the archiving task with the archiving type list to obtain a matching item. When the matching item is greater than a preset matching threshold, the archiving type is determined, thereby avoiding efficiency limitations caused by manual selection and effectively solving the problems of unclear archiving types and inconsistent classification standards in traditional archiving methods. A path judgment unit is used to hierarchically expand the path field of the archiving task, divide the path characters into multiple path segments with hierarchical separators, and judge whether each path segment meets the requirements according to the path pattern list. If the result is yes, it is marked as a valid path segment, and the valid path segments are combined into The archiving path structure avoids problems such as path confusion and hierarchical errors, and provides a clear path benchmark for resource scheduling and index establishment of subsequent archiving tasks; the merging unit is used to merge the archiving type with the archiving path structure to obtain the archiving source data identification item and assign it an archiving task number, building a unique identification mechanism for the task level in the archiving system, which is convenient for task tracking, resource binding and data structure association; the priority unit is used to generate processing priorities based on the archiving tasks and their task queues, and bind them to the archiving task numbers to obtain task configuration data, thereby realizing reasonable allocation of resources and effectively avoiding resource blocking and delays in critical tasks.

[0079] The priority unit is used to generate a processing priority based on the archiving task and its task queue, and bind it with the archiving task number to obtain task configuration data, which specifically includes:

[0080] According to the waiting time of all tasks in the archiving queue where the archiving task is located, it is normalized by the Sigmoid function to ensure that the tasks with longer waiting time are more likely to be scheduled first, and the task waiting score item is obtained; according to the proportional relationship between the current system resource idleness and the minimum resource requirement of the task, when resources are sufficient, tasks with high resource usage tend to be processed in advance, and the resource adaptability score item is obtained; according to the hierarchical depth of the field path and the length of the corresponding paragraph, a product logarithm function is constructed to quantify the difficulty of archiving data processing, ensure that complex tasks obtain reasonable scheduling priority, and obtain the data complexity score item; The service waiting score, resource adaptability score, and data complexity score are weighted and summed to determine the positive driver of the archiving task processing priority, resulting in a value score. The clarity of the archiving task's path structure is calculated based on the proportion of valid path fields, resulting in a path structure confidence penalty. The processing priority is determined based on the ratio of the value score to the path structure confidence penalty. When the proportion of path segments marked as valid in the archiving task is high, the penalty approaches 1, with no significant weakening effect on the processing priority. When the path segment structure is incomplete or matching fails, the ratio decreases, causing the denominator to increase, thereby weakening the task priority. To achieve consistent references across modules in the task processing chain, after the processing priority is generated, the archiving task number is called and the priority tag is bound to the archiving task number one by one to form a complete task configuration data structure.

[0081] The calculation formula for the processing priority is:

[0082] ,

[0083] in, The processing priority of the archiving task, The waiting time for archiving tasks. is the average waiting time of all archiving tasks in the archiving queue, is the minimum waiting time in the archive queue, is the amount of idle resources on the current cloud platform, is the minimum resource requirement for archiving tasks, is the number of path fields in the archiving task, is the index of the path field, Path field in the archive task level, The maximum level of the path field in the archiving task. Path field in the archive task length, The minimum length of the path field in the archiving task. is the number of valid path segments in the archiving task, is the coefficient.

[0084] in, is the weight coefficient, and its sum is 1.

[0085] In the financial data archiving scenario, most tasks involve high-frequency and high-density information such as transaction records, audit logs, and settlement vouchers. Financial systems are highly sensitive to processing delays, and task waiting time must be an important consideration in scheduling. However, since such systems usually run in dedicated environments with sufficient resource guarantees, the risk of resource shortages is low, and resource adaptation has a relatively small impact on task scheduling. On the other hand, archiving tasks are usually based on simple fields and shallow structures, and the complexity does not significantly require processing power, so the weight of this item can be appropriately reduced. Therefore, the task waiting score should be significantly larger, the resource adaptability score should be moderate, and the data complexity score should be smaller. The values ​​are 0.6, 0.3 and 0.1 respectively.

[0086] In the scenario of enterprise operational data archiving, task types include system call logs, user behavior records, monitoring status updates, etc. The biggest bottleneck in archiving this type of data is usually resource scheduling rather than waiting time, because archiving tasks are often triggered in batches, and system resource constraints become a key factor in scheduling and sorting. On the other hand, the field structure complexity of such tasks varies greatly, including nested objects, dynamic fields, etc. The processing cost is significantly affected by the content, so the complexity item also needs to be considered. However, since the enterprise system focuses on overall throughput efficiency, although queuing time is meaningful, its impact on priority sorting is less than that of resources and content. Therefore, the resource adaptability score item is the largest, followed by the data complexity score item, and the task waiting score item. The values ​​are 0.2, 0.5 and 0.3 respectively.

[0087] In the intelligent manufacturing data archiving scenario, archiving tasks come from multimodal sources such as equipment operation data, sensor readings, image acquisition information, etc., and are often accompanied by production control systems with extremely high real-time requirements. In this environment, a slightly longer waiting time in the queue may cause synchronization failure, seriously affecting the correctness of task scheduling, so the waiting time item must be treated as the dominant item; but at the same time, manufacturing scenarios mostly use edge computing nodes or resource-constrained terminals, and the resource status is highly unstable. The importance of resource adaptability and waiting time is almost equivalent; in addition, a large number of fields in the archived content have deep hierarchical structures (such as multi-channel data streams, periodic signals, etc.), and the complexity will significantly increase the processing load. Therefore, the difference in the weights of the three items should not be too large, but generally the task waiting score item and resource adaptability score item are dominant. The values ​​are 0.4, 0.45 and 0.15 respectively.

[0088] in, is the penalty factor coefficient, and its value is 1.0. This value is set after quantitative analysis of the relationship between path confidence and priority. In the archiving task scheduling process, the structural integrity of the path field directly affects whether the task can be accurately identified and processed. If there are many invalid structures in the path segment, it will lead to the failure of subsequent field mapping, resource anchoring and data component generation. Considering that when the path field confidence is insufficient, the task execution opportunity should not be directly deprived, but its priority should still be moderately suppressed, so It is fixedly set to 1.0, so that when the proportion of valid paths is insufficient, the overall decrease in priority is linear, which is neither excessively penalizing nor sufficient to distinguish tasks with clear structures from tasks with fuzzy structures.

[0089] In a preferred embodiment of the present invention, the resource module includes:

[0090] The node screening unit is used to call the cloud platform resource management interface according to the archived task number, obtain the list of currently available processing nodes, and screen the processing nodes whose idleness and computing power meet the minimum execution standards to obtain the node candidate set;

[0091] The resource status unit is used to request the resource status of each processing node in the node candidate set, call the node interface to extract the memory identification, execution path and cache address of the processing node, and obtain the resource status characteristics;

[0092] The binding unit is used to group the resource status features into a triple structure and bind it with the corresponding archive task number to obtain a resource status mapping item;

[0093] The number grouping unit is used to group the resource status mapping items according to the archiving task numbers and assign resource snapshot numbers to obtain a resource status data set.

[0094] In an embodiment of the present invention, a node screening unit is used to call the cloud platform resource management interface according to the archiving task number, obtain a list of currently available processing nodes, and screen out processing nodes whose idleness and computing power meet the minimum execution standard to obtain a node candidate set, thereby avoiding delays and failures caused by insufficient resources or scheduling conflicts during the task processing; a resource status unit is used to make a resource status request for each processing node in the node candidate set, call the node interface to extract the memory identifier, execution path and cache address of the processing node, and obtain resource status characteristics, so that the system can clearly understand in which computing environment the current archive will run; a binding unit is used to form the resource status characteristics into a triple structure, and bind it to the corresponding archiving task number to obtain a resource status mapping item, thereby realizing the task's unique indexing capability for node resources and providing a basis for subsequent field path positioning of processing nodes; a number grouping unit is used to group the resource status mapping items according to the archiving task number and assign resource snapshot numbers to obtain a resource status data set, which is not only convenient for subsequent execution module calls and reproduction, but also convenient for result verification and archiving after task completion.

[0095] The node screening unit is used to call the cloud platform resource management interface according to the archived task number to obtain a list of currently available processing nodes, and screen out processing nodes whose idleness and computing power meet the minimum execution standards to obtain a node candidate set, which specifically includes:

[0096] The system automatically initiates the node screening process. Based on the received archived task number, the node screening unit sends a query request to the resource control center via a standard cloud platform API (e.g., a resource management API that supports the RESTful protocol). This query invokes the resource status synchronization module to obtain a list of processing nodes in the cloud computing platform that are currently "idle," "partially idle," or "schedulable." Each entry in this processing node list contains basic information such as the node's unique identification code, number of CPU cores, current CPU load, total memory capacity, used memory, whether GPU resources are bound, GPU utilization, network latency score, and task execution history. This information is returned in a structured format for subsequent screening.

[0097] The system will then select a processing node from the candidate node list that meets the following two constraints based on the processing priority corresponding to the task: First, the node's current idleness must be greater than or equal to the preset scheduling threshold. The idleness is calculated as: (1-current CPU load / total number of cores) x 100%; Second, the node's computing power must meet the minimum standard for task execution. This minimum standard is the configuration lower limit of the resource requirements required by the preset model of the task, including minimum memory capacity (for example, not less than 16GB), whether the GPU is available, and whether the network I / O meets the bandwidth threshold (for example, greater than 500Mbps).

[0098] The number grouping unit is used to group the resource status mapping items according to the archiving task number and assign resource snapshot numbers to obtain a resource status data set, which specifically includes:

[0099] First, the system traverses all current resource status mapping entries and logically groups them by task number, based on the archived task number field contained in each mapping entry. This ensures that the resource status mapping entries in each group belong to the same archived task. This grouping operation is implemented using a HashMap structure, with the task number as the primary key and the resource status mapping entry list as the value set.

[0100] After completing the logical grouping of resource status mapping items, the number grouping unit generates a unique resource snapshot number for each resource status group under each task number. This resource snapshot number is a sortable, non-repeatable timestamp number (e.g., "RS20250614T154500001") generated within the system and is used to identify a snapshot of the resource environment at the time of task execution. This number not only uniquely identifies the resource binding status of a batch but also allows for reverse lookup and status reuse during task execution, ensuring consistency between task and resource scheduling.

[0101] In a preferred embodiment of the present invention, the processing module includes:

[0102] The path parsing unit is used to extract the structured field group from the archiving task, and perform path parsing on each structured field according to the field path, extract the path label of each level, obtain the field path stack, and set the last level field value as the field end value to obtain a path value pair;

[0103] A sub-segment mapping unit is used to detect the path-value pair, determine the sub-segment type of the field value, and call the corresponding field mapping structure according to the sub-segment type to obtain a structure reference table, wherein the structure reference table includes a path hierarchical structure, a field type label, and a standard field code;

[0104] The time partitioning unit is used to extract unstructured content blocks from the archiving task, sort them in chronological order, and then partition them into fixed time segments and insert anchor points at the segmentation points to obtain content slices;

[0105] The sorting unit is used to arrange each content slice in sequence to obtain a content time stream, wherein the content time stream includes the start and end time of the paragraph, the number of words in the paragraph, the anchor field path and the paragraph index value.

[0106] In an embodiment of the present invention, a path parsing unit is used to extract a structured field group from an archiving task, and perform path parsing on each structured field according to a field path, extract each level of path labels, obtain a field path stack, and set the last level field value as the field end value to obtain a path value pair, which can convert a traditional flat data structure into a tree-like field path stack, so that the relationship between the fields is clearly expressed, laying a structural foundation for subsequent field mapping and comparison; a sub-segment mapping unit is used to detect the path value pair, determine the sub-segment type of the field value, and call the corresponding field mapping structure according to it to obtain a structure reference table, which includes a path hierarchical structure, a field type label and a standard field code, unifies the expression method of heterogeneous fields, and provides a reference for subsequent field mapping. Segment anchoring and cross-task comparison provide a unified reference interface; the time division unit is used to extract unstructured content blocks from archiving tasks, sort them in chronological order, and then divide them, divide them according to fixed time lengths and insert anchor points at the division points to obtain content slices, and organize unstructured content according to the time dimension, which solves the problem of disordered and untraceable unstructured content in traditional archiving, and avoids resource waste or processing exceptions caused by a single data block that is too large or too small; the sorting unit is used to arrange each content slice in sequence to obtain a content time stream, which includes the start and end time of the paragraph, the number of words in the paragraph, the anchor field path and the paragraph index value, which not only realizes the temporal logic control of data archiving, but also provides a basis for subsequent data binding and event indexing.

[0107] The path parsing unit is used to extract the structured field group from the archiving task, and perform path parsing on each structured field according to the field path, extract the path label of each level, obtain the field path stack, and set the last level field value as the field end value to obtain the path value pair, which specifically includes:

[0108] A list of structured field groups is read from the data structure of the archiving task. For each structured field in the list, a key-value pair of "path string-value" is used to extract the data. The path string is a combination of hierarchical labels separated by " / ", such as " / patient / record / bloodType". The system splits the path string with " / " as the delimiter, extracts each layer of path labels step by step, and pushes them into the field path stack in order. At the same time, the final field value corresponding to the field is extracted and set as the field endpoint value. Subsequently, the system assembles the path stack and the field endpoint value into a path value pair structure. The path value pair includes: path label list, field hierarchical depth, endpoint value and path unique index identifier, which is used for unique positioning and logical binding of subsequent fields.

[0109] The sub-segment mapping unit is used to detect the path-value pair, determine the sub-segment type of the field value, and call the corresponding field mapping structure based on it to obtain a structure reference table. The structure reference table includes a path hierarchical structure, field type labels, and standard field codes, specifically including:

[0110] The system performs a field value type identification process on the endpoint value in a path-value pair. Based on a preset set of field value discrimination rules, the system identifies the corresponding sub-segment type of the field value through regular expression matching, literal type analysis, and contextual statistical analysis. These methods include, but are not limited to, numeric types (integer, float), text types (string, text block), enumeration types (enum), and time types (timestamp). Once the sub-segment type identification is complete, the system searches the structured field standard mapping library for the corresponding field mapping structure. This mapping structure maps each field value type to the corresponding standard path template, field type label (such as "identification field" and "numeric field"), and a globally unique standard field code. Based on this, the system constructs a structure reference table. The structure reference table is a multi-field structure record table, with each record corresponding to a path-value pair. Its fields include: full path hierarchy information (i.e., the label combination from the root path to the endpoint path), field type label (used to assist in semantic indexing and extraction), and standard field code (which serves as the basic identifier for cross-task and cross-system data comparison). This structure reference table serves as the key semantic interface for subsequent field binding and structure anchoring.

Claims

1. A multimodal data archiving and processing system based on cloud computing, characterized in that: The system comprises: The configuration module is used to obtain the archiving request and the archiving task, and determine the archiving type, processing priority and task number according to the archiving request and the task number, and obtain the task configuration data; The resource module is used to record the memory identification, execution path and cache address of the processing node before the archiving task is started, and obtain the resource status data set; A processing module is used to extract structured field groups and unstructured content blocks based on task configuration data, generate a structure reference table by mapping structured field groups by fields, and generate a content time stream by segmenting unstructured content blocks in chronological order; The recording module is used to record each field mapping and time segmentation operation, obtain operation event data, and determine its reference relationship with the processing node to obtain process link data; The synchronization module is used to match the resource status data set with the processing nodes in the process link data, determine the task number, resource status and data fragment of each processing node, and obtain the state anchor data; an execution module for combining the structure reference table and the content time stream with the state anchor data into a two-layer archive component, wherein the two-layer archive component includes a data component stream and an operation link set; The archiving module is used to archive the double-layer archiving components and generate process index data.

2. The multimodal data archiving and processing system based on cloud computing according to claim 1, characterized in that: The configuration module includes: A type determination unit is configured to receive an archiving request, match the type field of the archiving task with the archiving type list, obtain a matching item, and determine the archiving type when the matching item is greater than a preset matching threshold; The path judgment unit is used to perform hierarchical expansion on the path field of the archiving task, divide the path characters into multiple path segments using the hierarchical separator, and judge whether each path segment meets the requirements according to the path pattern list. If the result is yes, it marks it as a valid path segment and combines the valid path segments into an archiving path structure; The merging unit is used to merge the archiving type and the archiving path structure to obtain the archiving source data identification item and assign an archiving task number to it; The priority unit is used to generate a processing priority according to the archiving task and the task queue it is in, and bind it with the archiving task number to obtain task configuration data.

3. The multimodal data archiving and processing system based on cloud computing according to claim 2, characterized in that: The resource module includes: The node screening unit is used to call the cloud platform resource management interface according to the archived task number, obtain the list of currently available processing nodes, and screen the processing nodes whose idleness and computing power meet the minimum execution standards to obtain the node candidate set; The resource status unit is used to request the resource status of each processing node in the node candidate set, call the node interface to extract the memory identification, execution path and cache address of the processing node, and obtain the resource status characteristics; The binding unit is used to group the resource status features into a triple structure and bind it with the corresponding archive task number to obtain a resource status mapping item; The number grouping unit is used to group the resource status mapping items according to the archiving task numbers and assign resource snapshot numbers to obtain a resource status data set.

4. The multimodal data archiving and processing system based on cloud computing according to claim 3, characterized in that: The processing module includes: The path parsing unit is used to extract the structured field group from the archiving task, and perform path parsing on each structured field according to the field path, extract the path label of each level, obtain the field path stack, and set the last level field value as the field end value to obtain a path value pair; A sub-segment mapping unit is used to detect the path-value pair, determine the sub-segment type of the field value, and call the corresponding field mapping structure according to the sub-segment type to obtain a structure reference table, wherein the structure reference table includes a path hierarchical structure, a field type label, and a standard field code; The time partitioning unit is used to extract unstructured content blocks from the archiving task, sort them in chronological order, and then partition them into fixed time segments and insert anchor points at the segmentation points to obtain content slices; The sorting unit is used to arrange each content slice in sequence to obtain a content time stream, wherein the content time stream includes the start and end time of the paragraph, the number of words in the paragraph, the anchor field path and the paragraph index value.

5. The multimodal data archiving and processing system based on cloud computing according to claim 4, characterized in that: The recording module includes: The field operation unit is used to monitor each field mapping operation in the generation of the structure reference table, seal the path label, field type and sub-segment mapping structure during path resolution, and record the task number of the operation to obtain the field operation data; The paragraph operation unit is used to track each time slice operation in the content time stream generation, classify and identify the anchor point of each content slice, and record the paragraph index, paragraph word count and segmentation number to obtain paragraph operation data; A combining unit is used to combine and sort the field operation data and the paragraph operation data in the order of their recording time, and add the operation number to them to obtain the operation event data set; The node mapping unit is used to query the processing node number of each operation event in the operation event data set in the resource status data set, determine the mapping relationship, and obtain process link data.

6. The multimodal data archiving and processing system based on cloud computing according to claim 5, characterized in that: The synchronization module includes: The resource matching unit is used to extract the processing node number of each operation event in the process link data, and query the resource status data set for a triple structure with the same processing node number and task number to obtain a matching resource group; A state binding unit is used to use the memory identifier, execution path, and cache address of the matching resource group as state parameters, and associate them with the path label or paragraph index in the operation event to obtain a state binding item; The annotation merging unit is used to annotate the task number and data fragment of the state-bound item, generate the state-anchored sub-item, and merge all the state-anchored sub-items into state-anchored data according to the task number.

7. The multimodal data archiving and processing system based on cloud computing according to claim 6, characterized in that: The execution module includes: A field anchoring unit is used to retrieve the processing node number and memory address corresponding to each field path label in the structure reference table from the state anchoring data according to the path label of the path label, and to construct a field anchor pair; The field binding unit is used to locate the co-occurrence position of the paragraph index and the field path in the field anchor pair in the content time stream, determine the logical relationship between the paragraph index and the field path, and construct the field binding item; a combining unit, configured to combine the processing node number and memory address in the field anchor pair with the field path and paragraph index in the field binding item into a four-element index block, and append a task number thereto to obtain a data component unit; The sorting and grouping unit is used to sort the data component units according to the hierarchical structure of the field path, and to archive and group them according to the task number to obtain the data component flow.

8. The multimodal data archiving and processing system based on cloud computing according to claim 7, characterized in that: The execution module also includes: The operation node unit is used to extract the operation number, processing node number and task number of each operation event from the state anchor data to obtain the operation node set; An operation sequence unit is used to sort the processing nodes in the operation node set in ascending order according to the event number, and group them based on the task number to obtain a task operation sequence; The event linked list unit is used to read the event number and data fragment one by one in the task operation sequence, and establish a one-way event connection pointer between two adjacent operation events to obtain the event linked list data; The identifier merging unit is used to add a chain head anchor identifier and a chain tail end identifier to each event chain list, construct a task operation link entry, and combine all task operation link entries to obtain an operation link set.

9. The multimodal data archiving and processing system based on cloud computing according to claim 8, characterized in that: The archiving type list includes a plurality of archiving type items, each archiving type item includes a type number field, a keyword set field, and an archiving template identification field, wherein the keyword set field includes a number of matching keywords related to the archiving type. When determining the archiving type, keywords are extracted from the type field in the archiving task and compared with the keyword set field. When the number of matching keywords exceeds a preset matching threshold, the archiving type item is determined to be the archiving type of the archiving task. The path pattern list includes multiple path template items, each path template item includes a template number field, a path format field, a maximum level field and a wildcard allowed field, wherein the path format field is a legal path structure format, containing one or more levels of path segments, and the path segments are divided by separators. When the path field in the archiving task is expanded into multiple path segments, the number and content of the path segments are judged item by item to determine whether they comply with the path format field and the maximum level field. When they comply with the path template item, the path segment is marked as a valid path segment.

10. The multimodal data archiving and processing system based on cloud computing according to claim 9, characterized in that: The data component stream is used to perform content backtracking of the archiving task during the archiving process; the operation link set is used to perform operation backtracking of the archiving task during the archiving process.

Images