Heterogeneous storage switching method, system, medium and electronic device for multimodal data

Abstract

The present invention provides a heterogeneous storage switching method, system, medium and electronic device for multimodal data, including a sub-storage system acquiring a multimodal service data packet and digitally signing the multimodal service data packet, packaging the digital signature and the multimodal service data packet and storing them in the cache of the current sub-storage system; after the main storage system is started, it starts to acquire the multimodal service data packet, and according to the established data protocol, it fixedly or dynamically parses the multimodal service data packet, extracts each modal data, and stores them by category according to the type of data; the main storage system processes all cached data packets in the sub-storage system, and stores them by category according to the type of data. The present invention solves the problem that the network multimodal service data cannot be recorded during the power-on startup of the system due to the slow startup of the main storage system, thereby achieving the effect of not losing important data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a heterogeneous storage switching method, system, medium and electronic device for multimodal data. Background Art

[0002] Currently, the demand for multimodal business data storage is growing. Such systems exhibit characteristics such as data heterogeneity, high concurrent processing capabilities, high bandwidth traffic, and classified storage of multiple types of structured data. In the face of these emerging storage challenges, the designed storage systems are often complex and equipped with full-featured operating systems. However, this also leads to the problem of long storage system startup time, which in turn causes the problem that business data cannot be processed and classified during system startup in a timely manner.

[0003] Patent document CN103294574A discloses a device and method for quickly implementing business storage switching, which is enabled and used when a back-end storage fails. The device includes local storage resources, and the method includes: calculating the length of the current remaining cache space; judging whether to perform cache supplement based on the calculated current remaining cache space length and the current length of business data to be written. If the current remaining cache space is less than the current length of business data to be written, cache supplement is performed: the unfinished back-end storage business data in the cache and the current business data to be written are written into the local storage resource; if the current remaining cache space is larger than the current business data to be written, cache supplement is not performed: the current business data to be written is written into the cache.

[0004] However, although patent document CN103294574A can promptly start the local cache re-recording function when a storage system failure occurs, it cannot guarantee that business data will not be lost during the system startup process.

[0005] Patent document CN114661832A discloses a multimodal heterogeneous data storage method and system based on data quality, comprising the following steps: 1) distributing and storing original text data in the original database in key-value format; 2) performing data modeling on original multimedia data and distributing and storing them in the file database in the form of files; 3) converting key-value data into relational data and building a relational database; 4) building a graph database according to the relationships between entities in the relational database; 5) performing data modeling on the activity data of the entity in a chain structure and building a chain database; 6) converting multimedia data into text data and storing them in the multimedia database and the original database according to the data type; 7) linking the entity data of each sub-database by building a multi-level index structure; 8) building a log file maintenance system for a multimodal database for the data integration method and each sub-database.

[0006] However, patent document CN114661832A directly stores the entire data and cannot be classified and stored. Summary of the invention

[0007] In view of the defects in the prior art, an object of the present invention is to provide a heterogeneous storage switching method, system, medium and electronic device for multimodal data.

[0008] A heterogeneous storage switching method for multimodal data provided by the present invention includes:

[0009] Step S1: the sub-storage system obtains a multimodal service data packet and digitally signs the multimodal service data packet, and packages the digital signature and the multimodal service data packet and stores them in a cache of the current sub-storage system;

[0010] Step S2: After the main storage system is started, it starts to obtain multimodal service data packets, and according to the established data protocol, it fixedly or dynamically parses the multimodal service data packets, extracts each modal data, and stores them according to the type of data;

[0011] Step S3: The main storage system processes all cached data packets in the sub-storage system and stores them according to the types of data.

[0012] Preferably, the method comprises the following sub-steps:

[0013] Step S1.1: The sub-storage system quickly completes startup and captures multi-modal service data packets generated by the service system in real time through the network;

[0014] Step S1.2: The sub-storage system uses the MD5 algorithm to digitally sign the multimodal service data packet;

[0015] Step S1.3: The sub-storage system packages the multimodal service data packet and the digital signature and stores them in the cache of the sub-storage system;

[0016] Step S1.4: Check whether the primary storage system notifies to stop storage. If yes, the process ends; if no, return to step S1.1.

[0017] Preferably, step S2 includes the following sub-steps:

[0018] Step S2.1: The main storage system is started and the multimodal service data packets generated by the service system are captured in real time through the network;

[0019] Step S2.2: Before completing the cache data packet processing of all sub-storage systems, it is also necessary to use the MD5 algorithm to digitally sign and cache the multimodal service data packet;

[0020] Step S2.3: The main storage system parses the multimodal service data packet fixedly or dynamically according to the established data protocol, extracts each modal data, and stores the data in a classified manner according to the type of data until the system is shut down.

[0021] Preferably, step S3 includes the following sub-steps:

[0022] Step S3.1: The main storage system notifies the sub-storage system to stop real-time capture of multimodal service data packets generated by the storage service system;

[0023] Step S3.2: The main storage system sequentially reads the previously cached multimodal service data packets and digital signature packets from the sub-storage systems;

[0024] Step S3.3: The main storage system compares the above digital signature with the digital signature saved in the local cache in step S2.2. If there is no hit, it means that the multimodal business data packet corresponding to the digital signature is the business data in the startup time window, then the above multimodal business data packet is parsed and classified and stored according to the data packet type; if a hit record is found in the digital signature comparison, it means that the remaining unprocessed data in the sub-storage system has overlapped with the main storage system, then the cache data packet processing of the sub-storage system is completed.

[0025] A heterogeneous storage switching system for multimodal data provided by the present invention includes:

[0026] Module M1: The sub-storage system obtains a multimodal service data packet and digitally signs the multimodal service data packet, and packages the digital signature and the multimodal service data packet and stores them in a cache of the current sub-storage system;

[0027] Module M2: After the main storage system is started, it starts to obtain multimodal service data packets, and according to the established data protocol, it fixedly or dynamically parses the multimodal service data packets, extracts each modal data, and stores them according to the types of data;

[0028] Module M3: The main storage system processes all cached data packets in the sub-storage system and stores them according to the types of data.

[0029] Preferably, the method comprises the following sub-steps:

[0030] Module M1.1: The sub-storage system quickly starts up and captures multi-modal service data packets generated by the service system in real time through the network;

[0031] Module M1.2: The sub-storage system uses the MD5 algorithm to digitally sign the multimodal service data packet;

[0032] Module M1.3: The sub-storage system packages the multimodal service data packet and the digital signature and stores them in the cache of the sub-storage system;

[0033] Module M1.4: Whether the main storage system notifies to stop storage, if so, the process ends; if not, return to module M1.1.

[0034] Preferably, the module M2 comprises the following sub-steps:

[0035] Module M2.1: The main storage system is started and the multi-modal service data packets generated by the service system are captured in real time through the network;

[0036] Module M2.2: Before completing the cache data packet processing of all sub-storage systems, it is also necessary to use the MD5 algorithm to digitally sign and cache the multimodal service data packets;

[0037] Module M2.3: The main storage system parses the multimodal service data packet fixedly or dynamically according to the established data protocol, extracts the data of each modality, and stores the data by classification according to the type of data until the system is shut down.

[0038] Preferably, the module M3 comprises the following sub-steps:

[0039] Module M3.1: The main storage system notifies the sub-storage system to stop real-time capture of multi-modal service data packets generated by the storage service system;

[0040] Module M3.2: The main storage system sequentially reads the previously cached multimodal service data packets and digital signature packets from the sub-storage systems;

[0041] Module M3.3: The main storage system compares the above digital signature with the digital signature stored in the local cache from module M2.2. If there is no hit, it means that the multimodal business data packet corresponding to the digital signature is the business data in the startup time window, then the above multimodal business data packet is parsed and classified and stored according to the data packet type; if a hit record is found in the digital signature comparison, it means that the remaining unprocessed data in the sub-storage system has overlapped with the main storage system, then the cache data packet processing of the sub-storage system is completed.

[0042] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the heterogeneous storage switching method for multimodal data are implemented.

[0043] An electronic device provided according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the heterogeneous storage switching method for multimodal data are implemented.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The present invention, when the main storage system is started, caches the network multimodal service data packets that cannot be normally stored within the time window to a sub-storage system (implemented by FPGA) that can be quickly started, and reads them out for processing when the main storage system is started. This solves the problem that the network multimodal service data cannot be recorded during the power-on startup of the system due to the slow startup of the main storage system, thereby achieving the effect of not losing important data.

[0046] 2. The present invention uses switch port mirroring or a dedicated TAP device to be deployed at the front end of the master / sub storage system, thereby replicating network multimodal service data packets. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0048] Figure 1 It is a schematic diagram of the working method of the present invention;

[0049] Figure 2 It is a schematic diagram of the multimodal data processing flow of the main storage system in the present invention;

[0050] Figure 3 A schematic diagram of the process flow of the main storage system processing all cached data packets in the sub-storage system in the present invention;

[0051] Figure 4 It is a schematic diagram of the multimodal data processing flow of the neutron storage system of the present invention. DETAILED DESCRIPTION

[0052] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0053] When storing high-speed network multimodal service data, the present invention utilizes the fast startup capability of FPGA to store multimodal service data that cannot be normally stored within the startup time window of the main storage system in a sub-storage system controlled by FPGA, and reads out and processes the data when the main storage system is started. The present invention connects the main and sub-storage systems to a network switch or a dedicated network TAP device with a port mirroring function, thereby enabling the main / sub-storage system to simultaneously obtain network multimodal service data packets.

[0054] According to a heterogeneous storage switching method for multimodal data provided by the present invention, the entire system is powered on and started, including a business system, a main / sub storage system, such as Figure 1 As shown, the heterogeneous storage switching method includes:

[0055] Step S1: The sub-storage system obtains a multimodal service data packet and digitally signs the multimodal service data packet, and packages the digital signature and the multimodal service data packet and stores them in the cache of the current sub-storage system. If the main storage system does not notify to stop storage, the sub-storage system will continue to perform the storage action in step S1 until the main storage system notifies to stop storage. Specifically, Figure 4 As shown, it includes the following sub-steps:

[0056] Step S1.1: The sub-storage system quickly completes startup and captures multi-modal service data packets generated by the service system in real time through the network.

[0057] Step S1.2: The sub-storage system uses the MD5 algorithm to digitally sign the multimodal service data packet.

[0058] Step S1.3: The sub-storage system packages the multimodal service data packet and the digital signature and stores them in the cache of the sub-storage system.

[0059] Step S1.4: Check whether the primary storage system notifies to stop storage. If yes, the process ends; if no, return to step S1.1.

[0060] Step S2: After the main storage system is started, it starts to obtain multimodal service data packets, and according to the established data protocol, it fixedly or dynamically parses the multimodal service data packets, extracts each modal data, and stores them according to the type of data. Figure 2 As shown, it includes the following sub-steps:

[0061] Step S2.1: The main storage system is started up and the multi-modal service data packets generated by the service system are captured in real time through the network.

[0062] Step S2.2: Before completing the processing of cached data packets in all sub-storage systems, it is necessary to use the MD5 algorithm to digitally sign and cache the multimodal service data packets.

[0063] Step S2.3: The main storage system parses the multimodal service data packet fixedly or dynamically according to the established data protocol, extracts each modal data, and stores the data in a classified manner according to the type of data until the system is shut down.

[0064] Step S3: The main storage system processes all cached data packets in the sub-storage system. Specifically, Figure 3As shown, it includes the following sub-steps:

[0065] Step S3.1: The primary storage system notifies the secondary storage system to stop real-time capture of multimodal service data packets generated by the storage service system.

[0066] Step S3.2: The main storage system sequentially reads the previously cached multimodal service data packets and digital signature packets from the sub-storage systems.

[0067] Step S3.3: The primary storage system compares the digital signature with the digital signature stored in the local cache in step S2.2.

[0068] Step S3.4: If no match is found, it indicates that the multimodal service data packet corresponding to the digital signature is service data in the startup time window, then the multimodal service data packet is parsed and classified and stored according to the data packet type;

[0069] Step S3.5: If the main storage system finds a hit record in the above digital signature comparison, it means that the remaining unprocessed data in the sub-storage system has overlapped with the main storage system, and the cache data packet processing of the sub-storage system is completed.

[0070] The present invention also provides a heterogeneous storage switching system for multimodal data. The heterogeneous storage switching system for multimodal data can be implemented by executing the process steps of the heterogeneous storage switching method for multimodal data, that is, those skilled in the art can understand the heterogeneous storage switching method for multimodal data as a preferred implementation of the heterogeneous storage switching system for multimodal data.

[0071] A heterogeneous storage switching system for multimodal data provided by the present invention includes:

[0072] Module M1: The sub-storage system obtains a multimodal service data packet and digitally signs the multimodal service data packet, and packages the digital signature and the multimodal service data packet and stores them in the cache of the current sub-storage system. The module M1 includes the following sub-steps: Module M1.1: The sub-storage system quickly completes the startup, and captures the multimodal service data packet generated by the service system in real time through the network. Module M1.2: The sub-storage system uses the MD5 algorithm to digitally sign the multimodal service data packet. Module M1.3: The sub-storage system packages the multimodal service data packet and the digital signature and stores them in the cache of the sub-storage system. Module M1.4: Whether the main storage system notifies to stop storage, if so, the process ends. If not, return to module M1.1.

[0073] Module M2: After the main storage system is started, it starts to acquire multimodal business data packets, and according to the established data protocol, it parses the multimodal business data packets fixedly or dynamically, extracts each modal data, and stores them in categories according to the type of data. The module M2 includes the following sub-steps: Module M2.1: The main storage system is started, and captures the multimodal business data packets generated by the business system in real time through the network. Module M2.2: Before completing the cache data packet processing of all sub-storage systems, it is necessary to use the MD5 algorithm to digitally sign and cache the multimodal business data packets. Module M2.3: According to the established data protocol, the main storage system parses the multimodal business data packets fixedly or dynamically, extracts each modal data, and stores them in categories according to the type of data until the system is shut down.

[0074] Module M3: The main storage system processes all cached data packets in the sub-storage system and stores them in categories according to the type of data. The module M3 includes the following sub-steps: Module M3.1: The main storage system notifies the sub-storage system to stop real-time capture of the multimodal business data packets generated by the storage business system. Module M3.2: The main storage system reads the previously cached multimodal business data packets and digital signature packets from the sub-storage system in sequence. Module M3.3: The main storage system compares the above digital signature with the digital signature stored in the local cache from module M2.2. If there is a miss, it means that the multimodal business data packet corresponding to the digital signature is the business data in the startup time window. Then, the multimodal business data packet is parsed and stored in categories according to the data packet type. If a hit record is found in the digital signature comparison, it means that the remaining unprocessed data in the sub-storage system has overlapped with the main storage system, and the cache data packet processing of the sub-storage system is completed.

[0075] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the heterogeneous storage switching method for multimodal data are implemented.

[0076] An electronic device provided according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the heterogeneous storage switching method for multimodal data are implemented.

[0077] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0078] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for switching heterogeneous storage of multimodal data, characterized in that: include: Step S1: the sub-storage system obtains a multimodal service data packet and digitally signs the multimodal service data packet, and packages the digital signature and the multimodal service data packet and stores them in a cache of the current sub-storage system; Step S2: After the main storage system is started, it starts to obtain multimodal service data packets, and according to the established data protocol, it fixedly or dynamically parses the multimodal service data packets, extracts each modal data, and stores them according to the type of data; Step S3: The main storage system processes all cached data packets in the sub-storage system and stores them according to the types of data.

2. The heterogeneous storage switching method for multimodal data according to claim 1, characterized in that: The step S1 includes the following sub-steps: Step S1.1: The sub-storage system quickly completes startup and captures multi-modal service data packets generated by the service system in real time through the network; Step S1.2: The sub-storage system uses the MD5 algorithm to digitally sign the multimodal service data packet; Step S1.3: The sub-storage system packages the multimodal service data packet and the digital signature and stores them in the cache of the sub-storage system; Step S1.4: Check whether the primary storage system notifies to stop storage. If yes, the process ends; if no, return to step S1.

1.

3. The heterogeneous storage switching method for multimodal data according to claim 1, characterized in that: The step S2 includes the following sub-steps: Step S2.1: The main storage system is started and the multimodal service data packets generated by the service system are captured in real time through the network; Step S2.2: Before completing the cache data packet processing of all sub-storage systems, it is also necessary to use the MD5 algorithm to digitally sign and cache the multimodal service data packet; Step S2.3: The main storage system parses the multimodal service data packet fixedly or dynamically according to the established data protocol, extracts each modal data, and stores the data in a classified manner according to the type of data until the system is shut down.

4. The heterogeneous storage switching method for multimodal data according to claim 1, characterized in that: The step S3 includes the following sub-steps: Step S3.1: The main storage system notifies the sub-storage system to stop real-time capture of multimodal service data packets generated by the storage service system; Step S3.2: The main storage system sequentially reads the previously cached multimodal service data packets and digital signature packets from the sub-storage systems; Step S3.3: The main storage system compares the above digital signature with the digital signature saved in the local cache in step S2.

2. If there is no hit, it means that the multimodal business data packet corresponding to the digital signature is the business data in the startup time window, then the above multimodal business data packet is parsed and classified and stored according to the data packet type; if a hit record is found in the digital signature comparison, it means that the remaining unprocessed data in the sub-storage system has overlapped with the main storage system, then the cache data packet processing of the sub-storage system is completed.

5. A heterogeneous storage switching system for multimodal data, characterized in that: include: Module M1: The sub-storage system obtains a multimodal service data packet and digitally signs the multimodal service data packet, and packages the digital signature and the multimodal service data packet and stores them in a cache of the current sub-storage system; Module M2: After the main storage system is started, it starts to obtain multimodal service data packets, and according to the established data protocol, it fixedly or dynamically parses the multimodal service data packets, extracts each modal data, and stores them according to the types of data; Module M3: The main storage system processes all cached data packets in the sub-storage system and stores them according to the types of data.

6. The heterogeneous storage switching system for multimodal data according to claim 5, characterized in that: The module M1 comprises the following sub-steps: Module M1.1: The sub-storage system quickly starts up and captures multi-modal service data packets generated by the service system in real time through the network; Module M1.2: The sub-storage system uses the MD5 algorithm to digitally sign the multimodal service data packet; Module M1.3: The sub-storage system packages the multimodal service data packet and the digital signature and stores them in the cache of the sub-storage system; Module M1.4: Whether the main storage system notifies to stop storage, if so, the process ends; if not, return to module M1.

1.

7. The heterogeneous storage switching system for multimodal data according to claim 5, characterized in that: The module M2 includes the following sub-steps: Module M2.1: The main storage system is started and the multi-modal service data packets generated by the service system are captured in real time through the network; Module M2.2: Before completing the cache data packet processing of all sub-storage systems, it is also necessary to use the MD5 algorithm to digitally sign and cache the multimodal service data packets; Module M2.3: The main storage system parses the multimodal service data packet fixedly or dynamically according to the established data protocol, extracts the data of each modality, and stores the data by classification according to the type of data until the system is shut down.

8. The heterogeneous storage switching system for multimodal data according to claim 5, characterized in that: The module M3 includes the following sub-steps: Module M3.1: The main storage system notifies the sub-storage system to stop real-time capture of multi-modal service data packets generated by the storage service system; Module M3.2: The main storage system sequentially reads the previously cached multimodal service data packets and digital signature packets from the sub-storage systems; Module M3.3: The main storage system compares the above digital signature with the digital signature stored in the local cache from module M2.

2. If there is no hit, it means that the multimodal business data packet corresponding to the digital signature is the business data in the startup time window, then the above multimodal business data packet is parsed and classified and stored according to the data packet type; if a hit record is found in the digital signature comparison, it means that the remaining unprocessed data in the sub-storage system has overlapped with the main storage system, then the cache data packet processing of the sub-storage system is completed.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the heterogeneous storage switching method for multimodal data according to any one of claims 1 to 4 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the heterogeneous storage switching method for multimodal data according to any one of claims 1 to 4 are implemented.

Images