Analog large data storage method and device, electronic equipment and storage medium

Abstract

The embodiment of the present application relates to the technical field of big data, and discloses a kind of simulation big data storage method, device, electronic equipment and computer readable storage medium.The present application is allocated signal ID to each big data simulation signal, and the grouping is carried out to big data simulation signal;According to the grouping result, big data simulation signal is stored in the corresponding file;Using signal ID and file, search and locate big data simulation signal.The present application can improve the retrieval positioning efficiency of actual big data storage.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data technology, and in particular to a method, apparatus, electronic device and computer-readable storage medium for simulating big data storage. Background Technology

[0002] In the actual development and maintenance of engineering projects, operations related to big data mining, prediction, and evaluation are frequently involved, typically requiring high efficiency in real-time storage, retrieval, and location of big data. However, due to the characteristics of real-world big data, such as high dimensionality, large volume, high complexity, and multivariate correlations, directly storing, testing, and analyzing real big data presents challenges due to high cost, low efficiency, and significant difficulty. Therefore, there is an urgent need to provide a method that can guide the storage of real-world big data and improve the retrieval and location efficiency of big data storage. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, electronic device, and computer-readable storage medium for simulating big data storage, in order to guide actual big data storage and improve the retrieval and location efficiency of actual big data storage.

[0004] To address the aforementioned technical problems, embodiments of the present invention provide a method for simulating big data storage, comprising: assigning a signal ID to each big data simulation signal and grouping the big data simulation signals; storing the big data simulation signals into corresponding files according to the grouping results; and searching for and locating the big data simulation signals using the signal IDs and files.

[0005] The embodiments of the present invention also provide a simulated big data storage device, comprising: a big data grouping module, used to assign a signal ID to each big data simulation signal and group the big data simulation signals; a big data storage module, used to store the big data simulation signals into corresponding files according to the grouping results; and a big data search module, used to search for and locate the big data simulation signals using the signal IDs and files.

[0006] Embodiments of the present invention also provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described analog big data storage method.

[0007] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described analog big data storage method.

[0008] In this embodiment of the invention, by assigning a signal ID to each big data simulation signal and grouping the big data simulation signals, and then storing the big data simulation signals in corresponding files according to the grouping results, the use of big data simulation signals to replace real big data can reduce costs while retaining the characteristics of real big data. More importantly, grouping makes the files storing big data simulation signals orderly, and signal IDs make the big data signals within the files orderly. Finally, using the signal IDs and files, big data simulation signals can be quickly searched and located. Therefore, this invention can be used to guide the storage of actual big data and improve the retrieval and location efficiency of actual big data storage.

[0009] In addition, before assigning a signal ID to each big data simulation signal, the method includes: constructing a big data signal simulation model using the MATLAB Simulink tool; adding a signal source module to the big data signal simulation model; adjusting the parameters of the signal source module to simulate the characteristics of big data signals, and outputting the big data simulation signal.

[0010] In addition, the process of assigning a signal ID to each big data simulation signal includes generating a signal ID for each big data simulation signal according to a preset signal ID encoding rule.

[0011] In addition, the grouping of big data simulation signals includes: extracting features of the big data simulation signals, and statistically analyzing the features of the big data simulation signals to obtain key features; and grouping the big data simulation signals using a signal grouping algorithm that matches the key features of the big data simulation signals.

[0012] In addition, the method of searching and locating big data signals using signal IDs and files includes: using the anti-grouping algorithm in the signal grouping algorithm to determine the file where the big data simulation signal is located; and determining the position of the big data simulation signal in the corresponding file based on the signal ID of each big data simulation signal.

[0013] In addition, the step of storing the big data simulation signal into the corresponding file according to the grouping results includes: using the TDMS module in the LabVIEW platform to store the big data simulation signal into the corresponding file in real time according to the grouping results.

[0014] In addition, after searching and locating big data simulation signals using signal IDs and files, the method further includes: using a sampling algorithm to remove big data simulation signals that exceed a preset time threshold from the searched and located big data simulation signals, and displaying big data simulation signals that do not exceed the preset time threshold from the searched and located big data simulation signals. Attached Figure Description

[0015] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0016] Figure 1 This is a flowchart of a simulated big data storage method according to an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of the structure of an analog big data storage device according to another embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of the structure of an electronic device according to another embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details are presented in the various embodiments of the present invention to facilitate a better understanding of this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for ease of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with and referenced by each other without contradiction.

[0020] One embodiment of the present invention relates to a method for simulating big data storage. In this embodiment, by assigning a signal ID to each big data simulation signal and grouping the big data simulation signals, the big data simulation signals are stored in corresponding files according to the grouping results. Using big data simulation signals to replace real big data can reduce costs while retaining the characteristics of real big data. More importantly, grouping makes the files storing big data simulation signals orderly, and signal IDs make the big data signals within the files orderly. Finally, using the signal IDs and files, big data simulation signals can be quickly searched and located. Therefore, the present invention can be used to guide the storage of actual big data and improve the retrieval and location efficiency of actual big data storage.

[0021] like Figure 1 As shown, in step 101, a signal ID is assigned to each big data simulation signal, and the big data simulation signals are grouped.

[0022] In this embodiment, a big data simulation signal is used to replace real big data. On the one hand, the big data simulation signal can simulate the characteristics of real big data, ensuring the effectiveness of related big data processing operations. On the other hand, the big data simulation signal can construct a sufficient amount of data that reflects real big data application scenarios. For example, due to user privacy and security protection policies, it is not possible to directly obtain relevant big data in certain specific fields, such as the medical field and the aviation field. The big data simulation signal can replace the actual big data in these fields.

[0023] In one example, a large data signal simulation model is built using MATLAB Simulink, a tool for modeling, simulating, and analyzing dynamic systems. A signal source module, such as a sine wave generator or a random signal generator, is added to the large data signal simulation model to simulate the output of the large data signal. The parameters of the signal source module are adjusted to simulate the characteristics of the large data signal, outputting the simulated large data signal. These characteristics include, but are not limited to, frequency characteristics, amplitude characteristics, and noise characteristics.

[0024] In this embodiment, assigning a signal ID to each big data simulation signal ensures that each big data simulation signal is ordered, facilitating the retrieval and location of each big data simulation signal.

[0025] In one example, a signal ID for each big data simulation signal is generated according to a preset signal ID encoding rule. For instance, the preset signal ID encoding rule could be based on the time each big data simulation signal was generated, or it could be based on the business attributes, user attributes, or other characteristics of the real big data simulated by each big data simulation signal. It should be noted that the signal ID of each big data simulation signal is globally unique.

[0026] In this embodiment, grouping the big data simulation signal helps to improve the processing efficiency of the big data simulation signal.

[0027] In one example, features of a large-scale simulated signal are extracted, and statistical analysis is performed on these features to obtain key features. These features include, but are not limited to, frequency domain features, time features, energy features, and correlation features. By calculating the average, variance, skewness, kurtosis, and correlation coefficient of the large-scale simulated signal using statistical algorithms, key features can be identified from multiple features. The large-scale simulated signal is then grouped using a signal grouping algorithm that matches its key features. Common signal grouping algorithms include time- or frequency-domain-based grouping, feature-based grouping, and clustering-based grouping. For example, if the key feature of the large-scale simulated signal is correlation, then the k-means clustering algorithm, which matches the correlation feature, can be used to group the large-scale simulated signal.

[0028] In step 102, based on the grouping results, the big data simulation signal is stored in the corresponding file;

[0029] In one example, the TDMS module in the LabVIEW platform is used to store large-scale simulation signals into corresponding files in real time based on the grouping results. LabVIEW is a graphical programming environment and development platform widely used in the field of signal processing. The TDMS module in the LabVIEW platform is a file format and module for efficient storage and management of data, providing a flexible and efficient way to handle the real-time storage and retrieval of large-scale simulation signals.

[0030] In this embodiment, the files storing big data simulation signals are ordered by grouping, and the big data signals within the files are ordered by signal ID.

[0031] In step 103, the big data simulation signal is searched and located using the signal ID and file.

[0032] In this embodiment, the file is ordered and the big data signal in the file is also ordered. Therefore, by using the signal ID and the file, the big data simulation signal can be searched and located quickly.

[0033] In one example, the inverse grouping algorithm from the signal grouping algorithm is used to determine the file containing the big data simulation signal. The inverse grouping algorithm is the inverse transform algorithm relative to the signal grouping algorithm; it can be used to determine the file containing the big data simulation signal. Based on the signal ID of each big data simulation signal, its position within the corresponding file is determined.

[0034] In one example, the TDMS module in the LabVIEW platform is used to read the simulated data signal of the search and localization in real time.

[0035] Because big data has a time-sensitive nature and contains a certain amount of historical data, historical data with low operational frequency is typically not displayed. Therefore, in a specific example, a sampling algorithm is used to remove big data simulation signals exceeding a preset time threshold from the search and positioning big data simulation signals, displaying only those that do not exceed the preset time threshold. The sampling algorithm is used to select a subset from a dataset. Using the sampling algorithm can improve the processing and analysis efficiency of large numbers of big data simulation signals. The preset time threshold can be set according to actual business needs.

[0036] In this embodiment, by assigning a signal ID to each big data simulation signal and grouping the big data simulation signals, and then storing the big data simulation signals in corresponding files according to the grouping results, the use of big data simulation signals to replace real big data can reduce costs while retaining the characteristics of real big data. More importantly, grouping makes the files storing big data simulation signals orderly, and signal IDs make the big data signals within the files orderly. Finally, using the signal IDs and files, big data simulation signals can be quickly searched and located. Therefore, this invention can be used to guide the storage of actual big data and improve the retrieval and location efficiency of actual big data storage.

[0037] The steps described above are for clarity only. In practice, they can be combined into one step or some steps can be broken down into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0038] Another embodiment of the present invention relates to an analog large data storage device, such as... Figure 2 As shown, it includes: a big data grouping module 201, used to assign a signal ID to each big data simulation signal and group the big data simulation signals; a big data storage module 202, used to store the big data simulation signals into corresponding files according to the grouping results; and a big data search module 203, used to search for and locate big data simulation signals using the signal ID and the file.

[0039] In one example, the big data grouping module 201 is used to assign a signal ID to each big data simulation signal and to group the big data simulation signals.

[0040] In one example, a large data signal simulation model is built using MATLAB Simulink, a tool for modeling, simulating, and analyzing dynamic systems. A signal source module, such as a sine wave generator or a random signal generator, is added to the large data signal simulation model to simulate the output of the large data signal. The parameters of the signal source module are adjusted to simulate the characteristics of the large data signal, outputting the simulated large data signal. These characteristics include, but are not limited to, frequency characteristics, amplitude characteristics, and noise characteristics.

[0041] In this embodiment, assigning a signal ID to each big data simulation signal ensures that each big data simulation signal is ordered, facilitating the retrieval and location of each big data simulation signal.

[0042] In one example, a signal ID for each big data simulation signal is generated according to a preset signal ID encoding rule. For instance, the preset signal ID encoding rule could be based on the time each big data simulation signal was generated, or it could be based on the business attributes, user attributes, or other characteristics of the real big data simulated by each big data simulation signal. It should be noted that the signal ID of each big data simulation signal is globally unique.

[0043] In this embodiment, grouping the big data simulation signal helps to improve the processing efficiency of the big data simulation signal.

[0044] In one example, features of a large-scale simulated signal are extracted, and statistical analysis is performed on these features to obtain key features. These features include, but are not limited to, frequency domain features, time features, energy features, and correlation features. By calculating the average, variance, skewness, kurtosis, and correlation coefficient of the large-scale simulated signal using statistical algorithms, key features can be identified from multiple features. The large-scale simulated signal is then grouped using a signal grouping algorithm that matches its key features. Common signal grouping algorithms include time- or frequency-domain-based grouping, feature-based grouping, and clustering-based grouping. For example, if the key feature of the large-scale simulated signal is correlation, then the k-means clustering algorithm, which matches the correlation feature, can be used to group the large-scale simulated signal.

[0045] It is not difficult to see that this embodiment is a device embodiment corresponding to the above method embodiment, and this embodiment can be implemented in conjunction with the above method embodiment. The relevant technical details mentioned in the above method embodiment are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiment.

[0046] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0047] Another embodiment of the present invention relates to an electronic device, such as Figure 3 As shown, it includes at least one processor 301; and a memory 302 communicatively connected to the at least one processor; wherein the memory 302 stores instructions executable by the at least one processor 301, the instructions being executed by the at least one processor 301 to enable the at least one processor 301 to perform the analog big data storage method as described above.

[0048] The memory 302 and processor 301 are connected via a bus, which can include any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors 301 and memory 302 together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 301 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 301.

[0049] Processor 301 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 302 can be used to store data used by processor 301 during operation.

[0050] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the above-described method embodiments.

[0051] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0052] Those skilled in the art will understand that the above embodiments are specific examples of implementing the present invention, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of the present invention.

Claims

1. A method for simulating big data storage, characterized in that, The method includes: Assign a signal ID to each big data simulation signal; Extract features from big data simulation signals and statistically analyze these features to obtain key features; The big data simulation signal is grouped using a signal grouping algorithm that matches the key features of the big data simulation signal. Based on the grouping results, the big data simulation signals are stored in the corresponding files; Using signal IDs and files, search and locate large-scale simulated signals; The method of searching and locating large-scale simulated signals using signal IDs and files includes: The file containing the large data simulation signal is determined using the inverse grouping algorithm in signal grouping algorithms; Based on the signal ID of each big data simulation signal, determine the position of the big data simulation signal in the corresponding file.

2. The simulated big data storage method according to claim 1, characterized in that, Before assigning a signal ID to each big data simulation signal, the method includes: Constructing a large-scale signal simulation model using MATLAB Simulink; Add a signal source module to the big data signal simulation model; Adjust the parameters of the signal source module to simulate the characteristics of big data signals and output the big data simulation signal.

3. The simulated big data storage method according to claim 1, characterized in that, The process of assigning a signal ID to each big data simulation signal includes: Based on the preset signal ID encoding rules, generate the signal ID for each big data simulation signal.

4. The simulated big data storage method according to claim 1, characterized in that, The step of storing the big data simulation signal into the corresponding file according to the grouping results includes: Using the TDMS module in the LabVIEW platform, large-scale simulation signals are stored in real time to the corresponding files based on the grouping results.

5. The simulated big data storage method according to claim 1, characterized in that, After using signal IDs and files to search and locate large-scale simulated signals, the method further includes: Using a sampling algorithm, big data simulation signals that exceed a preset time threshold are removed from the big data simulation signals used for search and positioning, and big data simulation signals that do not exceed the preset time threshold are displayed.

6. A simulated large data storage device, characterized in that, include: The big data grouping module is used to assign a signal ID to each big data simulation signal, extract the features of the big data simulation signal, perform statistics on the features of the big data simulation signal to obtain key features, and use a signal grouping algorithm that matches the key features of the big data simulation signal to group the big data simulation signal. The big data storage module is used to store the big data simulation signals into corresponding files based on the grouping results; The big data search module is used to search for and locate big data simulation signals using signal IDs and files; Specifically, the big data search module is further used to determine the file where the big data simulation signal is located by using the anti-grouping algorithm in the signal grouping algorithm, and to determine the position of the big data simulation signal in the corresponding file according to the signal ID of each big data simulation signal.

7. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the analog big data storage method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the analog big data storage method according to any one of claims 1 to 5.

Images