A data storage system and a data writing method

Abstract

The present invention relates to the technical field of data processing, and specifically relates to a data storage system and a data writing method, including: a traversal module for traversing the data content stored in the storage system; a classification module for classifying the data content stored in the data storage system based on similarity; a creation module for creating a data pool in the data storage system and distinguishing and storing the data content that has been classified by the classification module based on the data pool; through the refined classification of the data content in the data storage system, the present invention creates a data pool to distinguish and store the data content, reduces duplicate data based on data similarity analysis during this process, thereby avoiding the ineffective occupation of storage space in the data storage system, and at the same time marks key data information, names the data pool used for storing the data content, realizes the reliable management of data storage, and provides a fast and efficient service for the data storage system when used for data retrieval and retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and specifically relates to a data storage system and a data writing method. Background Art

[0002] Data writing is the process of adding new data to a storage system, which requires ensuring the accuracy and integrity of the data. Data storage is to store data for a long time for access and use at any time. A good storage system can efficiently manage data and ensure its security and availability. Writing and storage cooperate with each other to provide strong support for data analysis and business decision-making.

[0003] A data writing method disclosed in the invention patent with the application number 202311170097.6 is applied to a processor. The method is characterized by including: reading the stored data currently stored in each erasure unit of a memory; identifying the target data to be written into each erasure unit of the memory; comparing the stored data with the target data to generate a comparison result; sending a control instruction corresponding to the comparison result to the memory; the control instruction includes a first instruction and a second instruction. The first instruction is used for the processor to control the memory not to perform an erasure operation on the erasure unit and write the corresponding target data into the erasure unit; the second instruction is used for the processor to control the memory to perform an erasure operation on the erasure unit and then write the corresponding target data into the erased erasure unit.

[0004] This application aims to solve the problem that: "When erasing data in a memory, different memories have different minimum erasure units. Each erasure operation needs to be performed in units of the minimum erasure unit, and the duration required for each erasure operation is usually long, ranging from several milliseconds to several hundred milliseconds. However, the duration required for writing data into the memory is usually short, only ranging from several tens of microseconds to several hundred microseconds. In actual production and manufacturing, when it is necessary to frequently modify the data in the memory or write a large amount of data into the memory, multiple erasure operations need to be performed, resulting in a slow data writing speed and thus an extended production and manufacturing cycle."

[0005] However, currently, when a data storage system manages stored data, it often manages the data by setting simple data storage intervals and distinguishing and storing the data. The application of its distinguished storage intervals is limited to the user-defined storage locations, and it does not have good storage management conditions. Therefore, when there is a large amount of data content stored in the data storage system, the process of retrieving the data content in the data storage system is more difficult.

[0006] Therefore, we propose a data storage system. Summary of the Invention

[0007] In view of the above-mentioned drawbacks of the prior art, the present invention provides a data storage system and a data writing method, which solve the technical problems proposed in the above-mentioned background art.

[0008] To achieve the above objectives, the present invention is realized through the following technical solutions:

[0009] In a first aspect, a data storage system includes:

[0010] A traversal module for traversing the data content stored in the storage system;

[0011] A classification module for classifying the data content stored in the data storage system based on similarity;

[0012] A creation module for creating a data pool in the data storage system and distinguishing and storing the data content classified by the classification module based on the data pool;

[0013] A retrieval module for editing text information and performing a data retrieval operation in the data storage system using the edited text information as the retrieval content;

[0014] A queue module for monitoring the number of times the data content in each data pool in the data storage system is retrieved, evaluating the activity of the data content based on the number of times the data content is retrieved, and arranging the data content in each data pool using the activity of the data content;

[0015] A control unit for setting the operation period of the queue module, controlling the queue module to continuously operate based on the operation period, and refreshing the arrangement queue of the data in each data pool in the data storage system.

[0016] Furthermore, during the operation stage of the traversal module, the storage system does not perform data writing operations. The traversal module is internally provided with sub-modules, including:

[0017] An identification unit for synchronously operating during the operation stage of the traversal module to identify the similarity between groups of data in the storage system;

[0018] Among them, during the operation stage of the traversal module, the recognition result of the similarity between groups of data by the recognition unit is synchronously obtained, and the same data in the storage system is reduced based on the recognition result. The data content stored in the storage system traversed by the traversal module has all been subjected to similarity recognition processing by the recognition unit.

[0019] Furthermore, when the identification unit identifies the similarity between groups of data in the storage system, the similarity recognition logic between two sets of data content used is expressed as:

[0020]

[0021] Where: SIMM(X,Y) is the similarity between data X and data Y; n is the set of elements in data X; m is the set of elements in data Y; p(x i ,y j ) is the probability that the element pair (x i ,y j ) appears in data X and data Y; p(x i ) is the probability that the i-th element appears in data X; p(y j ) is the probability that the j-th element appears in data Y; γ is a correction;

[0022] Among them, the larger the value of SIMM(X,Y), the higher the similarity between the two sets of data. When SIMM(X,Y) ≥ 95%, it is determined that the two sets of data are the same data.

[0023] Furthermore, the value of the correction γ applied in the similarity recognition logic between the two sets of data content follows:

[0024]

[0025] Where: μ X , μ Y are the means of data X and data Y; σ XY is the covariance of data X and data Y; C 1 , C 2 are constants; σ X , σ Y are the standard deviations of data X and data Y;

[0026] Among them, the constants C 1 , C 2 are both greater than 0.

[0027] Furthermore, a sub-module is set under the classification module, including:

[0028] A setting unit for setting the similarity classification interval when the classification module operates to classify the stored data in the data storage system;

[0029] A marking unit for marking the data content classified by the classification module;

[0030] Among them, the similarity classification interval set in the setting unit is user-defined by the system end-user. The similarity classification interval is initially defaulted to 40%, that is, the data content with a similarity of not less than 40% is classified as the same type of data. When the marking unit marks the completed classified data content, the same part of the data in each group of data in the same type of data is used as the marking content to mark each data in the same type of data, and the similarity between each group of data in the same type of data is not less than 40%.

[0031] Furthermore, the number of data pools created in the creation module is equal to the number of data classifications in the classification module. Each group of data pools is used to store a group of homogeneous data, and each group of data pools is named based on the marking content of the data stored therein.

[0032] Furthermore, the operation of editing text information in the retrieval module is performed by the system-side user. After the text information for retrieval is completed, the retrieval module first uses the data pool as the retrieval target, retrieves the data pool with the same name as the edited text information, and after determining the data pool, the system-side user uses the retrieval module to edit the text information again. The retrieval module performs a data content retrieval operation in the determined data pool based on the text information edited again by the system-side user.

[0033] Furthermore, the activity of the data content in the queue module is expressed as:

[0034]

[0035] In the formula: f(a) is the activity of data content a; q is the number of times data content a is retrieved within the operation period of the queue module; t is the average time for data content a to be retrieved once within the operation period of the queue module;

[0036] Based on the above formula, the corresponding activities of each data content in the data pool are obtained. The queue module arranges each data content in the data pool in descending order based on the activity of the data content, so that the data content with high activity is in the front position of the queue, and the data content with low activity is in the rear position of the queue;

[0037] The operation period of the queue module set during the operation stage of the control unit follows:

[0038] When the number of times the data content at the very front position of the data content queue is retrieved within the same time threshold as the previous operation period is less than the number of times it was retrieved in the previous operation period, the queue module refreshes and runs once.

[0039] Furthermore, an identification unit is connected to the inside of the traversal module through wireless network interaction. The traversal module is connected to the classification module through wireless network interaction. The lower level of the classification module is connected to a setting unit and a marking unit through wireless network interaction. The classification module is connected to the creation module through wireless network interaction. The creation module is connected to the marking unit through wireless network interaction. The creation module is connected to the retrieval module through wireless network interaction. The creation module is connected to the queue module and the control unit through wireless network interaction.

[0040] In a second aspect, a data writing method includes the following steps:

[0041] Traverse the data content to be written into the data storage system, and identify each data type in the data content, that is, the format of the data corresponding to each data type;

[0042] Identify the data size proportion of each formatted data in the data corresponding to each data content;

[0043] Select the data format with the largest proportion among the same type of data as the data conversion format, and perform format conversion on the data corresponding to all data formats other than the data format with the largest proportion among the same type of data, so that the data in all the same type of data is in a unified data format;

[0044] Set the data content writing logic, and based on the data content writing logic, after the format conversion of each same type of data content is completed, perform the operation of writing the data content into the data storage system;

[0045] Among them, the types of data content include video, picture, audio, and character. The formats of video data include: MP4, AVI, MOV, WMV, FLV, MKV, 3GP; the formats of picture data include: JPEG / JPG, PNG, GIF, BMP, TIFF / TIF, PSD, SVG, WebP, ICO, PCX; the formats of audio data include: MP3, WMA, WAV, ASF, AAC, VQF, FLAC, APE, MIDI, AMR; the formats of character data include: ASCII, UTF-8, UTF-16, UTF-32, GBK, Latin-1, Unicode, Base64, URL encoding; the data content writing logic includes: among the data content classified based on data types, the data with a small amount is written first, or characters are written prior to audio, audio is written prior to pictures, and pictures are written prior to video.

[0046] Adopting the technical solution provided by the present invention, compared with the known public technologies, it has the following beneficial effects:

[0047] The present invention provides a data storage system and a data writing method. During the operation of the system, through the refined classification of the data content in the data storage system, a data pool is created to distinguish and store the data content, and during this process, duplicate data is reduced based on data similarity analysis, thereby avoiding the ineffective occupation of storage space in the data storage system. At the same time, key data information is marked, and the data pool used for storing the data content is named, so as to realize the reliable management of data storage, provide a faster and more efficient service for the data storage system when used for data retrieval and retrieval, and at the same time, based on the data writing method, during the data writing stage of the data storage system, through data management control, ensure that the data written into the data storage system can be written and stored more quickly. Description of the Drawings

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0049] Figure 1 It is a schematic structural diagram of a data storage system;

[0050] Figure 2 It is a schematic flowchart of a data writing method. Detailed implementation manners

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0052] The following further describes the present invention with reference to the embodiments.

[0053] Embodiment 1:

[0054] A data storage system in this embodiment, as Figure 1 shown, includes:

[0055] A traversal module, configured to traverse the data content stored in the storage system;

[0056] During the operation stage of the traversal module, the storage system does not perform data writing operations. The traversal module is internally provided with sub-modules, including:

[0057] An identification unit, configured to run synchronously during the operation stage of the traversal module to identify the similarity between groups of data in the storage system;

[0058] Among them, during the operation stage of the traversal module, the identification results of the similarity between groups of data obtained by the identification unit synchronously are used to reduce the same data in the storage system. The data content stored in the storage system traversed by the traversal module are all data contents that have been subjected to similarity identification processing by the identification unit;

[0059] When the identification unit identifies the similarity between groups of data in the storage system, the similarity identification logic between two groups of data contents used is expressed as:

[0060]

[0061] Where: SIMM(X,Y) is the similarity between data X and data Y; n is the set of elements in data X; m is the set of elements in data Y; p(x i ,y j ) is the probability that the element pair (x i ,y j ) appears in data X and data Y; p(x i ) is the probability that the i-th element appears in data X; p(y j ) is the probability that the j-th element appears in data Y; γ is a correction;

[0062] Among them, the larger the value of SIMM(X,Y), the higher the similarity between the two sets of data. When SIMM(X,Y)≥95%, it is determined that the two sets of data are the same data;

[0063] In the similarity recognition logic of the content of the two sets of data, the value of the applied correction γ follows:

[0064]

[0065] Where: μ X 、μ Y are the means of data X and data Y; σ XY is the covariance of data X and data Y; C 1 、C 2 are constants; σ X 、σ Y are the standard deviations of data X and data Y;

[0066] Among them, the constants C 1 、C 2 are both greater than 0

[0067] A classification module for classifying the stored data content in the data storage system based on similarity;

[0068] Sub-modules are set under the classification module, including:

[0069] A setting unit for setting the similarity classification interval when the classification module runs to classify the stored data in the data storage system;

[0070] A marking unit for marking the data content classified by the classification module;

[0071] Among them, the similarity classification interval set in the setting unit is user-defined by the system-side user. The initial default value of the similarity classification interval is 40%, that is, the data content with a similarity of not less than 40% is classified as the same type of data. When the marking unit marks the classified data content, the same part of the data in each group of the same type of data is used as the marking content to mark each data in the same type of data, and the similarity between each group of data in the same type of data is not less than 40%;

[0072] The creation module is used to create a data pool in the data storage system and distinguish and store the classified data content by the classification module based on the data pool;

[0073] The retrieval module is used to edit text information and perform data retrieval operations in the data storage system using the edited text information as the retrieval content;

[0074] The queue module is used to monitor the number of times the data content in each data pool in the data storage system is retrieved, evaluate the activity of the data content based on the number of times the data content is retrieved, and arrange the data content in each data pool according to the activity of the data content;

[0075] The activity of the data content in the queue module is expressed as:

[0076]

[0077] In the formula: f(a) is the activity of the data content a; q is the number of times the data content a is retrieved during the operation period of the queue module; t is the average time for the data content a to be retrieved once during the operation period of the queue module;

[0078] Based on the above formula, the corresponding activities of each data content in the data pool are obtained. The queue module arranges the data content in the data pool in descending order based on the activity of the data content, so that the data content with high activity is in the front position of the queue, and the data content with low activity is in the rear position of the queue;

[0079] The operation period of the queue module set in the operation stage of the control unit follows:

[0080] When the number of times the data content at the very front position of the data content queue is retrieved within the same time threshold as the previous operation period is less than the number of times it was retrieved in the previous operation period, the queue module refreshes and runs once;

[0081] The control unit is used to set the operation period of the queue module, control the continuous operation of the queue module based on the operation period, and arrange and refresh the queues of the data in each data pool in the data storage system;

[0082] The traversal module is interactively connected to an identification unit through a wireless network, the traversal module is interactively connected to a classification module through a wireless network, the classification module is interactively connected to a setting unit and a marking unit through a wireless network, the classification module is interactively connected to a creation module through a wireless network, the creation module is interactively connected to the marking unit through a wireless network, the creation module is interactively connected to a retrieval module through a wireless network, and the creation module is interactively connected to a queue module and a control unit through a wireless network.

[0083] In this embodiment, the traversal module runs to traverse the data content stored in the storage system, the identification unit runs synchronously in the traversal module running stage, identifies the similarity between each group of data in the storage system, the classification module further classifies the data content stored in the data storage system based on the similarity, the setting unit synchronously sets the similarity classification interval when the classification module runs to classify the data stored in the data storage system, the marking unit marks the data content classified by the classification module in real time, and then the creation module creates a data pool in the data storage system, and the data content classified by the classification module is distinguished and stored based on the data pool. The retrieval module further edits the text information, and performs the data retrieval operation in the data storage system with the edited text information as the retrieval content. The queue module post-operates to monitor the number of times the data content in each data pool in the data storage system is retrieved, the data content activity is evaluated based on the number of times the data content is retrieved, and the data content in each data pool is arranged using the data content activity, and the queue module operation cycle is synchronously set by the control unit, and the queue module is controlled to run continuously based on the operation cycle to arrange the queue refresh of the data in each data pool in the data storage system.

[0084] Through the operation of the system in the above embodiment, further refined data storage logic is provided to the data storage system, ensuring that the data stored in the data storage system is easier to retrieve and manage.

[0085] Embodiment 2:

[0086] The number of data pools created in the creation module is equal to the number of data categories in the classification module. Each group of data pools is used to store a group of similar data. Each group of data pools is named based on the tag content of the data content stored therein;

[0087] The operation of editing text information in the retrieval module is performed by the system-side user. After the text information used for retrieval is edited, the retrieval module first uses the data pool as the retrieval target to search for the data pool with the same name as the edited text information. After determining the data pool, the system-side user applies the retrieval module to edit the text information again. The retrieval module performs the retrieval operation of the data content in the determined data pool based on the text information edited again by the system-side user.

[0088] In this embodiment, through the above settings, further operation data support is provided for the system operation in Embodiment 1, ensuring the stability of the system operation in Embodiment 1.

[0089] Embodiment 3:

[0090] A data writing method includes the following steps:

[0091] Traverse the data content to be written into the data storage system, and identify each data type in the data content, that is, the format of the data corresponding to each data type;

[0092] Identify the data size ratio of each format data in the data corresponding to each data content;

[0093] Select the data format with the largest proportion in the same type of data as the data conversion format, and perform format conversion on the data corresponding to all data formats except the data format with the largest proportion in the same type of data, so that the data in all the same type of data is unified in data format;

[0094] Set the data content writing logic, and based on the data content writing logic, after the format conversion of each same type of data content is completed, perform the operation of writing the data content into the data storage system;

[0095] Among them, the types of data content include video, picture, audio, and character. The formats of video data include: MP4, AVI, MOV, WMV, FLV, MKV, 3GP; the formats of picture data include: JPEG / JPG, PNG, GIF, BMP, TIFF / TIF, PSD, SVG, WebP, ICO, PCX; the formats of audio data include: MP3, WMA, WAV, ASF, AAC, VQF, FLAC, APE, MIDI, AMR; the formats of character data include: ASCII, UTF-8, UTF-16, UTF-32, GBK, Latin-1, Unicode, Base64, URL encoding; the data content writing logic includes: among the data content classified based on the data type, the data with a small amount is written first, or characters are written before audio, audio is written before pictures, and pictures are written before video.

[0096] In summary, in the above embodiments, the system creates data pools to distinguish stored data content through refined classification of the data content in the data storage system, and reduces duplicate data based on data similarity analysis during this process, thereby avoiding the ineffective occupation of storage space in the data storage system. At the same time, key data information is marked to name the data pools used for data content storage, thereby realizing the reliable management of data storage. When the data storage system is used for data retrieval and extraction, it provides a more efficient service. At the same time, based on the data writing method, during the data writing stage of the data storage system, through data management control, it is ensured that the data written into the data storage system can be written and stored more quickly.

[0097] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data storage system, characterized in that: include: A traversal module, used to traverse the data content stored in the storage system; A classification module, used to classify the data content stored in the data storage system based on similarity; A creation module is used to create a data pool in the data storage system, and to differentiate and store the data content classified by the classification module based on the data pool; The number of data pools created in the creation module is equal to the number of data categories in the classification module. Each group of data pools is used to store a group of similar data. Each group of data pools is named based on the tag content of the data content stored therein; A retrieval module, used for editing text information and performing a data retrieval operation in a data storage system using the edited text information as retrieval content; A queue module is used to monitor the number of times data content in each data pool in the data storage system is retrieved, evaluate the activity of the data content based on the number of times the data content is retrieved, and arrange the data content in each data pool using the activity of the data content; The data content activity in the queue module is expressed as: ； Where: is the activity of data content a; is the number of times data content a is retrieved during the operation cycle of the queue module; It is the average time for data content a to be retrieved once during the operation cycle of the queue module; Based on the above formula, the corresponding activity of each data content in the data pool is obtained. The queue module arranges the data content in the data pool in descending order based on the activity of the data content, so that the data content with high activity is at the front position of the queue, and the data content with low activity is at the back position of the queue; The queue module operation cycle set during the control unit operation phase follows: When the data content at the front position of the data content queue is retrieved less than the number of times it was retrieved in the previous operation cycle within the same time threshold as the previous operation cycle, the queue module refreshes and runs once; The control unit is used to set the operation cycle of the queue module, control the queue module to run continuously based on the operation cycle, and arrange the queues and refresh the data in each data pool in the data storage system.

2. A data storage system according to claim 1, characterized in that: During the operation phase of the traversal module, the storage system does not perform data writing operations. The traversal module is internally provided with submodules, including: The identification unit is used to run synchronously with the traversal module running phase to identify the similarities between groups of data in the storage system; Among them, during the operation phase of the traversal module, the recognition unit synchronously obtains the recognition results of the similarities between each group of data, and eliminates the same data in the storage system based on the recognition results. The data contents stored in the storage system traversed by the traversal module are all data contents that have been processed by the recognition unit for similarity recognition.

3. A data storage system according to claim 2, characterized in that: The similarity recognition logic between the two sets of data contents is expressed as: ； Where: is the similarity between data X and data Y; is the set of elements in data X; is the set of elements in data Y; is the element pair of data X and data Y Probability of occurrence; is the probability of the i-th element appearing in data X; is the probability of the jth element appearing in data Y; To amend; in, The larger the value, the higher the similarity between the two sets of data. When the accuracy is ≥95%, the two sets of data are judged to be the same.

4. A data storage system according to claim 3, characterized in that: Modifications applied in the logic of identifying the similarity between the two sets of data contents The value is subject to: ； Where: , is the mean of data X and data Y; is the covariance of data X and data Y; , is a constant; , is the standard deviation of data X and data Y; Among them, the constant , Both are greater than 0.

5. A data storage system according to claim 1, characterized in that: The classification module is provided with submodules at the lower level, including: A setting unit, used for setting a similarity classification interval when the classification module is running to classify the data stored in the data storage system; A marking unit, used for marking the data content classified by the classification module; Among them, the similarity classification interval set in the setting unit is customized by the system user, and the initial default similarity classification interval is 40%, that is, data content with a similarity of not less than 40% is classified as similar data. When the marking unit marks the classified data content, the same part of the data in each group of data in the same data is used as the marking content, and each data in the same data is marked, and the similarity between each group of data in the same data is not less than 40%.

6. A data storage system according to claim 1, characterized in that: The operation of editing text information in the retrieval module is performed by the system-side user. After the text information used for retrieval is edited, the retrieval module first uses the data pool as the retrieval target to search for the data pool with the same name as the edited text information. After determining the data pool, the system-side user applies the retrieval module to edit the text information again. The retrieval module performs a retrieval operation on the data content in the determined data pool based on the text information edited again by the system-side user.

7. A data storage system according to claim 1, characterized in that: The traversal module is interactively connected to an identification unit via a wireless network, the traversal module is interactively connected to a classification module via a wireless network, the classification module is interactively connected to a setting unit and a marking unit via a wireless network, the classification module is interactively connected to a creation module via a wireless network, the creation module is interactively connected to the marking unit via a wireless network, the creation module is interactively connected to a retrieval module via a wireless network, and the creation module is interactively connected to a queue module and a control unit via a wireless network.

8. A data writing method, the method being an implementation method of a data storage system as claimed in any one of claims 1 to 7, characterized in that: The following steps are involved: Traverse the data content that needs to be written into the data storage system, and identify the data types in the data content, that is, the format of the data corresponding to each data type; Identify the data size ratio of each format of data in the data corresponding to each data content; Select the data format with the largest proportion among the same type of data as the data conversion format, and perform format conversion on the corresponding data of all data formats except the data format with the largest proportion among the same type of data, so that the data of all the same type of data has a unified data format; Setting data content writing logic, and executing the operation of writing data content into the data storage system after format conversion of each data content of the same type is completed based on the data content writing logic; Among them, the types of data content include video, picture, audio, and characters. The formats of video data include: MP4, AVI, MOV, WMV, FLV, MKV, and 3GP; the formats of picture data include: JPEG / JPG, PNG, GIF, BMP, TIFF / TIF, PSD, SVG, WebP, ICO, and PCX; the formats of audio data include: MP3, WMA, WAV, ASF, AAC, VQF, FLAC, APE, MIDI, and AMR; the formats of character data include: ASCII, UTF-8, UTF-16, UTF-32, GBK, Latin-1, Unicode, Base64, and URL encoding; the data content writing logic includes: in the data content classified based on the data type, data with small data volume is written first, or characters are written first before audio, audio is written first before pictures, and pictures are written first before video.

Images