A data storage method, device, electronic equipment and medium

Abstract

The application relates to the technical field of data storage, in particular to a data storage method and device, electronic equipment and medium. The method comprises the following steps: when a to-be-stored data write request is received, attribute information of the to-be-stored data is acquired; based on the attribute information and preset stored data, a plurality of stored data matching the attribute information is screened out; first key data corresponding to the to-be-stored data is extracted, and second key data corresponding to the stored data is extracted, wherein the key data is data that changes; the similarity between the first key data and the second key data is determined, and whether the similarity reaches a preset maximum similarity threshold is judged; if the similarity reaches the preset maximum similarity threshold, the to-be-stored data is used to cover similar data for data updating, the similar data is stored data corresponding to the second key data; if the similarity does not reach the preset maximum similarity threshold, the to-be-stored data is written. The application effectively improves the efficiency of determining repeated data.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data storage method, apparatus, electronic device, and medium. Background Technology

[0002] With the rapid advancement of computer technology, storage media have emerged to facilitate the storage and application of data. In practical use, people store large amounts of data in storage media. However, a large amount of duplicate data may be stored on the storage media. Since the storage space of the storage media is limited, storing duplicate data will reduce the storage space for non-duplicate data.

[0003] In related technologies, when a data storage request is received, all the content of the data to be stored is matched against the content of every previously stored data item. If it is determined that the content of the data to be stored is the same as the content of a previously stored data item, the data to be stored is considered a duplicate, and only the unique data is stored. However, when there is a large amount of previously stored data in the storage medium, the content matching process takes a long time. Therefore, it is clear that the efficiency of related technologies in identifying duplicate data is low. Summary of the Invention

[0004] To improve the efficiency of identifying duplicate data, this application provides a data storage method, apparatus, electronic device, and medium.

[0005] Firstly, this application provides a data storage method, which adopts the following technical solution:

[0006] A data storage method, comprising:

[0007] When a write request for data to be stored is received, the attribute information of the data to be stored is obtained;

[0008] Based on the attribute information and the preset stored data, several stored data that match the attribute information are selected;

[0009] Extract the first key data corresponding to the data to be stored, and extract the second key data corresponding to the stored data, wherein the key data is data that has changed;

[0010] Determine the similarity between the first key data and the second key data, and determine whether the similarity reaches a preset maximum similarity threshold;

[0011] If the similarity reaches a preset maximum similarity threshold, the data to be stored is used to overwrite the similar data in order to update the data. The similar data is the stored data corresponding to the second key data.

[0012] If the similarity does not reach the preset maximum similarity threshold, then the data to be stored is written.

[0013] In a preferred embodiment, this application can be further configured such that extracting the first key data corresponding to the data to be stored includes:

[0014] Based on the attribute information and the data to be stored, determine whether the data to be stored is standard data;

[0015] If the data to be stored is not standard data, then based on the preset correspondence between attribute information and extraction template and the attribute information, a target extraction template corresponding to the attribute information is determined, and the first key data corresponding to the data to be stored is extracted based on the extraction template.

[0016] If the data to be stored is standard data, then based on the preset attribute information and conversion strategy and the attribute information, the target conversion strategy corresponding to the attribute information is determined;

[0017] The data to be stored is transformed based on the target transformation strategy, and the first key data corresponding to the transformed data to be stored is extracted.

[0018] In a preferred embodiment, this application can be further configured such that determining the similarity between the first key data and the second key data includes:

[0019] Determine the character similarity between the first character of the first key data and the second character of the second key data, wherein the first character is extracted from the first key data and the second character is extracted from the second key data;

[0020] Determine whether the character similarity reaches a preset maximum character similarity threshold;

[0021] If the character similarity does not reach the preset maximum character similarity threshold, then semantic recognition is performed on the first key data to obtain the first semantic, and semantic recognition is performed on the second key data to obtain the second semantic;

[0022] Determine the semantic similarity between the first semantic and the second semantic, and define the semantic similarity as the similarity between the first key data and the second key data.

[0023] In a preferred embodiment, this application can be further configured such that writing the data to be stored includes:

[0024] Obtain data storage partition information, and determine the number of storage areas based on the data storage partition information and the data to be stored;

[0025] Based on the preset correspondence between the number of storage areas and the write mode, and the number of storage areas, determine the target write mode corresponding to the number of storage areas.

[0026] Write the data to be stored based on the target write mode and the data to be stored.

[0027] In a preferred embodiment, this application can be further configured such that writing the data to be stored based on the target write mode and the data to be stored includes:

[0028] Based on the preset correspondence between write modes and write templates and the write mode, the target write template corresponding to the write mode is determined;

[0029] Based on the data to be stored and the target write template, generate write code;

[0030] Data writing instructions are generated based on the writing code to write the data to be stored.

[0031] In a preferred embodiment, this application may be further configured such that, after writing the data to be stored, it also includes:

[0032] Obtain the storage block identifier of the target data storage block, wherein the target data storage block is the storage block storing the data to be stored;

[0033] A verification signal is generated based on the storage block identifier to verify the target data storage block and obtain the verification result;

[0034] If the verification result indicates that the data to be stored has been successfully written, a successful write signal is generated.

[0035] If the verification result indicates that the data to be stored has not been successfully written, a data storage error signal is generated to provide an error alert.

[0036] In a preferred embodiment, this application can be further configured such that, before obtaining the attribute information of the data to be stored, the following steps are included:

[0037] The system detects whether the data to be stored contains virus information and whether the data to be stored contains sensitive words.

[0038] If the data to be stored contains virus information or sensitive words, the data to be stored is determined to be dangerous data, and a write rejection signal is generated.

[0039] Otherwise, the attribute information of the data to be stored is obtained.

[0040] Secondly, this application provides a data storage device, which adopts the following technical solution:

[0041] A data storage device, comprising:

[0042] The acquisition module is used to acquire the attribute information of the data to be stored when a write request for the data to be stored is received.

[0043] The matching module is used to filter out several stored data that match the attribute information based on the attribute information and preset stored data.

[0044] The extraction module is used to extract the first key data corresponding to the data to be stored, and to extract the second key data corresponding to the stored data, wherein the key data is data that has changed;

[0045] A similarity determination module is used to determine the similarity between the first key data and the second key data, and to determine whether the similarity reaches a preset maximum similarity threshold; if the similarity reaches the preset maximum similarity threshold, an update module is triggered; if the similarity does not reach the preset maximum similarity threshold, a write module is triggered.

[0046] An update module is used to overwrite similar data with the data to be stored in order to update the data, wherein the similar data is the stored data corresponding to the second key data;

[0047] The writing module is used to write the data to be stored.

[0048] Thirdly, this application provides an electronic device that adopts the following technical solution:

[0049] At least one processor;

[0050] Memory;

[0051] At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, the at least one application being configured to: execute the data storage method as described in any of the first aspects.

[0052] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution:

[0053] A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the data storage method as described in any of the first aspects.

[0054] In summary, this application includes the following beneficial technical effects:

[0055] When a data write request is received, the attribute information of the data to be stored is obtained to narrow down the filtering scope from the attribute dimension. Then, based on the attribute information and preset stored data, several stored data with the same attributes are filtered out, and the first key data of the data to be stored and the second key data of the stored data are extracted respectively. The similarity between the first key data and the second key data is determined, and the similarity is compared with the preset maximum similarity threshold. By directly judging the similarity between data with a probability of change, it is faster to determine whether the data is duplicate. When the similarity reaches the preset maximum similarity threshold, it indicates that the data to be stored is duplicate data; otherwise, it indicates that the data to be stored is not duplicate data and needs to be stored. Compared with related technologies that match all the contents of each stored data one by one, this application effectively improves the efficiency of duplicate data determination by matching data that may change. Attached Figure Description

[0056] Figure 1 This is a schematic diagram illustrating an application scenario for data storage provided in an embodiment of this application.

[0057] Figure 2 This is a flowchart illustrating a data storage method provided in an embodiment of this application.

[0058] Figure 3 This is a schematic diagram of the structure of a data storage device provided in an embodiment of this application.

[0059] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] The following is in conjunction with the appendix Figure 1 To be continued Figure 4 This application will be described in further detail.

[0061] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of this application.

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application are described clearly and completely. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0064] like Figure 1 As shown in the data storage scenario diagram provided in this application embodiment, a user generates a data write request on a user terminal device and sends the data write request to an electronic device. When the electronic device receives the data write request, it obtains the attribute information of the data to be stored and analyzes the attribute information of the data to be stored to filter out the stored data that matches the attribute information; then it obtains the first key data and the second key data, and determines whether it is duplicate data based on the similarity between the first key data and the second key data. When it is determined to be duplicate data, the electronic device stores the data to be stored and overwrites the duplicate data; when it is determined not to be duplicate data, the electronic device performs a write operation on the stored data.

[0065] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0066] This application provides a data storage method executed by an electronic device, which can be a server or a terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected via wired or wireless communication. This application does not impose any limitations on this connection. Figure 2 As shown, the method includes steps S101, S102, S103, S104, S105, and S106, wherein:

[0067] Step S101: When a write request for the data to be stored is received, obtain the attribute information of the data to be stored.

[0068] Specifically, users can generate data write requests on their client devices to store data. In this embodiment, upon receiving a data write request, a data storage process is executed. Specifically, a monitoring program is pre-integrated into the electronic device to monitor data write requests. Once a write request is detected, the program acquires the attribute information of the data to be stored. More specifically, after the user confirms storage, a data storage instruction is automatically generated. Confirmation methods can include: the user clicking a button on the application or confirming storage via voice.

[0069] The attribute information of the data to be stored may include: document attributes, image attributes, video attributes, and audio attributes. The specific implementation method for obtaining the attribute information of the data to be stored includes: using a suffix-based function call to invoke the suffix of the data to be stored, and determining the attribute information of the data to be stored by judging the suffix. The suffix-based function call includes: strrchr(), explode(), or pathinfo(). This application embodiment does not limit the specific suffix-based function call; users can set it themselves.

[0070] Step S102: Based on the attribute information and the preset stored data, filter out a number of stored data that match the attribute information.

[0071] Specifically, the preset stored data refers to data that has already been stored before the electronic device stores the data to be stored. After the electronic device completes storage, it automatically records the data information corresponding to each stored data. This data information includes at least: data volume, data attribute information, key data (i.e., second key data), and the corresponding storage block identifier. In this embodiment, the data can be stored in a list format. The attribute information of the data to be stored is matched one by one with the attribute information corresponding to all stored data to determine several stored data that match the data to be stored. Matching can be performed using a character matching algorithm. This embodiment does not limit the specific process of the character matching algorithm; users can set it themselves.

[0072] It is understandable that when the data to be stored is a mixture of multiple attribute data, the data to be stored can be separated according to the data attributes to obtain several sub-data to be stored. Then, the attribute information of the sub-data to be stored is matched one by one with the preset stored data to complete the data storage. This application embodiment does not limit the data separation process of the data to be stored, and the user can set it himself.

[0073] Step S103: Extract the first key data corresponding to the data to be stored, and extract the second key data corresponding to the stored data, wherein the key data is data that has changed.

[0074] Specifically, the first key data differs depending on the attribute information of the data to be stored. For example, when the attribute information of the data to be stored is document data, the document name, document author, and document date are all the first key data. When the attribute information of the data to be stored is image data, the objects in the image and their corresponding color distribution information are the first key data. When the attribute information of the data to be stored is audio data, the audio name and the sound source are the first key data. When the attribute information of the data to be stored is video data, the video name, video actors, and video host are the first key data. The second key data corresponding to the stored data with different attribute information also differs, and the second key data of the stored data corresponds to the first key data of the data to be stored with the same attribute information. It is understandable that key data is data in the stored data that has a high probability of change. For example, when the data to be stored is video data (performance program), and the video actors change, the data to be stored will inevitably be different from the stored data. Therefore, comparing based on key data can effectively improve the efficiency of identifying duplicate data compared to comparing based on all data; the aforementioned key data can be the first key data or the second key data, and the corresponding data to be stored can be data to be stored or data already stored.

[0075] Step S104: Determine the similarity between the first key data and the second key data, and determine whether the similarity reaches the preset maximum similarity threshold.

[0076] Specifically, by comparing the first key data and the second key data, the similarity between the first key data and the second key data can be determined. In this embodiment of the application, the similarity between the first key data and the second key data can be determined by character similarity and semantic similarity. It can be understood that by making an initial judgment through character similarity and then making a secondary judgment through semantic similarity, the accuracy of determining the similarity of key data can be effectively improved.

[0077] Furthermore, the similarity and a preset maximum similarity threshold are judged. When the similarity reaches the preset maximum similarity threshold, step S105 is executed, indicating that the data to be stored is duplicated with the already stored data; otherwise, step S106 is executed, indicating that the data to be stored is not duplicated with the already stored data, and therefore the data to be stored is written to the data for data storage. This embodiment does not limit the preset maximum similarity threshold; it can be set by a technician based on their work experience.

[0078] Step S105: If the similarity reaches the preset maximum similarity threshold, the data to be stored is used to overwrite the similar data to update the data. The similar data is the stored data corresponding to the second key data.

[0079] Specifically, the implementation method for overwriting similar data with data to be stored includes: obtaining the storage block identifier of the starting storage block corresponding to the already stored data, the data capacity of the data to be stored, and the storage block capacity; determining the number of storage blocks based on the data capacity and the storage block capacity; determining the identifier of the ending storage block based on the starting storage block identifier and the number of storage blocks, and storing the data to be stored into a storage block group consisting of the starting storage block to the ending storage block for data update. It is understandable that when the data to be stored is duplicated with already stored data, data update can effectively reduce the storage of duplicate data and reduce the storage space occupied by duplicate data. Furthermore, before the electronic device performs a data update, a data update signal can be generated to remind the user that the data to be stored is duplicate data; performing a data update can further reduce space usage.

[0080] Step S106: If the similarity does not reach the preset maximum similarity threshold, write the data to be stored.

[0081] Specifically, when the similarity does not reach the preset maximum similarity threshold, it indicates that the data to be stored is not duplicate data and a write operation is required to store the data to be stored. The specific implementation process of writing the data to be stored can be referred to in the following embodiments.

[0082] In this embodiment, when a data write request is received, the attribute information of the structured data to be stored is obtained to narrow down the filtering range from the attribute dimension. Then, based on the attribute information and preset stored data, several stored data with the same attributes are filtered out, and the first key data of the data to be stored and the second key data of the stored data are extracted respectively. The similarity between the first key data and the second key data is determined, and the similarity is compared with a preset maximum similarity threshold. By directly judging the similarity between data with a probability of change, it is faster to determine whether the data is duplicate. When the similarity reaches the preset maximum similarity threshold, it indicates that the data to be stored is duplicate data; otherwise, it indicates that the data to be stored is not duplicate data and needs to be stored. Compared with the related technology that matches all the contents of each stored data one by one, this application effectively improves the efficiency of duplicate data determination by matching data that may change.

[0083] One possible implementation of this application embodiment is that step S103, which extracts the first key data corresponding to the data to be stored, may specifically include:

[0084] Based on attribute information and the data to be stored, determine whether the data to be stored is data to be transformed;

[0085] If the data to be stored is not the data to be transformed, then based on the correspondence between the preset attribute information and the extraction template and the attribute information, the target extraction template corresponding to the attribute information is determined, and the first key data corresponding to the data to be stored is extracted based on the extraction template;

[0086] If the data to be stored is transformed data, then based on the preset attribute information and transformation strategy, the target transformation strategy corresponding to the attribute information is determined;

[0087] The target transformation strategy is used to transform the data to be stored, and the first key data corresponding to the transformed data to be stored is extracted.

[0088] Specifically, the system determines whether the data to be stored is conversion data based on the correspondence. This correspondence is further defined by the correspondence between attribute information and conversion data, and the attribute information itself. Specifically, the correspondence includes: if the attribute information of the data to be stored is document data or image data, then the data to be stored is not conversion data; if the attribute information of the data to be stored is video data or audio data, then the data to be stored is conversion data. Conversion data refers to data that needs to be converted from non-document data to document data. The correspondence between attribute information and extraction templates can be pre-set by technicians based on their work experience and input into the electronic device. If the attribute information of the data to be stored is document data, then document templates from a pre-set document template library can be matched one-to-one with the data to be stored to obtain the first key data of the data to be stored. The pre-set document template library includes multiple documents and their corresponding document templates. If the attribute information of the data to be stored is image data, then the corresponding target extraction template is the object shape and color distribution information in the image. That is, the object shape and corresponding color distribution information in the image data are extracted, and the object shape and corresponding color distribution information in the image are used as the first key data. The pre-defined correspondence between attribute information and conversion strategies is set by technicians based on their work experience and input into the electronic device. The specific content of this correspondence is as follows: when the attribute information of the data to be stored is audio data, the corresponding conversion strategy is audio signal conversion, that is, converting the audio signal into the corresponding text. This application embodiment does not limit the specific process of audio signal and text conversion; users can set it themselves. When the attribute information of the data to be stored is video data, the corresponding conversion strategy is video signal conversion, converting the video corresponding to the data to be stored into the corresponding text. In this application embodiment, to achieve accurate video signal conversion, preferably, OCR technology is used to convert the video into text. Further, for the converted audio data, when the audio data is the audio of a certain piece of music, the first key data extracted from it can be the music name, the name of the musician, or the name of the instrument, etc. For the converted video data, when the video data is the video of a certain program, the first key data extracted from it can be the program name, the name of the actor, or the program host, etc.

[0089] In this embodiment, when the attribute information of the data to be stored is different, the first key data corresponding to the data to be stored will be extracted differently. Therefore, it is necessary to determine whether the data to be stored is transformed data based on the attribute information and the data to be stored. When the data to be stored is not transformed data, the target extraction template can be directly determined based on the correspondence between the attribute information and the extraction template, so that the first key data of the data to be stored can be extracted directly using the target extraction template, thereby effectively improving the extraction efficiency. When the data to be stored is transformed data, the target transformation strategy of the data to be stored is determined based on the attribute information and the transformation strategy, and the data to be stored is transformed and the first key data is extracted, so as to avoid the problem of incorrect judgment of the data to be stored due to failure to transform the data to be stored.

[0090] One possible implementation of this application embodiment includes step S104, which determines the similarity between the first key data and the second key data, including:

[0091] Determine the character similarity between the first character of the first key data and the second character of the second key data, where the first character is extracted from the first key data and the second key character is extracted from the second key data;

[0092] Determine whether the character similarity has reached the preset maximum character similarity threshold;

[0093] If the character similarity does not reach the preset maximum character similarity threshold, then semantic recognition is performed on the first key data to obtain the first semantic, and semantic recognition is performed on the second key data to obtain the second semantic;

[0094] Determine the semantic similarity between the first semantic and the second semantic, and define the semantic similarity as the similarity between the first key data and the second key data.

[0095] Specifically, the first character is all the characters of the first key data. For example, if the first key data is: actor's name: Zhang San, then the corresponding first characters include: "actor", "member", "name", ":", "Zhang", and "San". The first characters are then hashed to obtain their corresponding hash values, and a string is generated based on these hash values. The process for determining the second character of the second key data is the same as that for the first key data, and will not be repeated in this embodiment. The string corresponding to the first key data is matched with the string of the second key data to determine character similarity, and then the character similarity is compared with a preset maximum character similarity threshold. When the character similarity reaches the preset maximum character similarity threshold, it indicates that the first character of the first key data and the second character of the second key data are the same. In this embodiment, based on actual conditions, preferably, the maximum character similarity threshold is 100%, meaning the data to be stored is the same as the stored data. When the character similarity does not reach the preset maximum similarity threshold, semantic recognition is performed on the first and second key data respectively, which can be done using a semantic recognition network. Semantic recognition, wherein the semantic recognition network is obtained by training a neural network model. This application embodiment does not limit the specific training process of the semantic recognition network; users can set it themselves. Further, the semantic recognition network inputs the recognition results, namely the first semantic and the second semantic, into a semantic recognition result network model. The semantic recognition result network model outputs the semantic similarity between the first and second semantics. The specific training process of the semantic recognition result network model includes: inputting multiple sample semantic groups and their corresponding actual semantic similarities into an untrained semantic recognition result network model, training the untrained semantic recognition result network model to obtain the training semantic similarities corresponding to each of the multiple sample semantic groups, calculating the loss value between the multiple training semantic similarities and the actual semantic similarities using a preset loss function, and determining the untrained semantic recognition result network model whose loss value reaches a preset loss threshold as the final semantic recognition result network model. This application embodiment does not limit the specific loss function and the preset loss threshold; users can set them themselves. Thus, semantic similarity can be obtained, and the semantic similarity can be determined as the similarity between the first key data and the second key data.

[0096] Further, it is determined whether the semantic similarity between the first key data and the second key data reaches a preset maximum semantic similarity threshold. If so, it indicates that although the first key data and the second key data are different, their meanings are the same. In this case, the data to be stored is used to overwrite the similar data, and the data is updated. If the semantic similarity does not reach the preset maximum semantic similarity threshold, it indicates that the first key data and the second key data have different meanings, that is, they are essentially different. In this case, the data to be stored can be written. In this embodiment, preferably, the preset maximum semantic similarity threshold is 99%. It is understood that semantic recognition is quite difficult. In order to effectively improve the accuracy of semantic recognition and avoid judging non-repeating data as repeating data, the preset maximum semantic similarity threshold is set to 99%. It is understood that when there are at least two first key data in the data to be stored, the similarity between all the first key data and their corresponding second key data should reach the preset maximum similarity threshold.

[0097] In this embodiment, the first character of the first key data and the second character of the second key data are determined, and the character similarity is judged to achieve an initial judgment of the data from the character dimension. When some data in the data are changed to words with similar meanings, the data characters change, but the essence of the data does not change. Therefore, when the character similarity does not reach the preset maximum character similarity threshold, semantic recognition is performed on the first key data and the second key data to judge whether the data is repeated from the semantic dimension again, and the semantic similarity is determined as the similarity between the first key data and the second key data, so as to effectively improve the accuracy of data similarity judgment.

[0098] One possible implementation of this application embodiment, step S106, writing the data to be stored, may specifically include:

[0099] Obtain data storage partition information, and determine the number of storage areas based on the data storage partition information and the data to be stored;

[0100] Based on the preset correspondence between the number of storage areas and the write mode, and the number of storage areas, determine the target write mode corresponding to the number of storage areas.

[0101] Write the data to be stored based on the target write mode and the data to be stored.

[0102] Specifically, data storage partition information can characterize the user's demand for data storage partitions during the data storage process; furthermore, the number of storage areas can be determined based on the data partition information. For example, if data to be stored is written within three hours, all data to be stored within three hours will be written to the same storage area. If data to be stored is written within six hours, then the number of storage areas can be determined to be two.

[0103] The preset correspondence between the number of storage areas and the write mode is pre-input by relevant technical personnel and stored in the electronic device. This correspondence may include: when the total number of data storage areas is one, the corresponding data write mode can be static write mode; when the total number of data storage areas is at least two, it is determined whether the amount of data to be stored is not greater than a preset data amount threshold; if so, the write mode of the data to be stored is determined to be dynamic write mode; if the amount of data to be stored is greater than the preset data amount threshold, the resource occupancy of the data to be stored is obtained, and it is determined whether the resource occupancy exceeds a preset resource occupancy threshold; if so, the data write mode of the data to be stored is determined to be multi-threaded write mode; otherwise, the write mode of the data to be stored is determined to be direct write mode. In this embodiment, the preset data amount threshold and the preset resource occupancy threshold are not limited; users can set them themselves. Furthermore, the corresponding target write mode can be used to perform write operations on the data to be stored.

[0104] In this embodiment, data storage partition information is obtained, and the number of storage areas is determined based on the data storage partition information and the data to be stored. Then, the target write mode is determined based on the correspondence between the number of storage areas and the write mode and the number of storage areas. Different write modes are used for different numbers of data storage areas, which can effectively improve the writing efficiency of the data to be stored and reduce the writing time.

[0105] One possible implementation of this application embodiment is that step S106, based on the target writing mode and the data to be stored, writes the data to be stored, which may specifically include:

[0106] Based on the preset correspondence between write modes and write templates, and the write mode, determine the target write template corresponding to the write mode;

[0107] Based on the data to be stored and the target write template, generate write code;

[0108] Data write instructions are generated based on the write code to write the data to be stored.

[0109] Specifically, the preset correspondence between write modes and write templates can be pre-input into the electronic device by relevant technical personnel. This correspondence can be pre-set by relevant technical personnel based on their work experience. This application embodiment does not limit the specific content of the correspondence between write modes and write templates. By matching the write modes with the above correspondence one by one, the target write template corresponding to the write mode can be determined. Then, the data to be stored is filled into the target write template to generate write code, and write instructions are generated based on the write code to write the data to be stored.

[0110] In this embodiment, a target write template is determined based on the preset correspondence between the write mode and the write template, and different write templates are used for different write modes. Then, the data to be stored is filled into the target write template, and write code is generated. The write template is the code write template, so the corresponding write code can be generated. Then, a write instruction is generated based on the write code to write the data to be stored, thereby effectively improving the writing efficiency of the data to be stored.

[0111] In one possible implementation of this application embodiment, after step S106 writes the data to be stored, the method further includes:

[0112] Obtain the storage block identifier of the target data storage block, which is the storage block where the data to be stored is stored;

[0113] A verification signal is generated based on the storage block identifier to verify the target data storage block and obtain the verification result;

[0114] If the verification result indicates that the data to be stored has been successfully written, a successful write signal is generated.

[0115] If the verification result indicates that the data to be stored was not successfully written, a data storage error signal will be generated to provide an error notification.

[0116] Specifically, in this embodiment, the data to be stored can be stored in several target data storage blocks. This embodiment does not limit the number of target data storage blocks; it depends on the actual situation of the data to be stored. The storage block identifier is the logical address of the storage block. Further, the storage block identifier of the target data storage block can be obtained by calling an application programming interface (API). A checksum algorithm can be used to verify the target data storage block. This embodiment does not limit the specific process of the call or the specific process of the checksum algorithm, thus obtaining the verification result. The verification result includes whether the data to be stored was successfully written or not. When the verification result is that the data to be stored was successfully written, it indicates that the data has been completely stored, and a successful write signal can be generated as a reminder. When the verification result is that the data to be stored was not successfully written, it indicates that some data in the data to be stored is missing, and a data storage error signal is generated for timely repair to avoid data loss. Both the successful write signal and the data storage error signal can be transmitted wirelessly to the user device for reminder.

[0117] In this embodiment, the storage block identifier of the target data storage block is obtained, and the target data storage block is verified to obtain the verification result. If the verification result indicates that the data to be stored has been successfully written, a successful write signal is generated, indicating that the data has been successfully stored. If the verification result indicates that the data has not been successfully written, a data storage error signal is generated, indicating that the data has not been successfully written. By verifying the data storage block, the success rate of data writing is effectively improved.

[0118] One possible implementation of this application embodiment includes, before step S101 of obtaining the attribute information of the data to be stored:

[0119] The system detects whether the data to be stored contains virus information and whether it contains sensitive words.

[0120] If the data to be stored contains virus information or sensitive words, the data to be stored is determined to be dangerous data, and a write refusal signal is generated.

[0121] Otherwise, retrieve the attribute information of the data to be stored.

[0122] Specifically, the data source information of the data to be stored is obtained, and the data source is detected to determine whether virus information exists. The virus information can be virus code; that is, all virus codes in a preset virus code library are matched one by one with the codes in the data source. If a match is successful, it indicates that the data to be stored contains virus information; otherwise, it indicates that no virus information exists. Further, preset sensitive words are matched one by one with the data to be stored. It can be understood that when the data to be stored is not transformed data, it can be directly matched with the preset sensitive words; when the data to be stored is transformed data, the transformed data to be stored is matched with the preset sensitive words. This application embodiment does not limit the preset sensitive words. When the data to be stored contains virus information, or when the data to be stored contains sensitive words, it is considered dangerous data, and the storage of the data is refused, and a write refusal signal is generated to alert technicians; otherwise, the attribute information of the data to be stored is obtained.

[0123] In this embodiment, the data to be stored is detected from the perspectives of viruses and sensitive words to determine whether the data to be stored is dangerous data, so as to avoid damage to the storage medium due to the storage of dangerous data. Detection from the perspectives of viruses and sensitive words effectively improves the accuracy of detection.

[0124] The above embodiments describe a data storage method from the perspective of process flow. The following embodiments describe a data storage device from the perspective of virtual module or virtual unit. For details, please refer to the following embodiments.

[0125] This application provides a data storage device, such as... Figure 3 As shown, the data storage device may specifically include:

[0126] The acquisition module 201 is used to acquire the attribute information of the data to be stored when a write request for the data to be stored is received.

[0127] The matching module 202 is used to filter out a number of stored data that match the attribute information based on the attribute information and the preset stored data.

[0128] The extraction module 203 is used to extract the first key data corresponding to the data to be stored, and to extract the second key data corresponding to the stored data, wherein the key data is data that has changed;

[0129] The similarity determination module 204 is used to determine the similarity between the first key data and the second key data, and to determine whether the similarity reaches the preset maximum similarity threshold; if the similarity reaches the preset maximum similarity threshold, the update module 205 is triggered; if the similarity does not reach the preset maximum similarity threshold, the write module 206 is triggered.

[0130] The update module 205 is used to overwrite similar data with the data to be stored in order to update the data. The similar data is the stored data corresponding to the second key data.

[0131] The writing module 206 is used to write data to be stored.

[0132] In one possible implementation of this application embodiment, when the extraction module 203 extracts the first key data corresponding to the data to be stored, it is specifically used for:

[0133] Based on attribute information and the data to be stored, determine whether the data to be stored is data to be transformed;

[0134] If the data to be stored is not the data to be transformed, then based on the correspondence between the preset attribute information and the extraction template and the attribute information, the target extraction template corresponding to the attribute information is determined, and the first key data corresponding to the data to be stored is extracted based on the extraction template;

[0135] If the data to be stored is transformed data, then based on the preset attribute information and transformation strategy, the target transformation strategy corresponding to the attribute information is determined;

[0136] The target transformation strategy is used to transform the data to be stored, and the first key data corresponding to the transformed data to be stored is extracted.

[0137] In one possible implementation of this application embodiment, when the similarity determination module 204 determines the similarity between the first key data and the second key data, it is specifically used for:

[0138] Determine the character similarity between the first character of the first key data and the second character of the second key data, where the first character is extracted from the first key data and the second character is extracted from the second key data;

[0139] Determine whether the character similarity has reached the preset maximum character similarity threshold;

[0140] If the character similarity does not reach the preset maximum character similarity threshold, then semantic recognition is performed on the first key data to obtain the first semantic, and semantic recognition is performed on the second key data to obtain the second semantic;

[0141] Determine the semantic similarity between the first semantic and the second semantic, and define the semantic similarity as the similarity between the first key data and the second key data.

[0142] In one possible implementation of this application embodiment, when the writing module 206 performs the writing of data to be stored, it is specifically used for:

[0143] Obtain data storage partition information, and determine the number of storage areas based on the data storage partition information and the data to be stored;

[0144] Based on the preset correspondence between the number of storage areas and the write mode, and the number of storage areas, determine the target write mode corresponding to the number of storage areas.

[0145] Write the data to be stored based on the target write mode and the data to be stored.

[0146] In one possible implementation of this application embodiment, when the writing module 206 performs the writing of data to be stored based on the target writing mode and the data to be stored, it is specifically used for:

[0147] Based on the preset correspondence between write modes and write templates, and the write mode, determine the target write template corresponding to the write mode;

[0148] Based on the data to be stored and the target write template, generate write code;

[0149] Data write instructions are generated based on the write code to write the data to be stored.

[0150] In one possible implementation of this application embodiment, the data storage device further includes:

[0151] The verification module is used for:

[0152] Obtain the storage block identifier of the target data storage block, which is the storage block where the data to be stored is stored;

[0153] A verification signal is generated based on the storage block identifier to verify the target data storage block and obtain the verification result;

[0154] If the verification result indicates that the data to be stored has been successfully written, a successful write signal is generated.

[0155] If the verification result indicates that the data to be stored was not successfully written, a data storage error signal will be generated to provide an error notification.

[0156] In one possible implementation of this application embodiment, the data storage device further includes:

[0157] The data to be stored detection module is used for:

[0158] The system detects whether the data to be stored contains virus information and whether it contains sensitive words.

[0159] If the data to be stored contains virus information or sensitive words, the data to be stored is determined to be dangerous data, and a write refusal signal is generated.

[0160] Otherwise, retrieve the attribute information of the data to be stored.

[0161] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the data storage device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0162] This application provides an electronic device, such as... Figure 4 As shown, Figure 4 The illustrated electronic device includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device may also include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one type, and the structure of this electronic device does not constitute a limitation on the embodiments of this application.

[0163] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0164] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0165] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0166] The memory 303 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0167] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0168] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments. Compared with related technologies, when a data write request is received, this application obtains the structural information of the structured data to be stored to narrow down the filtering scope from a structural perspective; then, based on the structural information and preset stored data, it filters out several stored data with the same structure, and extracts the first key data of the data to be stored and the second key data of the stored data respectively, determines the similarity between the first key data and the second key data, and determines whether the data is duplicated more quickly by directly judging the similarity between data with a probability of change. Compared with related technologies that match all the contents of each stored data one by one, this application effectively improves the efficiency of duplicate data determination by matching data that may change and determining duplicate data.

[0169] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0170] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data storage method, characterized in that, include: When a write request for data to be stored is received, the attribute information of the data to be stored is obtained; Based on the attribute information and the preset stored data, several stored data that match the attribute information are selected; Extract the first key data corresponding to the data to be stored, and extract the second key data corresponding to the stored data, wherein the key data is data that has changed; Determine the similarity between the first key data and the second key data, and determine whether the similarity reaches a preset maximum similarity threshold; If the similarity reaches a preset maximum similarity threshold, the data to be stored is used to overwrite the similar data in order to update the data. The similar data is the stored data corresponding to the second key data. If the similarity does not reach the preset maximum similarity threshold, then the data to be stored is written. The extraction of the first key data corresponding to the data to be stored includes: Based on the attribute information and the data to be stored, determine whether the data to be stored is conversion data; If the data to be stored is not transformed data, then based on the preset correspondence between attribute information and extraction template and the attribute information, a target extraction template corresponding to the attribute information is determined, and the first key data corresponding to the data to be stored is extracted based on the extraction template. If the data to be stored is transformed data, then based on the preset attribute information and transformation strategy and the attribute information, the target transformation strategy corresponding to the attribute information is determined; The data to be stored is transformed based on the target transformation strategy, and the first key data corresponding to the transformed data to be stored is extracted. The process of writing the data to be stored includes: Obtain data storage partition information, and determine the number of storage areas based on the data storage partition information and the data to be stored; Based on the preset correspondence between the number of storage areas and the write mode, and the number of storage areas, determine the target write mode corresponding to the number of storage areas. Based on the preset correspondence between write modes and write templates and the write mode, the target write template corresponding to the write mode is determined; Based on the data to be stored and the target write template, generate write code; Data writing instructions are generated based on the writing code to write the data to be stored; Obtain the storage block identifier of the target data storage block, wherein the target data storage block is the storage block storing the data to be stored; A verification signal is generated based on the storage block identifier to verify the target data storage block and obtain the verification result; If the verification result indicates that the data to be stored has been successfully written, a successful write signal is generated. If the verification result indicates that the data to be stored has not been successfully written, a data storage error signal is generated to provide an error alert.

2. The data storage method according to claim 1, characterized in that, Determining the similarity between the first key data and the second key data includes: Determine the character similarity between the first character of the first key data and the second character of the second key data, wherein the first character is extracted from the first key data and the second character is extracted from the second key data; Determine whether the character similarity reaches a preset maximum character similarity threshold; If the character similarity does not reach the preset maximum character similarity threshold, then semantic recognition is performed on the first key data to obtain the first semantic, and semantic recognition is performed on the second key data to obtain the second semantic; Determine the semantic similarity between the first semantic and the second semantic, and define the semantic similarity as the similarity between the first key data and the second key data.

3. The data storage method according to claim 1 or 2, characterized in that, Before obtaining the attribute information of the data to be stored, the following steps are included: The system detects whether the data to be stored contains virus information and whether the data to be stored contains sensitive words. If the data to be stored contains virus information or sensitive words, the data to be stored is determined to be dangerous data, and a write rejection signal is generated. Otherwise, the attribute information of the data to be stored is obtained.

4. A data storage device, characterized in that, include: The acquisition module is used to acquire the attribute information of the data to be stored when a write request for the data to be stored is received. The matching module is used to filter out several stored data that match the attribute information based on the attribute information and preset stored data. The extraction module is used to extract the first key data corresponding to the data to be stored, and to extract the second key data corresponding to the stored data, wherein the key data is data that has changed; A similarity determination module is used to determine the similarity between the first key data and the second key data, and to determine whether the similarity reaches a preset maximum similarity threshold; if the similarity reaches the preset maximum similarity threshold, an update module is triggered; if the similarity does not reach the preset maximum similarity threshold, a write module is triggered. An update module is used to overwrite similar data with the data to be stored in order to update the data, wherein the similar data is the stored data corresponding to the second key data; A writing module is used to write the data to be stored; The extraction module is specifically used to determine whether the data to be stored is conversion data based on the attribute information and the data to be stored; if the data to be stored is not conversion data, then based on the preset correspondence between attribute information and extraction template and the attribute information, a target extraction template corresponding to the attribute information is determined, and the first key data corresponding to the data to be stored is extracted based on the extraction template; if the data to be stored is conversion data, then based on the preset attribute information and conversion strategy and the attribute information, a target conversion strategy corresponding to the attribute information is determined; the data to be stored is converted based on the target conversion strategy, and the first key data corresponding to the converted data to be stored is extracted. The writing module is specifically used to acquire data storage partition information, and determine the number of storage areas based on the data storage partition information and the data to be stored; determine a target writing mode corresponding to the number of storage areas based on a preset correspondence between the number of storage areas and writing modes and the number of storage areas; determine a target writing template corresponding to the writing mode based on a preset correspondence between the writing mode and writing template and the writing mode; generate writing code based on the data to be stored and the target writing template; and generate a data writing instruction based on the writing code to write the data to be stored. The verification module is used to obtain the storage block identifier of the target data storage block, which is the storage block storing the data to be stored; generate a verification signal based on the storage block identifier to verify the target data storage block and obtain a verification result; if the verification result indicates that the data to be stored has been successfully written, a successful write signal is generated; if the verification result indicates that the data to be stored has not been successfully written, a data storage error signal is generated to provide an error reminder.

5. An electronic device, characterized in that, include: At least one processor; Memory; At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, said at least one application being configured to: perform the data storage method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed in a computer, causes the computer to perform the data storage method according to any one of claims 1 to 3.

Images