Data storage method and device, computer equipment, storage medium and computer program product
By matching chips and storage partitions based on the criticality and storage requirements of the data to be processed, the problem of low storage efficiency of traditional chips is solved, and more efficient storage resource utilization is achieved.
Patent Information
- Application Number
- CN202411038745.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-15
AI Technical Summary
Traditional chip data storage methods lead to insufficient storage partition resources, resulting in low data storage efficiency.
According to the data content of the to be processed, determine its target keyity and storage space requirements, build a data storage sequence through sorting, match the target memory chip and partition from the candidate memory chip, and accurately allocate storage resources.
It improves the chip's data storage efficiency, avoids insufficient storage partition resources, and achieves more efficient storage resource utilization.
Smart Images

Figure CN118963666A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data storage method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art
[0002] With the development of integrated circuit chips, chip data storage has become crucial to the chip's operating efficiency.
[0003] In the conventional technology, in the process of storing chip data, the chip is generally divided into multiple equal storage partitions for storing chip data; however, this method easily leads to insufficient storage partition resources of the chip, resulting in low data storage efficiency of the chip. Summary of the invention
[0004] Based on this, it is necessary to provide a data storage method, device, computer equipment, computer-readable storage medium and computer program product that can improve the data storage efficiency of the chip in response to the above technical problems.
[0005] In a first aspect, the present application provides a data storage method, comprising:
[0006] Determining the target criticality and storage space requirement of each of the data to be processed according to the data content of the data to be processed;
[0007] Sorting the data to be processed according to their target criticality, building a data storage sequence based on the sorted data to be processed, and determining the target storage chip corresponding to each data to be processed from the candidate storage chips according to the data processing task of each data to be processed;
[0008] According to the arrangement order of the data to be processed in the data storage sequence, based on the storage space requirement of each data to be processed and the storage capacity of the storage partition in the target storage chip corresponding to each data to be processed, determine the target storage partition corresponding to each data to be processed from the storage partition in the target storage chip corresponding to each data to be processed;
[0009] Each of the data to be processed is stored in a target storage partition corresponding to each of the data to be processed.
[0010] In one embodiment, determining the target criticality and storage space requirement of each of the data to be processed according to the data content of the data to be processed includes:
[0011] Determining the criticality weight of each of the data to be processed according to the data content of the data to be processed;
[0012] Converting key data content in the data content of each of the data to be processed to obtain key text information of each of the data to be processed;
[0013] Identifying semantic information of key text information of each of the data to be processed as semantic information of each of the data to be processed;
[0014] Determine the sub-criticality corresponding to the semantic information of each of the data to be processed;
[0015] Respectively obtaining an average value of the sub-key degrees corresponding to the semantic information of each of the data to be processed as the average sub-key degree of each of the data to be processed;
[0016] The product of the criticality weight of each of the data to be processed and the average sub-criticality is obtained respectively as the target criticality of each of the data to be processed.
[0017] In one embodiment, determining the criticality weight of each data to be processed according to the data content of the data to be processed includes:
[0018] Identifying the data type and data volume of each of the data to be processed according to the data content of the data to be processed;
[0019] According to the data type and data volume of each of the data to be processed, query the target corresponding relationship to obtain the data processing difficulty of each of the data to be processed; the target corresponding relationship is used to characterize the corresponding relationship between the data type, data volume and data processing difficulty;
[0020] The data processing difficulty of each of the data to be processed is normalized to obtain the processed data processing difficulty of each of the data to be processed as the criticality weight of each of the data to be processed.
[0021] In one embodiment, the step of sorting the data to be processed according to the target criticality of each data to be processed, and constructing a data storage sequence based on the sorted data to be processed includes:
[0022] Sorting the data to be processed according to the target criticality of each data to be processed to obtain sorted data to be processed;
[0023] Identify the criticality weights corresponding to the target data to be processed at the same sequence position in the sorted data to be processed;
[0024] According to the criticality weight corresponding to each target data to be processed in the sorted data to be processed, the target data to be processed in the sorted data to be processed are sorted again to obtain the re-sorted data to be processed;
[0025] A data storage sequence is constructed according to the re-sorted data to be processed.
[0026] In one embodiment, determining the target criticality and storage space requirement of each of the data to be processed according to the data content of the data to be processed includes:
[0027] Identifying the data volume of each of the data to be processed from the data content of each of the data to be processed, and determining the compression information of each of the data to be processed;
[0028] When the compression information of each of the data to be processed indicates that compression is not required, the data volume of each of the data to be processed is used as the storage space requirement of each of the data to be processed;
[0029] When the compression information of each of the data to be processed indicates that compression is required, the data volume of the compressed data corresponding to each of the data to be processed is used as the storage space requirement of each of the data to be processed.
[0030] In one embodiment, determining the target storage chip corresponding to each of the data to be processed from the candidate storage chips according to the data processing task of each of the data to be processed includes:
[0031] Determining the data processing function of each of the data to be processed according to the data processing task of each of the data to be processed;
[0032] From the candidate memory chips, a candidate memory chip corresponding to the data processing function of each of the data to be processed is screened out as a target memory chip corresponding to each of the data to be processed.
[0033] In a second aspect, the present application also provides a data storage device, comprising:
[0034] An information determination module, used to determine the target criticality and storage space requirement of each of the data to be processed according to the data content of the data to be processed;
[0035] A chip determination module is used to sort the data to be processed according to the target criticality of each data to be processed, and to construct a data storage sequence according to the sorted data to be processed, and to determine the target storage chip corresponding to each data to be processed from the candidate storage chips according to the data processing task of each data to be processed;
[0036] A partition determination module, configured to determine, according to the arrangement order of the data to be processed in the data storage sequence, a target storage partition corresponding to each of the data to be processed from the storage partitions in the target storage chip corresponding to each of the data to be processed based on the storage space requirements of each of the data to be processed and the storage capacity of the storage partitions in the target storage chip corresponding to each of the data to be processed;
[0037] The data storage module is used to store each of the to-be-processed data in a target storage partition corresponding to each of the to-be-processed data.
[0038] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0039] Determining the target criticality and storage space requirement of each of the data to be processed according to the data content of the data to be processed;
[0040] Sorting the data to be processed according to their target criticality, building a data storage sequence based on the sorted data to be processed, and determining the target storage chip corresponding to each data to be processed from the candidate storage chips according to the data processing task of each data to be processed;
[0041] According to the arrangement order of the data to be processed in the data storage sequence, based on the storage space requirement of each data to be processed and the storage capacity of the storage partition in the target storage chip corresponding to each data to be processed, determine the target storage partition corresponding to each data to be processed from the storage partition in the target storage chip corresponding to each data to be processed;
[0042] Each of the data to be processed is stored in a target storage partition corresponding to each of the data to be processed.
[0043] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:
[0044] Determining the target criticality and storage space requirement of each of the data to be processed according to the data content of the data to be processed;
[0045] Sorting the data to be processed according to their target criticality, building a data storage sequence based on the sorted data to be processed, and determining the target storage chip corresponding to each data to be processed from the candidate storage chips according to the data processing task of each data to be processed;
[0046] According to the arrangement order of the data to be processed in the data storage sequence, based on the storage space requirement of each data to be processed and the storage capacity of the storage partition in the target storage chip corresponding to each data to be processed, determine the target storage partition corresponding to each data to be processed from the storage partition in the target storage chip corresponding to each data to be processed;
[0047] Each of the data to be processed is stored in a target storage partition corresponding to each of the data to be processed.
[0048] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:
[0049] Determining the target criticality and storage space requirement of each of the data to be processed according to the data content of the data to be processed;
[0050] Sorting the data to be processed according to their target criticality, building a data storage sequence based on the sorted data to be processed, and determining the target storage chip corresponding to each data to be processed from the candidate storage chips according to the data processing task of each data to be processed;
[0051] According to the arrangement order of the data to be processed in the data storage sequence, based on the storage space requirement of each data to be processed and the storage capacity of the storage partition in the target storage chip corresponding to each data to be processed, determine the target storage partition corresponding to each data to be processed from the storage partition in the target storage chip corresponding to each data to be processed;
[0052] Each of the data to be processed is stored in a target storage partition corresponding to each of the data to be processed.
[0053] The above-mentioned data storage method, device, computer equipment, storage medium and computer program product first determine the target criticality and storage space requirement of each data to be processed according to the data content of the data to be processed, and then sort the data to be processed according to the target criticality of each data to be processed, and construct a data storage sequence based on the sorted data to be processed, and determine the target storage chip corresponding to each data to be processed from the candidate storage chips according to the data processing task of each data to be processed. Then, according to the arrangement order of the data to be processed in the data storage sequence, based on the storage space requirement of each data to be processed and the storage capacity of the storage partition in the target storage chip corresponding to each data to be processed, determine the target storage partition corresponding to each data to be processed from the storage partition in the target storage chip corresponding to each data to be processed. Finally, each data to be processed is stored in the target storage partition corresponding to each data to be processed. In this way, in the process of chip data storage, by comprehensively determining the chip and storage partition that match the data to be processed based on the data content of the data to be processed, it is beneficial to improve the data storage efficiency of the chip; moreover, this method no longer distributes storage resources evenly, but accurately allocates them according to the specific situation of the data and processing requirements, thereby effectively avoiding the situation of insufficient storage partition resources, thereby improving the data storage efficiency of the chip. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0055] Figure 1 A schematic diagram of a flow chart of a data storage method in one embodiment;
[0056] Figure 2 A schematic flow chart of the steps of determining the target criticality of each data to be processed in one embodiment;
[0057] Figure 3 A schematic flow chart of the steps of determining the criticality weight of each data to be processed in one embodiment;
[0058] Figure 4 A schematic diagram of a flow chart of a data storage method in another embodiment;
[0059] Figure 5 is a structural block diagram of a data storage device in one embodiment;
[0060] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0062] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0063] In an exemplary embodiment, Figure 1As shown, a data storage method is provided. This embodiment uses the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, and tablet computers; the server can be implemented as an independent server or a server cluster composed of multiple servers, such as a network server. In this embodiment, the method includes the following steps:
[0064] Step S101, determining the target criticality and storage space requirement of each data to be processed according to the data content of the data to be processed.
[0065] The data to be processed refers to data information that needs to be stored.
[0066] The data content refers to the specific content of the data to be processed.
[0067] Among them, the target criticality is used to characterize the comprehensive importance of the data to be processed.
[0068] The storage space requirement is used to represent the size of the storage space required for storing the data to be processed.
[0069] Exemplarily, the server determines the criticality weight corresponding to the data content of each data to be processed according to the data content of each data to be processed, as the criticality weight of each data to be processed; then, the server determines the target criticality of each data to be processed according to the data content and the criticality weight of each data to be processed; then, the server identifies the data volume of each data to be processed from the data content of each data to be processed, and determines the compression information of each data to be processed; then, the server determines the storage space requirement of each data to be processed according to the data volume and compression information of each data to be processed.
[0070] Step S102, sorting each data to be processed according to its target criticality, constructing a data storage sequence based on the sorted data to be processed, and determining the target storage chip corresponding to each data to be processed from the candidate storage chips according to the data processing task of each data to be processed.
[0071] The sorted data to be processed refers to the data to be processed obtained by sorting the data to be processed according to the target criticality of each data to be processed.
[0072] The data storage sequence refers to sequence information constructed based on the sorted data to be processed.
[0073] Among them, data processing tasks refer to processing tasks corresponding to the data to be processed, such as data calculation tasks, data analysis tasks, data conversion tasks, and data identification tasks.
[0074] The candidate memory chip refers to a memory chip that can be selected.
[0075] The target storage chip refers to the storage chip corresponding to the data to be processed.
[0076] Exemplarily, the server sorts each data to be processed according to its target criticality to obtain sorted data to be processed; for example, the server sorts each data to be processed in descending order of its target criticality to obtain sorted data to be processed; then, the server constructs a data storage sequence corresponding to the sorted data to be processed based on the sorted data to be processed; then, the server determines, from the candidate storage chips, the candidate storage chip corresponding to the data processing task of each data to be processed, as the target storage chip corresponding to each data to be processed, based on the data processing task of each data to be processed.
[0077] Step S103, according to the arrangement order of the data to be processed in the data storage sequence, based on the storage space requirement of each data to be processed and the storage capacity of the storage partition in the target storage chip corresponding to each data to be processed, determine the target storage partition corresponding to each data to be processed from the storage partition in the target storage chip corresponding to each data to be processed.
[0078] Among them, the arrangement order refers to the serial number information of the data to be processed in the data storage sequence. For example, in the data storage sequence (data to be processed A, data to be processed B, data to be processed C, data to be processed D, data to be processed E), the arrangement order of the data to be processed A is 1, the arrangement order of the data to be processed B is 2, the arrangement order of the data to be processed C is 3, the arrangement order of the data to be processed D is 4, and the arrangement order of the data to be processed E is 5.
[0079] The storage partition refers to an independent area divided from the overall storage space of the target storage chip.
[0080] The storage capacity refers to the amount of data that can be stored in a storage partition.
[0081] The target storage partition refers to the storage partition corresponding to each data to be processed.
[0082] Exemplarily, the server obtains the storage capacity of the storage partition in the target storage chip corresponding to each data to be processed, and according to the arrangement order of the data to be processed in the data storage sequence, based on the storage space requirement of each data to be processed and the storage capacity of the storage partition in the target storage chip corresponding to each data to be processed, determines the storage partition corresponding to each data to be processed from the storage partitions in the target storage chip corresponding to each data to be processed as the target storage partition.
[0083] For example, the server determines that the storage space requirements of each data to be processed are: data A to be processed is 7GB, data B to be processed is 2GB, data C to be processed is 3GB, data D to be processed requires 5GB, and data E to be processed requires 4GB. At the same time, the target storage chip corresponding to each data to be processed is determined from the candidate storage chips to be chip X. There are three storage partitions in chip X, the storage capacity of partition 1 is 6GB, the storage capacity of partition 2 is 10GB, and the storage capacity of partition 3 is 8GB. According to the data storage sequence (data to be processed A, data to be processed B, data to be processed C, data to be processed D, data to be processed E), the data to be processed A is processed first. Since data A to be processed requires 5GB of space, it is allocated to partition 3 (assuming that the partition screening condition is that the corresponding storage capacity is greater than or equal to the storage space requirement of the data to be processed, and is closest to the storage space requirement of the data to be processed). At this time, the remaining storage capacity of partition 3 is 1GB. Next, process the data B to be processed. The data B to be processed requires 2GB of space. Since the remaining storage capacity of partition 3 is insufficient, it is allocated to partition 1. At this time, the remaining storage capacity of partition 1 is 4GB. Then comes the data C to be processed. The data C to be processed requires 3GB of space. It is allocated to partition 1. At this time, the remaining storage capacity of partition 1 is 1GB. Next is the data D to be processed. The data D to be processed requires 5GB of space. Since the remaining storage capacity of partition 1 and partition 3 is insufficient, it is allocated to partition 2. At this time, the remaining storage capacity of partition 2 is 5GB. Finally, process the data E to be processed. The data E to be processed requires 4GB of space. It is allocated to partition 2. At this time, the remaining storage capacity of partition 2 is 1GB. In this way, the target storage partition corresponding to each data to be processed is determined, and the storage capacity of each partition is not exceeded.
[0084] Step S104, storing each data to be processed in a target storage partition corresponding to each data to be processed.
[0085] Exemplarily, the server performs security verification on each piece of data to be processed, and obtains a security verification result corresponding to each piece of data to be processed; when the security verification results corresponding to each piece of data to be processed are all safe, the server establishes a connection channel between each piece of data to be processed and a target storage partition; finally, the server stores each piece of data to be processed in the target storage partition corresponding to each piece of data to be processed through the connection channel between each piece of data to be processed and the target storage partition.
[0086] In the above data storage method, firstly, according to the data content of the data to be processed, the target criticality and storage space requirement of each data to be processed are determined, then according to the target criticality of each data to be processed, the data to be processed are sorted, and according to the sorted data to be processed, a data storage sequence is constructed, and according to the data processing task of each data to be processed, the target storage chip corresponding to each data to be processed is determined from the candidate storage chips, then, according to the arrangement order of the data to be processed in the data storage sequence, based on the storage space requirement of each data to be processed and the storage capacity of the storage partition in the target storage chip corresponding to each data to be processed, the target storage partition corresponding to each data to be processed is determined from the storage partition in the target storage chip corresponding to each data to be processed, and finally, each data to be processed is stored in the target storage partition corresponding to each data to be processed. In this way, in the process of chip data storage, by comprehensively determining the chip and storage partition matching the data to be processed according to the data content of the data to be processed, it is beneficial to improve the data storage efficiency of the chip; moreover, the method no longer evenly distributes storage resources, but accurately distributes according to the specific situation of the data and the processing requirements, thereby effectively avoiding the situation of insufficient storage partition resources, thereby improving the data storage efficiency of the chip.
[0087] In an exemplary embodiment, Figure 2 As shown, the above step S101, according to the data content of the data to be processed, determines the target criticality and storage space requirement of each data to be processed, and specifically includes the following steps:
[0088] Step S201, determining the criticality weight of each data to be processed according to the data content of the data to be processed.
[0089] Step S202: convert key data content in the data content of each data to be processed to obtain key text information of each data to be processed.
[0090] Step S203: identifying semantic information of key text information of each data to be processed as semantic information of each data to be processed.
[0091] Step S204: determining the sub-criticality corresponding to the semantic information of each piece of data to be processed.
[0092] Step S205 , respectively obtaining the average value of the sub-key degrees corresponding to the semantic information of each data to be processed as the average sub-key degree of each data to be processed.
[0093] Step S206 , respectively obtaining the product of the criticality weight of each data to be processed and the average sub-criticality as the target criticality of each data to be processed.
[0094] The criticality weight is used to characterize the importance of the data to be processed.
[0095] Among them, the key data content refers to the data content of each data to be processed whose importance is greater than the preset importance.
[0096] Among them, the key text information refers to the text information corresponding to the key data content.
[0097] The semantic information is used to represent the meaning of the key text information of each data to be processed.
[0098] The sub-keyness is used to represent the importance of the semantic information of each data to be processed.
[0099] The average sub-keyness of each data to be processed refers to the average value of the sub-keyness corresponding to the semantic information of each data to be processed.
[0100] Exemplarily, the server determines the criticality weight corresponding to the data content of each data to be processed according to the data content of each data to be processed, as the criticality weight of each data to be processed; then, the server identifies the importance corresponding to the data content of each data to be processed, and filters out the data content with the corresponding importance greater than the preset importance from the data content of each data to be processed as the key data content; then, the server identifies the data type corresponding to the key data content, and queries the correspondence between the data type and the data conversion network to obtain the data conversion network corresponding to the key data content; then, the server converts the key data content in the data content of each data to be processed according to the data conversion network corresponding to the key data content, to obtain the key text information of each data to be processed; then, the server identifies the semantic information of the key text information of each data to be processed through the semantic recognition network, as the semantic information of each data to be processed; then, the server queries the semantic database for the key information of each semantic information The server obtains the criticality information corresponding to the semantic information of each data to be processed by the corpus belonging to the corpus and the criticality information corresponding to each corpus, and obtains the criticality information corresponding to the semantic information of each data to be processed as the sub-criticality corresponding to the semantic information of each data to be processed; then, the server obtains the average value of the sub-criticality corresponding to the semantic information of each data to be processed, as the average sub-criticality of each data to be processed; for example, the server determines that the number of semantic information corresponding to each data to be processed is 5, then the sub-criticalities corresponding to these 5 semantic information (for example, 0.1, 0.2, 0.3, 0.4, 0.5) are averaged to obtain the average sub-criticality of each data to be processed (for example, 0.3); then, the server obtains the product of the criticality weight of each data to be processed and the average sub-criticality, as the target criticality of each data to be processed; for example, for each data to be processed, the criticality weight of the data to be processed (for example, 0.6) is multiplied by the average sub-criticality (for example, 0.3) to obtain the target criticality of each data to be processed (for example, 0.18).
[0101] It should be noted that the key data content in the data content of each data to be processed can also be extracted by the following method: identifying the data type corresponding to the data content, and extracting the key data content corresponding to the data content of the data type based on the data extraction method corresponding to the data type. For example, the data extraction method for text type is a text feature information recognition network, the data extraction method for picture type is a convolutional neural network based on an image edge recognition algorithm, and the data extraction method for speech type is a speech recognition network.
[0102] In this embodiment, by determining the criticality weight and identifying the semantic information of the key text information, the importance of each data to be processed can be determined more accurately; moreover, by determining the target criticality of each data to be processed, it is conducive to the rational allocation of resources, avoiding excessive processing and analysis of unimportant data, and saving time and computing costs.
[0103] In an exemplary embodiment, Figure 3 As shown, the above step S201, according to the data content of the data to be processed, determines the criticality weight of each data to be processed, which specifically includes the following steps:
[0104] Step S301, identifying the data type and data volume of each data to be processed according to the data content of the data to be processed.
[0105] Step S302, query the target correspondence relationship according to the data type and data volume of each data to be processed to obtain the data processing difficulty of each data to be processed; the target correspondence relationship is used to characterize the correspondence between the data type, data volume and data processing difficulty.
[0106] Step S303 , normalizing the data processing difficulty of each data to be processed to obtain the processed data processing difficulty of each data to be processed as the criticality weight of each data to be processed.
[0107] The data type refers to the type of data to be processed, such as integer type, character type, etc.
[0108] The data volume is used to represent the data size of the data to be processed.
[0109] Among them, the target correspondence is used to characterize the correspondence between data type, data volume and data processing difficulty.
[0110] Among them, data processing difficulty is used to characterize the complexity of processing the data to be processed.
[0111] The difficulty of processing data after processing refers to the difficulty of processing data after normalization.
[0112] Exemplarily, the server determines the format of each data to be processed according to the data content of each data to be processed, and uses the data type corresponding to the format of each data to be processed as the data type of each data to be processed; then, the server determines the data size corresponding to each data to be processed according to the data content of each data to be processed, as the data volume of each data to be processed; then, the server constructs a correspondence between the data type, data volume and data processing difficulty as a target correspondence; then, the server queries the target correspondence according to the data type and data volume of each data to be processed, and obtains the data processing difficulty corresponding to the data type and data volume of each data to be processed as the data processing difficulty of each data to be processed; then, the server normalizes the data processing difficulty (for example, 1, 2, 3, 4) of each data to be processed, and obtains the normalized data processing difficulty (for example, 0.1, 0.2, 0.3, 0.4) as the post-processing data processing difficulty of each data to be processed; then, the server uses the post-processing data processing difficulty of each data to be processed as the criticality weight of each data to be processed.
[0113] In this embodiment, the difficulty of data processing is determined by identifying the data type and data volume, and further normalization is performed to obtain the criticality weight, which can more accurately reflect the complexity and importance of each data to be processed in the processing process, which is conducive to the reasonable allocation of resources in the subsequent data processing process, thereby improving the overall data processing efficiency.
[0114] In an exemplary embodiment, the above step S102, sorting each data to be processed according to its target criticality, and constructing a data storage sequence based on the sorted data to be processed, specifically includes the following contents: sorting each data to be processed according to its target criticality to obtain the sorted data to be processed; identifying the criticality weights corresponding to the target data to be processed at the same sequence position in the sorted data to be processed; re-sorting each target data to be processed in the sorted data to be processed according to the criticality weights corresponding to each target data to be processed in the sorted data to be processed, to obtain the re-sorted data to be processed; and constructing a data storage sequence based on the re-sorted data to be processed.
[0115] The target data to be processed refers to the data to be processed at the same sequence position in the sorted data to be processed.
[0116] The re-sorted data to be processed refers to data to be processed obtained by re-sorting each target data to be processed in the sorted data to be processed according to the criticality weight corresponding to each target data to be processed in the sorted data to be processed.
[0117] Exemplarily, the server sorts each data to be processed according to its target criticality to obtain sorted data to be processed; for example, the server sorts each data to be processed in descending order according to its target criticality to obtain sorted data to be processed; then, the server identifies the data to be processed at the same sequence position in the sorted data to be processed as the target data to be processed; then, the server identifies the criticality weight corresponding to the target data to be processed; then, the server sorts each target data to be processed in the sorted data to be processed again in descending order according to the criticality weight corresponding to each target data to be processed in the sorted data to be processed, to obtain re-sorted data to be processed; then, the server constructs a data storage sequence corresponding to the re-sorted data to be processed based on the re-sorted data to be processed.
[0118] In this embodiment, by sorting and constructing a data storage sequence multiple times, the data to be processed with higher criticality can be placed in a more prioritized position, so that in the subsequent data storage process, the data to be processed with higher criticality can be stored preferentially, thereby improving the performance of the overall system.
[0119] In an exemplary embodiment, the above step S101 determines the target criticality and storage space requirement of each data to be processed according to the data content of the data to be processed, and specifically includes the following contents: identifying the data volume of each data to be processed from the data content of each data to be processed, and determining the compression information of each data to be processed; when the compression information of each data to be processed indicates that compression is not required, using the data volume of each data to be processed as the storage space requirement of each data to be processed; when the compression information of each data to be processed indicates that compression is required, using the data volume of the compressed data corresponding to each data to be processed as the storage space requirement of each data to be processed.
[0120] The compression information of each data to be processed is used to characterize the compression status of each data to be processed, including the data that needs to be compressed, the data that does not need to be compressed, and the data that partially needs to be compressed.
[0121] The compressed data refers to the data to be processed after data compression.
[0122] Exemplarily, the server identifies the data volume of each data to be processed from its data content; then, the server determines the compression information of each data to be processed based on the compression status of each data to be processed; when the compression information of each data to be processed indicates that compression is not required, the server uses the data volume of each data to be processed as the storage space requirement of each data to be processed; when the compression information of each data to be processed indicates that compression is required, the server uses the data volume of the compressed data corresponding to each data to be processed as the storage space requirement of each data to be processed.
[0123] It should be noted that, when the compression information of each data to be processed is partially in need of compression, the server takes the partially compressed data in each data to be processed as the first data to be processed, and takes the data to be processed other than the first data to be processed in each data to be processed as the second data to be processed; then, the server sums the data volume of the compressed data corresponding to the first data to be processed and the data volume of the second data to be processed to obtain the storage space requirement of each data to be processed.
[0124] In this embodiment, the storage space requirement of the data to be processed is determined according to the data volume and compression information of each data to be processed, so that the storage space required for each data to be processed can be accurately evaluated, thereby avoiding waste or shortage of storage space due to unreasonable estimation.
[0125] In an exemplary embodiment, the above step S102, according to the data processing tasks of each data to be processed, determines the target storage chip corresponding to each data to be processed from the candidate storage chips, and specifically includes the following contents: according to the data processing tasks of each data to be processed, determines the data processing function of each data to be processed; from each candidate storage chip, screens out the candidate storage chip corresponding to the data processing function of each data to be processed as the target storage chip corresponding to each data to be processed.
[0126] Among them, the data processing function refers to the functional information required to process the data processing task.
[0127] The candidate memory chip refers to a memory chip that can be selected.
[0128] Exemplarily, the server queries the correspondence between the data processing tasks and the data processing functions according to the data processing tasks of each data to be processed, and obtains the data processing functions of each data to be processed; then, the server selects the candidate storage chips corresponding to the data processing functions of each data to be processed from the candidate storage chips as the target storage chips corresponding to each data to be processed.
[0129] In this embodiment, by accurately matching each piece of data to be processed with a storage chip having the required data processing function, it is ensured that the data processing task can be executed efficiently, thereby giving full play to the advantages of each storage chip and achieving maximum utilization of resources.
[0130] In an exemplary embodiment, Figure 4 As shown, another data storage method is provided, and the method is applied to a server for example, including the following steps:
[0131] Step S401, based on the data content of the data to be processed, identify the data type and data volume of each data to be processed; based on the data type and data volume of each data to be processed, query the target correspondence relationship to obtain the data processing difficulty of each data to be processed; the target correspondence relationship is used to characterize the correspondence between the data type, data volume and data processing difficulty; the data processing difficulty of each data to be processed is normalized to obtain the processed data processing difficulty of each data to be processed as the criticality weight of each data to be processed.
[0132] Step S402, converting the key data content in the data content of each data to be processed to obtain the key text information of each data to be processed; identifying the semantic information of the key text information of each data to be processed as the semantic information of each data to be processed; determining the sub-criticality corresponding to the semantic information of each data to be processed; respectively obtaining the average value of the sub-criticality corresponding to the semantic information of each data to be processed as the average sub-criticality of each data to be processed; respectively obtaining the product of the criticality weight of each data to be processed and the average sub-criticality as the target criticality of each data to be processed.
[0133] Step S403, identifying the data volume of each data to be processed from its data content, and determining the compression information of each data to be processed; when the compression information of each data to be processed indicates that compression is not required, using the data volume of each data to be processed as the storage space requirement of each data to be processed; when the compression information of each data to be processed indicates that compression is required, using the data volume of the compressed data corresponding to each data to be processed as the storage space requirement of each data to be processed.
[0134] Step S404, sorting the data to be processed according to the target criticality of each data to be processed to obtain sorted data to be processed; identifying the criticality weights corresponding to the target data to be processed at the same sequence position in the sorted data to be processed.
[0135] Step S405, re-sorting each target data to be processed in the sorted data to be processed according to the criticality weight corresponding to each target data to be processed in the sorted data to be processed, to obtain re-sorted data to be processed; and constructing a data storage sequence based on the re-sorted data to be processed.
[0136] Step S406, determining the data processing function of each data to be processed according to the data processing task of each data to be processed; and selecting the candidate memory chips corresponding to the data processing function of each data to be processed from the candidate memory chips as the target memory chips corresponding to each data to be processed.
[0137] Step S407, according to the arrangement order of the data to be processed in the data storage sequence, based on the storage space requirement of each data to be processed and the storage capacity of the storage partition in the target storage chip corresponding to each data to be processed, determine the target storage partition corresponding to each data to be processed from the storage partition in the target storage chip corresponding to each data to be processed.
[0138] Step S408, storing each data to be processed in a target storage partition corresponding to each data to be processed.
[0139] In the above data storage method, during the process of chip data storage, the chip and storage partition that match the data to be processed are comprehensively determined based on the data content of the data to be processed, which is beneficial to improving the data storage efficiency of the chip; moreover, this method no longer distributes storage resources evenly, but accurately allocates them according to the specific situation of the data and processing requirements, thereby effectively avoiding the situation of insufficient storage partition resources, thereby improving the data storage efficiency of the chip.
[0140] In an exemplary embodiment, in order to more clearly illustrate the data storage method provided by the embodiment of the present application, the data storage method is specifically described below with a specific embodiment. In one embodiment, the present application also provides a multi-chip storage performance optimization method and system. In the process of chip data storage, the target criticality and storage space requirement of each data to be processed are first determined according to the data content of the data to be processed, and then the data to be processed are sorted according to the target criticality of each data to be processed, and the data storage sequence is constructed according to the sorted data to be processed, and the target storage chip corresponding to each data to be processed is determined from the candidate storage chips according to the data processing task of each data to be processed. Then, according to the arrangement order of the data to be processed in the data storage sequence, based on the storage space requirement of each data to be processed and the storage capacity of the storage partition in the target storage chip corresponding to each data to be processed, the target storage partition corresponding to each data to be processed is determined from the storage partition in the target storage chip corresponding to each data to be processed. Finally, each data to be processed is stored in the target storage partition corresponding to each data to be processed. Specifically including the following contents:
[0141] The multi-chip storage performance optimization method provided in the embodiment of the present application can be applied to the application environment of the Internet of Things chip and the integrated circuit chip. The method can be applied to a terminal, a server, or a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. Among them, the terminal identifies the target chip corresponding to each data information based on the data processing task of each data information and the functional information of each chip, and then identifies the dynamic space demand for storage space of other data content of each data information when the data information is retrieved by different target chips based on the data content of each data information and the storage space demand of each data information. Then, through the data storage sequence corresponding to the criticality of each data information, each data information is stored in the target storage partition corresponding to each data information according to the dynamic space demand, so as to ensure that when each chip calls the data information in each storage partition, the utilization rate of the storage space in the storage partition continues to maintain the maximum utilization rate, avoiding the waste of storage space and insufficient storage space caused by allocating storage space to different chips, thereby improving the utilization rate of the storage space and increasing the storage capacity of each storage partition of the storage space without changing the hardware structure of the storage space, thereby improving the comprehensive utilization rate of the storage space of multiple chips.
[0142] In one embodiment, a multi-chip storage performance optimization method is provided, which is described by taking the method applied to a terminal as an example, and includes the following steps:
[0143] Step S1, obtaining functional information of different chips, data processing tasks of multiple data information, data content of each data information, and storage information of storage space, and identifying chip functional requirement information of each data information based on the data processing task of each data information.
[0144] In this embodiment, the terminal obtains data information that needs to be stored in the storage space corresponding to each chip, and identifies the data processing task of each data information and the data content of each data information. Among them, the data processing task includes but is not limited to data calculation tasks, data analysis tasks, data conversion tasks, data identification tasks and other tasks that can be processed by the chip, wherein the data content in each data information is the data that needs to be processed, and the data information is the identification information of the data content, for example, the name identification, storage address mark, source identification and other identification information. Then, the terminal obtains the functional information of each chip and the storage information of the storage space by querying the basic information of the chip and the basic information of the storage space. Among them, the functional information is the data processing function information of each chip for data processing of the data information. Among them, the storage information of the storage space includes the storage partitions contained in the storage space and the storage capacity of each storage partition. Then, the terminal identifies the chip function requirement information of each data information based on the data processing task of each data information. Among them, the chip function requirement information is the data processing function of the chip required for each data processing stage corresponding to the data processing function of the data information. The specific identification process will be described in detail later.
[0145] Step S2, for each data information, based on the chip function requirement information of the data information and the function information of each chip, identify each target chip corresponding to the data information, and based on the data processing task of the data information, identify the data processing sequence of each target chip.
[0146] In this embodiment, the terminal identifies the target chips corresponding to each data information based on the chip function requirement information of the data information and the function information of each chip, and identifies the data processing sequence of each target chip based on the data processing task of the data information. The target chip is each chip that performs interactive data processing on the data information. Then, the target chips are sorted according to the data processing stage to which each target chip belongs, and the data processing sequence of each target chip is obtained. The specific identification process will be described in detail later.
[0147] Step S3, based on the data content of the data information, identifying the criticality of the data information, and based on the data content of the data information, identifying the storage space requirement of the data information.
[0148] In this embodiment, the terminal identifies the criticality of the data information based on the data content of the data information, and identifies the storage space requirement of the data information based on the data content of the data information. The criticality is used to characterize the degree of data processing of each data content, and the importance of each data information to the user. Specifically, the identification process will be described in detail later. The storage space requirement is the size of the storage area required by the data content of the data information. In the case where the data is not compressed, the storage space requirement is the data volume of the data content. In the case where the data is compressed, the storage space requirement is the data volume of the data content after compression.
[0149] Step S4, determining the data storage sequence of each data information based on the criticality of each data information, and determining the dynamic space requirement of each data information based on the data processing sequence of each target chip corresponding to each data information and the storage space requirement of each data information.
[0150] In this embodiment, the terminal determines the data storage sequence of each data information based on the criticality of each data information, and determines the dynamic space requirement of each data information based on the data processing sequence of each target chip corresponding to each data information and the storage space requirement of each data information. The dynamic space requirement is the storage space requirement of the remaining data content of the data information after different chips retrieve the data information. The specific determination process will be described in detail later.
[0151] Step S5, based on the dynamic space requirement of each data information, the data storage sequence of each data information, and the storage information of the storage space, determine the target storage partition to which each data information belongs and the target data storage sequence corresponding to each storage partition.
[0152] In this embodiment, the terminal determines the target storage partition to which each data information belongs and the target data storage sequence corresponding to each storage partition based on the dynamic space requirement of each data information, the data storage sequence of each data information, and the storage information of the storage space. The specific determination process will be described in detail later.
[0153] Based on the above scheme, the target chip corresponding to each data information is identified based on the data processing task of each data information and the functional information of each chip, and then the dynamic space demand of the other data content of each data information for the storage space when the data information is retrieved by different target chips is identified based on the data content of each data information and the storage space demand of each data information. Then, through the data storage sequence corresponding to the criticality of each data information, each data information is stored in the target storage partition corresponding to each data information according to the dynamic space demand, so as to ensure that when each chip retrieves the data information in each storage partition, the utilization rate of the storage space in the storage partition continues to maintain the maximum utilization rate, avoiding the waste of storage space and insufficient storage space caused by the allocation of storage space to different chips, thereby improving the utilization rate of the storage space, and without changing the hardware structure of the storage space, the storage capacity of each storage partition of the storage space is increased, thereby improving the comprehensive utilization rate of the storage space of multiple chips.
[0154] Optionally, based on the data processing task of each data information, the chip function requirement information of each data information is identified, including: for each data information, identifying the data processing flow corresponding to the data processing task of the data information, and based on each process node of the data processing flow, dividing the data processing flow into multiple data processing stages; for each data processing stage, identifying the processing target information of the data processing stage, and based on the processing target information, identifying the data processing function required for the processing stage; and using the data processing functions required for all data processing stages as the chip function requirement information of the data information.
[0155] In this embodiment, the terminal identifies the data processing flow corresponding to the data processing task of the data information for each data information, and divides the data processing flow into multiple data processing stages based on each flow node of the data processing flow. The flow node is a demarcation point corresponding to different data processing methods for the data information. Then, the terminal identifies the processing target information of the data processing stage for each data processing stage, and identifies the data processing function required for the data processing stage based on the processing target information. The processing target information is the sub-data content that needs to be processed in the data processing stage. For example, the data content includes (1, 2, 3, 4), and the data processing flow includes (a, b, c, d), wherein each data processing stage corresponds to a sub-data content, that is, the processing target information corresponding to the a data processing stage is 1, the processing target information corresponding to the b data processing stage is 2, the processing target information corresponding to the c data processing stage is 3, and the processing target information corresponding to the d data processing stage is 4. For example, the two data contents of the data information are merged, and the corresponding data processing function is a data splicing function. Finally, the terminal uses the data processing functions required for all data processing stages as the chip function requirement information of the data information.
[0156] Based on the above scheme, the data processing task of the data information is divided into multiple data processing stages, and the data processing functions required in each data processing stage are analyzed, so as to obtain the chip function requirement information of the data information.
[0157] Optionally, based on the chip function requirement information of the data information and the function information of each chip, the target chip corresponding to the data information is identified, including: for each data processing stage, based on the data processing function required for the data processing stage, identifying the target function information corresponding to the data processing function in the function information of each chip; and using the chips to which the target function information corresponding to all data processing functions belongs as the target chips corresponding to the data information.
[0158] In this embodiment, for each data processing stage, the terminal identifies the target function information corresponding to the data processing function in the function information of each chip based on the data processing function required in the data processing stage; and uses the chips to which the target function information corresponding to all data processing functions belongs as the target chips corresponding to the data information. In the case where the function information of a chip corresponds to multiple data processing functions, the terminal uses the chip as the target chip of the data processing stage corresponding to each data processing function.
[0159] Based on the above scheme, by identifying the chip to which the function information corresponding to the data processing function required in each data processing stage belongs, each target chip corresponding to the data information is obtained, thereby improving the efficiency of identifying the target chip corresponding to each data information.
[0160] Optionally, based on the data processing task of the data information, the data processing sequence of each target chip is identified, including: based on the data processing flow corresponding to the data processing task, the flow order of each data processing stage is identified, and the target chips corresponding to each data processing stage are sorted according to the flow order to obtain the data processing sequence corresponding to each target chip.
[0161] In this embodiment, the terminal identifies the process order of each data processing stage based on the data processing process corresponding to the data processing task, and sorts the target chips corresponding to each data processing stage according to the process order to obtain the data processing sequence corresponding to each target chip.
[0162] Based on the above scheme, each target chip is sorted through the data processing flow to obtain a data processing sequence corresponding to each target chip, thereby ensuring the processing efficiency of each target chip.
[0163] Optionally, based on the data content of the data information, the criticality of the data information is identified, including: based on the data content of the data information, identifying the data type and the data volume of the data information, and extracting each key data content of the data content; in a database, querying the data processing difficulty corresponding to each data volume of each data type, and based on the data type and the data volume of the data information, identifying the data processing difficulty of the data information, and normalizing the data processing difficulty to obtain a criticality weight value of the data information; converting each key data content into text information to obtain each key text information, and identifying the semantic information of each key text information through a semantic recognition network; in a semantic database, querying the corpus to which each semantic information belongs and the criticality information corresponding to each corpus to obtain a sub-criticality value corresponding to each semantic information, and based on the criticality weight value, performing weighted averaging on the sub-criticality values corresponding to each semantic information to obtain the criticality of the data information.
[0164] In this embodiment, the terminal identifies the data type and the data volume of the data information based on the data content of the data information, and extracts each key data content of the data content. The method of extracting the key data content of each data content is to identify the data type corresponding to the data content, and extract the key data content corresponding to the data content of the data type based on the data extraction method corresponding to the data type. For example, the data extraction method of the text type is a text feature information recognition network, the image type is a convolutional neural network based on an image edge recognition algorithm, and the data extraction method of the voice type is a voice recognition network.
[0165] The terminal queries the database for the data processing difficulty corresponding to each data volume of each data type. The database stores the data processing difficulty corresponding to the target data volume for each data type. The database is a database summarized and summarized by the staff based on a large amount of work experience and a large number of chips in historical operation data. Then, the terminal identifies the data processing difficulty of the data information based on the data type and data volume of the data information, and normalizes the data processing difficulty to obtain the criticality weight value of the data information.
[0166] The terminal converts each key data content into text information to obtain each key text information, and identifies the semantic information of each key text information through the semantic recognition network. Specifically, the terminal identifies the data type corresponding to the key data content and queries the data conversion network of the data type. Then, the terminal converts the key data content into text information of text type based on the data conversion network.
[0167] Then, the terminal queries the corpus to which each semantic information belongs and the key information corresponding to each corpus in the semantic database, obtains the sub-key value corresponding to each semantic information, and performs weighted average processing on the sub-key values corresponding to each semantic information based on the key weight value to obtain the key of the data information. The semantic database stores corpora corresponding to different key values, and each corpus contains multiple semantic information.
[0168] Based on the above scheme, by identifying the key data content of the data content, as well as the data type and data volume, the criticality values corresponding to the key data contents of each data information are weighted averaged to obtain the criticality of the data information, thereby improving the accuracy of the criticality of the identified data information.
[0169] Optionally, based on the criticality of each data information, the data storage sequence of each data information is determined, including: sorting each data information in order from large to small criticality to obtain an initial data storage sequence for each data information; identifying the criticality weight value of each data information at the same sequence position in the initial data storage sequence, and sorting each data information at the same sequence position in order from large to small criticality weight value to obtain a data storage sequence for each data information.
[0170] In this embodiment, the terminal sorts the data information in descending order of the keyness to obtain the initial data storage sequence of the data information. Then, the terminal identifies the key weight value of the data information at the same sequence position in the initial data storage sequence, and sorts the data information at the same sequence position in descending order of the key weight value to obtain the data storage sequence of the data information.
[0171] Based on the above solution, each data information is sorted by criticality, thereby ensuring that data information with high criticality can be preferentially stored and processed, thereby improving the processing efficiency of data information.
[0172] Optionally, based on the data processing sequence of each target chip corresponding to each data information and the storage space requirement of each data information, the dynamic space requirement of each data information is determined, including: for each data information, based on each data processing stage corresponding to the data information, each target chip corresponding to the data information, and the functional information of each target chip, identifying the target data content of the data information and the data volume of each target data content required by each target chip when processing the data information; based on the data processing sequence of each target chip, calculating the amount of sub-data that needs to be stored in the data information when each target chip performs the data processing stage, and arranging and processing each sub-data amount according to the data processing sequence of the target chip corresponding to each sub-data amount, to obtain the dynamic space requirement of the data information.
[0173] In this embodiment, for each data information, the terminal identifies the target data content of the data information required by each target chip when processing the data information, and the data volume of each target data content based on the data processing stages corresponding to the data information, the target chips corresponding to the data information, and the functional information of each target chip. The specific identification process is that each data processing stage corresponds to a sub-data content, and when the chip processes the data information, the data processing stage of the current data processing is identified. Then, when the chip processes the data information, the terminal identifies the current data processing stage currently executed by the chip, and uses the sub-data content corresponding to the current data processing stage as the target data content of the data information, and then the terminal uses the data volume of the sub-data content as the data volume of the target data content.
[0174] Then, based on the data processing sequence of each target chip, the terminal calculates the amount of sub-data that needs to be stored in the data information when each target chip performs the data processing stage. After that, the terminal arranges and processes each sub-data amount according to the data processing sequence of the target chip corresponding to each sub-data amount, and obtains the dynamic space requirement of the data information. Among them, the dynamic space requirement is due to the fact that when the chip processes each data content in the data processing sequence, as each data content completes the data processing, the remaining data amount of the data information gradually decreases, and among them, the sub-data content corresponding to each data processing stage in the data information has different data amounts. Therefore, when executing the sub-data corresponding to each data processing stage, the actual space requirement required by the data information is determined by the data amount of the sub-data content corresponding to the currently executed data processing stage. Therefore, the space requirement is a dynamically changing requirement.
[0175] Based on the above scheme, by identifying the space requirements corresponding to other data contents in the data information when each target chip processes the data information, a screening basis is provided for the subsequent identification of the target storage partition corresponding to each data information, and based on the dynamic space requirements, each data information is stored, thereby increasing the number of data information that the storage partition can accommodate, and improving the storage capacity of the storage partition when the total storage capacity of the storage partition is changed.
[0176] Optionally, based on the dynamic space requirement of each data information, the data storage sequence of each data information, and the storage information of the storage space, determine the target storage partition to which each data information belongs and the target data storage sequence corresponding to each storage partition, including: identifying the storage capacity of each storage partition of the storage space, and identifying the sub-data amounts at different sequence positions in the data storage sequence of each data information; based on the storage capacity of each storage partition, in the order of the data storage sequence from front to back, screen the data information whose sum of the sub-data amounts of each data information at the same sequence position is not greater than the storage capacity as the target data information corresponding to each storage partition, and calculate the sum of the sub-data amounts at different sequence positions of each target data information corresponding to each storage partition; when there is a sequence position where the sum of the sub-data amounts is greater than the storage capacity of the storage partition, sort the target data information of the storage partition according to the data storage sequence to obtain the target data storage sequence corresponding to the storage partition.
[0177] In this embodiment, the terminal identifies the storage capacity of each storage partition of the storage space, and identifies the sub-data amounts of different sequence positions in the data storage sequence of each data information. Then, based on the storage capacity of each storage partition, the terminal selects the data information whose sum of sub-data amounts of each data information at the same sequence position is not greater than the storage capacity in the order of the data storage sequence from front to back, as the target data information corresponding to each storage partition. The target data information corresponding to each storage partition is adjacent in the data storage sequence, and the sum of each sub-data amount of each same sequence position of the data information is not greater than the storage capacity of the storage partition is greater than the number threshold preset in the terminal. And calculate the sum of sub-data amounts of different sequence positions of each target data information corresponding to each storage partition. The terminal determines whether the sum of sub-data amounts of sequence positions is greater than the storage capacity of the storage partition, and when the sum of sub-data amounts of sequence positions is greater than the storage capacity of the storage partition, the target data information of the storage partition is sorted according to the data storage sequence to obtain the target data storage sequence corresponding to the storage partition.
[0178] Specifically, for example, there are A data storage sequence, B data storage sequence, and C data storage sequence, A sequence contains (data information 1, data information 2, data information 3), B sequence contains (data information 4, data information 5, data information 6), C sequence contains (data information 7, data information 8, data information 9), wherein the storage partitions have a partition storage capacity of 50Mb, b partition storage capacity of 30Mb, and c partition storage capacity of 40Mb; wherein the sub-data volume of each data information (that is, the data volume of all data contents currently remaining in the data information) is, data information 1 has 15MB remaining, data information 2 has 10MB remaining, data information 3 has 5MB remaining, data information 4 has 7MB remaining, data information 5 has 20MB remaining, data information 6 has 4MB remaining, data information 7 has 11MB remaining, data information 8 has 40MB remaining, and data information 9 has The remainder is 25MB; the unknown data information of the same sequence include, the first sequence position, data information 1, data information 4, data information 7; the second sequence position, data information 2, data information 5 data information 8; the third sequence position, data information 3, data information 6, data information 9, the sum of the sub-data volume of each sequence position is, the first sequence position, 33Mb, the third sequence position, 70Mb, the third sequence position, 34Mb, then partition A can correspond to data information 1, data information 4, data information 7, or data information 3, data information 6, data information 9, and for each data information at the second sequence position, since the sum of data information 2 and data information 5 is 30Mb, therefore, if the data information at the sequence position is required to be stored in partition A, then the target data storage sequence is 1, data information 2 and data information 5, 2, data information 8.
[0179] Based on the above scheme, each data information is stored based on the dynamic space demand, which increases the number of data information that the storage partition can accommodate and improves the storage capacity of the storage partition when the total storage capacity of the storage partition is changed.
[0180] In the above-mentioned embodiment, during the process of storing data in the chip, the chip and storage partition that match the data to be processed are comprehensively determined according to the data content of the data to be processed, which is conducive to improving the data storage efficiency of the chip; moreover, the method no longer distributes storage resources evenly, but accurately distributes them according to the specific situation of the data and processing requirements, thereby effectively avoiding the situation of insufficient storage partition resources, thereby improving the data storage efficiency of the chip.
[0181] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0182] Based on the same inventive concept, the embodiment of the present application also provides a data storage device for implementing the data storage method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more data storage device embodiments provided below can refer to the limitations on the data storage method above, and will not be repeated here.
[0183] In an exemplary embodiment, Figure 5 As shown, a data storage device is provided, including: an information determination module 501, a chip determination module 502, a partition determination module 503 and a data storage module 504, wherein:
[0184] The information determination module 501 is used to determine the target criticality and storage space requirement of each data to be processed according to the data content of the data to be processed.
[0185] The chip determination module 502 is used to sort each data to be processed according to its target criticality, and to construct a data storage sequence based on the sorted data to be processed, and to determine the target storage chip corresponding to each data to be processed from the candidate storage chips according to the data processing task of each data to be processed.
[0186] The partition determination module 503 is used to determine the target storage partition corresponding to each data to be processed from the storage partitions in the target storage chip corresponding to each data to be processed, based on the arrangement order of the data to be processed in the data storage sequence, the storage space requirement of each data to be processed and the storage capacity of the storage partition in the target storage chip corresponding to each data to be processed.
[0187] The data storage module 504 is used to store each to-be-processed data in a target storage partition corresponding to each to-be-processed data.
[0188] In an exemplary embodiment, the information determination module 501 is also used to determine the criticality weight of each data to be processed according to the data content of the data to be processed; convert the key data content in the data content of each data to be processed to obtain the key text information of each data to be processed; identify the semantic information of the key text information of each data to be processed as the semantic information of each data to be processed; determine the sub-criticality corresponding to the semantic information of each data to be processed; respectively obtain the average value of the sub-criticality corresponding to the semantic information of each data to be processed as the average sub-criticality of each data to be processed; respectively obtain the product of the criticality weight of each data to be processed and the average sub-criticality as the target criticality of each data to be processed.
[0189] In an exemplary embodiment, the information determination module 501 is also used to identify the data type and data volume of each data to be processed based on the data content of the data to be processed; query the target correspondence relationship based on the data type and data volume of each data to be processed to obtain the data processing difficulty of each data to be processed; the target correspondence relationship is used to characterize the correspondence between the data type, data volume and data processing difficulty; the data processing difficulty of each data to be processed is normalized to obtain the processed data processing difficulty of each data to be processed as the criticality weight of each data to be processed.
[0190] In an exemplary embodiment, the chip determination module 502 is also used to sort each data to be processed according to its target criticality to obtain sorted data to be processed; identify the criticality weights corresponding to the target data to be processed at the same sequence position in the sorted data to be processed; re-sort each target data to be processed in the sorted data to be processed according to the criticality weights corresponding to each target data to be processed in the sorted data to be processed to obtain re-sorted data to be processed; and construct a data storage sequence based on the re-sorted data to be processed.
[0191] In an exemplary embodiment, the information determination module 501 is also used to identify the data volume of each data to be processed from the data content of each data to be processed, and determine the compression information of each data to be processed; when the compression information of each data to be processed indicates that compression is not required, the data volume of each data to be processed is used as the storage space requirement of each data to be processed; when the compression information of each data to be processed indicates that compression is required, the data volume of the compressed data corresponding to each data to be processed is used as the storage space requirement of each data to be processed.
[0192] In an exemplary embodiment, the chip determination module 502 is also used to determine the data processing function of each data to be processed according to the data processing task of each data to be processed; and to screen out the candidate storage chips corresponding to the data processing function of each data to be processed from each candidate storage chip as the target storage chip corresponding to each data to be processed.
[0193] Each module in the above data storage device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0194] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as target criticality and storage space requirements. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data storage method is implemented.
[0195] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0196] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.
[0197] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0198] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.
[0199] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0200] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0201] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A data storage method, characterized in that: The method comprises: Determining the target criticality and storage space requirement of each of the data to be processed according to the data content of the data to be processed; Sorting the data to be processed according to their target criticality, building a data storage sequence based on the sorted data to be processed, and determining the target storage chip corresponding to each data to be processed from the candidate storage chips according to the data processing task of each data to be processed; According to the arrangement order of the data to be processed in the data storage sequence, based on the storage space requirement of each data to be processed and the storage capacity of the storage partition in the target storage chip corresponding to each data to be processed, determine the target storage partition corresponding to each data to be processed from the storage partition in the target storage chip corresponding to each data to be processed; Each of the data to be processed is stored in a target storage partition corresponding to each of the data to be processed.
2. The method according to claim 1, characterized in that Determining the target criticality and storage space requirement of each of the data to be processed according to the data content of the data to be processed includes: Determining the criticality weight of each of the data to be processed according to the data content of the data to be processed; Converting key data content in the data content of each of the data to be processed to obtain key text information of each of the data to be processed; Identifying semantic information of key text information of each of the data to be processed as semantic information of each of the data to be processed; Determine the sub-criticality corresponding to the semantic information of each of the data to be processed; Respectively obtaining an average value of the sub-key degrees corresponding to the semantic information of each of the data to be processed as the average sub-key degree of each of the data to be processed; The product of the criticality weight of each of the data to be processed and the average sub-criticality is obtained respectively as the target criticality of each of the data to be processed.
3. The method according to claim 2, characterized in that Determining the criticality weight of each of the data to be processed according to the data content of the data to be processed includes: Identifying the data type and data volume of each of the data to be processed according to the data content of the data to be processed; According to the data type and data volume of each of the data to be processed, query the target corresponding relationship to obtain the data processing difficulty of each of the data to be processed; the target corresponding relationship is used to characterize the corresponding relationship between the data type, data volume and data processing difficulty; The data processing difficulty of each of the data to be processed is normalized to obtain the processed data processing difficulty of each of the data to be processed as the criticality weight of each of the data to be processed.
4. The method according to claim 1, characterized in that: The step of sorting the data to be processed according to the target criticality of each data to be processed, and constructing a data storage sequence based on the sorted data to be processed includes: Sorting the data to be processed according to the target criticality of each data to be processed to obtain sorted data to be processed; Identify the criticality weights corresponding to the target data to be processed at the same sequence position in the sorted data to be processed; According to the criticality weight corresponding to each target data to be processed in the sorted data to be processed, the target data to be processed in the sorted data to be processed are sorted again to obtain the re-sorted data to be processed; A data storage sequence is constructed according to the re-sorted data to be processed.
5. The method according to claim 1, characterized in that Determining the target criticality and storage space requirement of each of the data to be processed according to the data content of the data to be processed includes: Identifying the data volume of each of the data to be processed from the data content of each of the data to be processed, and determining the compression information of each of the data to be processed; When the compression information of each of the data to be processed indicates that compression is not required, the data volume of each of the data to be processed is used as the storage space requirement of each of the data to be processed; When the compression information of each of the data to be processed indicates that compression is required, the data volume of the compressed data corresponding to each of the data to be processed is used as the storage space requirement of each of the data to be processed.
6. The method according to any one of claims 1 to 5, characterized in that The step of determining a target storage chip corresponding to each of the data to be processed from candidate storage chips according to the data processing task of each of the data to be processed comprises: Determining the data processing function of each of the data to be processed according to the data processing task of each of the data to be processed; From the candidate memory chips, a candidate memory chip corresponding to the data processing function of each of the data to be processed is screened out as a target memory chip corresponding to each of the data to be processed.
7. A data storage device, characterized in that: The device comprises: An information determination module, used to determine the target criticality and storage space requirement of each of the data to be processed according to the data content of the data to be processed; A chip determination module is used to sort the data to be processed according to the target criticality of each data to be processed, and to construct a data storage sequence according to the sorted data to be processed, and to determine the target storage chip corresponding to each data to be processed from the candidate storage chips according to the data processing task of each data to be processed; A partition determination module, configured to determine, according to the arrangement order of the data to be processed in the data storage sequence, a target storage partition corresponding to each of the data to be processed from the storage partitions in the target storage chip corresponding to each of the data to be processed based on the storage space requirements of each of the data to be processed and the storage capacity of the storage partitions in the target storage chip corresponding to each of the data to be processed; The data storage module is used to store each of the to-be-processed data in a target storage partition corresponding to each of the to-be-processed data.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
