Computing resource expansion system based on data storage
By processing computing tasks in parallel on multiple storage nodes, using task scheduling algorithms and data consistency maintenance, the performance bottlenecks and slow data calculation problems of traditional computer systems are solved, and efficient computing resource expansion and product information processing are achieved.
Patent Information
- Application Number
- CN202510411317.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-12
- Publication Date
- 2025-07-22
AI Technical Summary
Traditional computer systems have problems such as single point failure, performance bottlenecks and low resource utilization when dealing with computing resource requirements. Especially in warehouse supermarkets, the increase in commodity data leads to slow data calculation and slow cargo matching speed.
Design a computing resource expansion system based on data storage. By processing computing tasks in parallel on multiple storage nodes, using task scheduling algorithms and data consistency maintenance, it realizes coordinated operation between storage nodes, optimizes the balance of computing and storage, and processes product information in parallel.
It improves the utilization rate of computing resources and system performance, avoids data delays and slow calculations, ensures data consistency and reliability, and optimizes the processing efficiency of product information and the utilization rate of homepage product recommendations.
Smart Images
Figure CN120353580A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of commodity computing, and specifically provides a computing resource expansion system based on data storage. Background Art
[0002] The computing resource requirements in modern society are increasing continuously. However, traditional computer systems have certain limitations in meeting such requirements. Traditional computer systems usually adopt a centralized architecture, in which computing tasks are processed by a main server, and end users can only access the main server through terminals. This architecture has some problems, such as single point of failure, performance bottleneck, and low resource utilization rate, etc.
[0003] To solve these problems, a new computing resource expansion system has been proposed. This system is based on data storage and distributes computing tasks to multiple storage nodes for parallel processing. Different from the traditional centralized architecture, this system makes full use of the computing power of storage nodes, improving the utilization rate of computing resources and the performance of the system.
[0004] The core idea of this system is to closely combine computing tasks and data storage. By running computing tasks on storage nodes, the overhead of data transmission can be avoided, and multiple tasks can be processed in parallel. In addition, storage nodes can perform collaborative processing of tasks, further improving the performance of the system. With the rapid development of technologies and markets such as mobile Internet, Internet of Things, and AI computing, various fields need to be integrated with computers to form stronger competitiveness. Warehouse-style large supermarkets are different from traditional supermarkets and are managed by applying modern computer technologies, that is, through barcodes on commodities, fast payment settlement is implemented, and scientific and reasonable control is taken for the purchase, sale, and inventory of commodities. This not only facilitates people's shopping but also greatly improves the sales management level of the mall. However, with the expansion of warehouse-style supermarkets, the data generated from the purchase, sale, and inventory of commodities increases, making it more difficult to centralize and calculate the data in the centralized computing and storage center, resulting in slow data calculation, slow goods matching speed, etc. Therefore, it is very necessary to design a computing resource expansion system based on data storage. Summary of the Invention
[0005] The purpose of the present invention is to provide a computing resource expansion system based on data storage to solve the problems raised in the above background art.
[0006] To solve the above technical problems, the present invention provides the following technical solution: A computing resource expansion system based on data storage, including a cloud storage system, characterized in that the cloud storage system includes a data plane and a metadata plane, wherein the data plane is used to store user data, and the metadata plane is used to store the metadata corresponding to the data. The user data includes the data volume and the access volume. The increase in the data volume and access volume of the user will cause an increase in the number of entries and query volume stored in the metadata plane. The scalability of the metadata plane will directly affect the scalability of the entire cloud storage system; The metadata plane includes a metadata storage base, a data directory unit one linking the metadata storage base, and a data directory unit two linking the metadata storage base; The data directory unit one stores commodity information and supports the formation of a commodity information directory and the formation of a commodity information file; The data directory unit two stores a location information list of commodity files and additionally adds a transmission path for each commodity file to provide a path for each commodity file to convert the computing system. Each store has an independent computing system; Set a store as a storage node. Each storage node needs to have sufficient computing power and storage capacity to support the processing of computing tasks and the storage of data. The design of the storage node should consider the balance between computing and storage, as well as the communication and collaborative processing between nodes. Set each storage node as A1, A2, A3,..., A N .
[0007] According to the above technical solution, a task scheduling algorithm is set in each of the storage nodes, that is, the computing power in each storage node is set as C. Here, the computing power is the expected computable power. Divide C according to the operation load of 0-100%. 0 means the storage node has the strongest computing power, and 100% means the storage node has the weakest computing power. Set different load standards during the operation division of 0-100%: 0-60%: The computing power of the store is excellent and meets the standard for fully starting the task scheduling algorithm; 60%-90%: The computing power of the store is fully loaded and meets the standard for partially starting the task scheduling algorithm; 90%-100%: The computing power of the store is overloaded and does not meet the standard for starting the task scheduling algorithm; The computing power of each store is a fixed value, with a maximum of 100%. When the computing power of storage node A1 is in the range of 0 - 60%, it means that storage node A1 has the ability to calculate the commodity files of the remaining storage nodes. When the computing power of storage node A1 is in the range of 60% - 90%, it means that storage node A1 has the ability to calculate part of the commodity files of the remaining storage nodes. When the computing power of storage node A1 is in the range of 90% - 100%, it means that storage node A1 does not have the ability to calculate the commodity files of the remaining storage nodes. The remaining storage nodes and storage node A1 are based on the same standard of the task scheduling algorithm.
[0008] According to the above technical solution, each file is divided into several file blocks, each file block is stored in a surface folder, and several surface folders form a bottom folder. The file part is the surface folder, and the number of surface folders is selected according to the computable power of the selected storage node.
[0009] According to the above technical solution, since the commodity information processing is carried out on multiple storage nodes and there may be data dependency relationships between the commodity information, data consistency maintenance is set in the task scheduling algorithm. Data consistency maintenance can ensure that the execution results of the commodity information on different nodes are consistent and guarantee the correctness and reliability of the data. When the calculation result is returned through the transmission path, this calculation result is independently stored and backed up, and this independently stored and backed up calculation result can be obtained and used. When the computing power of its own storage node is lower than 60%, the calculation result is verified. Data loss and other situations may occur during the transmission process of the calculation result. If the verification result is correct, the calculation result is continued to be used. If the verification result is incorrect, the result is recalculated through its own storage node and notified by the system.
[0010] According to the above technical solution, the data volume of the user data includes the identity information entered when the user enters the store and the user information calculated from the data surface. The identity information of each user includes the frequency of the user entering the store and the purchase record of the commodities. The user information calculated from the data surface includes the average frequency of the user entering the store, the time since the last entry into the store, and the expected purchase of commodities; The expected purchased items are calculated based on the purchased item records. The items purchased by the user are marked for the first time. A purchase habit database of the user is formed through the first seven purchase records. The items purchased by the user multiple times are marked for the second time, and the first mark is deleted. The first mark is the mark for the formation of the user's purchase habit, and the marked items are not completely the items the user likes to use. The items marked for the second time are the items repeatedly purchased by the user. When the user logs in to the store app, the items marked for the second time are displayed on the home page for the user to directly obtain the purchase channels. At the same time, related items of the items marked for the second time are recommended on the home page or the second page to ensure the maximum utilization rate of the item information on the home page. To ensure the stability of the computing power.
[0011] According to the above technical solution, the method of marking deletion is used to reduce the proportion of data occupying the computing power range. The marks to be deleted will ultimately be divided into multiple small marks, and the information of the cancelled users also needs to be deleted. The user information to be deleted is divided into multiple information blocks. The small marks and information blocks are stored in the garbage data area. The method of deleting one by one is adopted to shorten the length of the continuous garbage data area. We disperse the pressure of data deletion, split the garbage data area into multiple small pieces, and at the same time enhance the ability of a single storage node to process garbage data, alleviating its impact on the query performance. The length of the garbage data is reduced by two divisions to ensure that the deletion speed is fast each time and it is convenient to intersperse other data calculation tasks in the middle.
[0012] According to the above technical solution, during the process of deleting garbage data, if the computing power of the storage node does not exceed 60%, the garbage data deletion task is suspended and other calculation tasks are interspersed to maximize the convenience for users. If the computing power of the storage node exceeds 60%, the garbage data deletion task is started to ensure that the computing power is not affected by the garbage data, significantly improving the performance of the system.
[0013] According to the above technical solution, when the user enters the store, the items marked for the second time will be pushed to the user's mobile terminal in the form of text messages, so that the items marked for the second time are always preferentially selected by the user.
[0014] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: In the present invention, by setting a task scheduling algorithm, the number of surface folders is selected according to the computable power of the selected storage nodes, realizing the coordinated operation among the storage nodes, avoiding the data delay and slow calculation phenomena caused by the unified calculation of the storage nodes. In order to realize the parallel processing of item information, the task scheduling algorithm of the present application is designed. This algorithm should consider factors such as the dependency relationship of item information, the load situation of storage nodes, and network latency to achieve the efficient allocation and scheduling of item information processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings are used to provide a further understanding of the present invention and form a part of the specification. They are used in conjunction with the embodiments of the present invention to explain the present invention and do not constitute a limitation on the present invention. In the accompanying drawings: Figure 1 is a schematic diagram of the system of the present invention. Detailed implementation manners
[0016] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.
[0017] Please refer to Figure 1 , the present invention provides a technical solution: a computing resource expansion system based on data storage, including a cloud storage system, characterized in that the cloud storage system includes a data plane and a metadata plane. Among them, the data plane is used to store user data, and the metadata plane is used to store the metadata corresponding to the data. The user data includes the data volume and the access volume. The increase in the data volume and access volume of the user will cause an increase in the number of entries and query volume stored in the metadata plane. The scalability of the metadata plane will directly affect the scalability of the entire cloud storage system.
[0018] The metadata plane includes a metadata storage base, a data directory unit one linking the metadata storage base, and a data directory unit two linking the metadata storage base.
[0019] The data directory unit one stores commodity information and also supports the formation of a commodity information directory and the formation of a commodity information file. For example: file / directory creation, search, deletion, and renaming, etc. List the goods information of each store in a file, and the files relying on the metadata storage base are displayed as L1, L2, L3,..., L N , where N is the total number of stores. By displaying the commodity information of a single store in an array, it is convenient for data search and data transmission.
[0020] The data directory unit two stores a list of the location information of the commodity files and additionally adds the transmission path of each commodity file, providing a path for each commodity file to convert the computing system. Each store has an independent computing system. Regarding a store as a storage node, each storage node needs to have sufficient computing power and storage capacity to support the processing of computing tasks and the storage of data. The design of the storage node should consider the balance between computing and storage, as well as the communication and collaborative processing between nodes. Set each storage node as A1, A2, A3,..., A N , L1, L2, L3,..., L Ncorresponds one by one to A1, A2, A3, ..., A N corresponds one by one; A task scheduling algorithm is set in each storage node, that is, the computing power in each storage node is set to C. Here, the computing power is the expected computable power. C is divided according to the operation load of 0 - 100%. 0 means the storage node has the strongest computing power, and 100% means the storage node has the weakest computing power. Different load standards are set during the operation division of 0 - 100%: 0 - 60%: The computing power of the store is excellent, meeting the standard for the full start of the task scheduling algorithm; 60% - 90%: The computing power of the store is fully loaded, meeting the standard for the partial start of the task scheduling algorithm; 90% - 100%: The computing power of the store is overloaded and does not meet the standard for the start of the task scheduling algorithm.
[0021] The computing power of each store is a fixed value, with a maximum of 100%. When the computing power of the A1 storage node is within the range of 0 - 60%, it means that the A1 storage node has the ability to calculate the commodity files of the other storage nodes. When the computing power of the A1 storage node is within the range of 60% - 90%, it means that the A1 storage node has the ability to calculate part of the commodity files of the other storage nodes. When the computing power of the A1 storage node is within the range of 90% - 100%, it means that the A1 storage node does not have the ability to calculate the commodity files of the other storage nodes. The other storage nodes are consistent with the A1 storage node based on the standard of the task scheduling algorithm.
[0022] When the A1 storage node meets the standard for the full start of the task scheduling algorithm, the other storage nodes can, when their own computing power is higher than 60%, transfer the commodity files originally belonging to themselves to the storage node within the range of 0 - 60%. Then the storage node calculates the files and returns the calculation results along the transmission path.
[0023] If the situation where its own computing power is higher than 90% occurs and the computing power of all other storage nodes is higher than 60%, the task scheduling algorithm preferentially selects the storage node in the range of 60% - 90%. Then, according to the strength of the current computing power of the storage nodes, it selects the storage node with the strongest computing power, transfers the file part of its own storage node to this storage node for calculation, and then returns the calculation results along the transmission path; Each file is divided into several file blocks, and each file block is stored in a surface folder. Several surface folders constitute a bottom folder. The file part in the present invention is the surface folder. The number of surface folders is selected according to the computable ability of the selected storage nodes to achieve coordinated operation among storage nodes, avoiding the data delay and slow calculation caused by the unified calculation of storage nodes. In order to achieve parallel processing of commodity information, the task scheduling algorithm of this application is designed. This algorithm should consider factors such as the dependency relationship of commodity information, the load condition of storage nodes, and network latency to achieve efficient allocation and scheduling of commodity information processing.
[0024] Since the processing of commodity information is distributed among multiple storage nodes and there may be data dependency relationships among commodity information, data consistency maintenance is set in the task scheduling algorithm. Data consistency maintenance can ensure that the execution results of commodity information on different nodes are consistent and guarantee the correctness and reliability of data. When the calculation result is returned through the transmission path, this calculation result is independently stored and backed up. This independently stored and backed up calculation result can be obtained and used. When the computing power of its own storage node is lower than 60%, the calculation result is verified. Data loss and other situations may occur during the transmission of the calculation result. If the verification result is correct, the calculation result is continued to be used. If the verification result is incorrect, the result is recalculated through its own storage node and notified by the system.
[0025] The data volume of user data includes the identity information entered when the user enters the store and the user information calculated from the data surface. The identity information of each user includes the frequency of the user entering the store and the purchase record of goods. The user information calculated from the data surface includes the average frequency of the user entering the store, the time since the last entry into the store, and the expected purchase of goods. The expected purchased items are calculated based on the purchased item records. The items purchased by the user are marked for the first time. The purchase habit database of the user is formed through the first seven purchase records. The items purchased by the user multiple times are marked for the second time, and the first mark is deleted. The first mark is the mark for the formation of the user's purchase habit, and the marked items are not completely the items the user likes to use. The items marked for the second time are the items repeatedly purchased by the user. When the user logs in to the store app, the items marked for the second time are displayed on the home page for the user to directly obtain the purchase channels. At the same time, the items related to the items marked for the second time are recommended on the home page or the second page to ensure the maximization of the utilization rate of the item information on the home page. To ensure the stability of the computing power, the method of mark deletion is used to reduce the proportion of the data occupying the computing power range. The marks to be deleted will ultimately be divided into multiple small marks, and the information of the cancelled users also needs to be deleted. The user information to be deleted is divided into multiple information blocks. The small marks and information blocks are stored in the garbage data area. The method of deleting one by one is used to shorten the length of the continuous garbage data area. We disperse the pressure of data deletion, split the garbage data area into multiple small pieces, and at the same time enhance the ability of a single storage node to process garbage data to alleviate its impact on the query performance. The length of the garbage data is reduced by two divisions to ensure that the deletion speed is fast each time and it is convenient to intersperse other data calculation tasks in the middle.
[0026] During the process of deleting garbage data, if the computing power of the storage node does not exceed 60%, the garbage data deletion task is suspended and other calculation tasks are interspersed to maximize the convenience for the user. If the computing power of the storage node exceeds 60%, the garbage data deletion task is started to ensure that the computing power is not affected by the garbage data and significantly improve the performance of the system.
[0027] When the user enters the store, the items marked for the second time will be pushed to the user's mobile terminal in the form of text messages, so that the items marked for the second time are always preferentially selected by the user.
[0028] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device.
[0029] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.
Claims
1. A computing resource expansion system based on data storage, including a cloud storage system, characterized in that, The cloud storage system includes a data plane and a metadata plane. The data plane is used to store user data, and the metadata plane is used to store the meta-information corresponding to the data. The user data includes the data volume and the access volume. The increase in the data volume and access volume of the user will cause an increase in the number of entries and query volume stored in the metadata plane. The scalability of the metadata plane will directly affect the scalability of the entire cloud storage system; The metadata plane includes a metadata storage base, a data directory unit one linking the metadata storage base, and a data directory unit two linking the metadata storage base; The data directory unit one stores commodity information and supports the formation of a commodity information directory and the formation of a commodity information file; The data directory unit two stores a list of location information of commodity files and additionally adds the transmission path of each commodity file, providing a path for each commodity file to be converted into a computing system. Each store has an independent computing system; Set a store as a storage node. Each storage node needs to have sufficient computing power and storage capacity to support the processing of computing tasks and the storage of data. The design of the storage node should take into account the balance between computing and storage, as well as the communication and collaborative processing between nodes. Set each storage node as A1, A2, A3, ..., A N； A task scheduling algorithm is set in each storage node, that is, the computing power in each storage node is set to C. Here, the computing power is the expected computable power. C is divided according to the operation load of 0-100%. 0 means the storage node has the strongest computing power, and 100% means the storage node has the weakest computing power. Different load standards are set during the operation division of 0-100%: 0-60%: The computing power of the store is excellent, meeting the standard for full startup of the task scheduling algorithm; 60%-90%: The computing power of the store is fully loaded, meeting the standard for partial startup of the task scheduling algorithm; 90%-100%: The computing power of the store is overloaded and does not meet the standard for startup of the task scheduling algorithm; The computing power of each store is a fixed value, with a maximum of 100%. When the computing power of the A1 storage node is within the range of 0-60%, it means that the A1 storage node has the ability to calculate the commodity files of the remaining storage nodes. When the computing power of the A1 storage node is within the range of 60%-90%, it means that the A1 storage node has the ability to calculate part of the commodity files of the remaining storage nodes. When the computing power of the A1 storage node is within the range of 90%-100%, it means that the A1 storage node does not have the ability to calculate the commodity files of the remaining storage nodes. The remaining storage nodes are consistent with the A1 storage node based on the standard of the task scheduling algorithm; Each file is divided into several file blocks. Each file block is stored in a surface folder. Several surface folders form a bottom folder. The file part, that is, the surface folder, selects the number of surface folders according to the computable power of the selected storage node; Since the processing of commodity information is carried out on multiple storage nodes, and there may be data dependencies between commodity information, data consistency maintenance is set in the task scheduling algorithm. Data consistency maintenance can ensure that the execution results of commodity information on different nodes are consistent, and guarantee the correctness and reliability of data. When the calculation result is returned through the transmission path, this calculation result is independently stored and backed up, and this independently stored backup can be obtained and used. When the computing power of its own storage node is less than 60%, the calculation result is verified. Data loss and other situations may occur during the transmission of the calculation result. If the verification result is correct, the calculation result is continued to be used. If the verification result is incorrect, the result is recalculated through its own storage node and notified by the system.
2. The computing resource expansion system based on data storage according to claim 1, wherein The data volume of the user data includes the identity information entered when the user enters the store and the user information calculated from the data surface. The identity information of each user includes the frequency of the user entering the store and the purchase record of the goods. The user information calculated from the data surface includes the average frequency of the user entering the store, the time since the last entry into the store, and the expected purchase of goods; The expected purchase of goods is calculated based on the purchase record. The goods purchased by the user are marked for the first time. The purchase habit database of the user is formed through the first seven purchase records, and the goods purchased by the user multiple times are marked for the second time, and the first mark is deleted. The first mark is the mark for the formation of the user's purchase habit, and the marked goods are not completely the goods that the user likes to use. The goods marked for the second time are the goods repeatedly purchased by the user. When the user logs in to the store app, the goods marked for the second time are displayed on the home page for the user to directly obtain the purchase channel. At the same time, the goods related to the goods marked for the second time are recommended on the home page or the second page to ensure the maximum utilization rate of the commodity information on the home page. To ensure the stability of the computing power.
3. The computing resource expansion system based on data storage according to claim 2, characterized in that, The method of marking and deleting is used to reduce the proportion of data occupying the computing power range. The marks to be deleted will eventually be divided into multiple small marks, and the information of the deleted users also needs to be deleted. The user information to be deleted is divided into multiple information blocks. The small marks and information blocks are stored in the garbage data area. The method of deleting one by one is used to shorten the length of the continuous garbage data area. We disperse the pressure of data deletion, split the garbage data area into multiple small pieces, and at the same time enhance the ability of a single storage node to process garbage data, alleviating its impact on the query performance. The length of the garbage data is reduced by two divisions to ensure that the deletion speed is fast each time and it is convenient to intersperse other data calculation tasks in the middle.
4. A computing resource extension system based on data storage according to claim 1, wherein During the process of deleting garbage data, if the computing power of the storage node does not exceed 60%, the garbage data deletion task is suspended and other computing tasks are interspersed to maximize the convenience for users. If the computing power of the storage node exceeds 60%, the garbage data deletion task is started to ensure that the computing power is not affected by garbage data and significantly improve the performance of the system.
5. A computing resource expansion system based on data storage according to claim 1, wherein When a user enters the store, the second marked product will be pushed to the user's mobile device via SMS, ensuring that the second marked product is always preferentially selected by the user.