Data storage method and device

Abstract

The invention discloses a data storage method and device, and relates to the technical field of distributed storage. The specific implementation mode comprises the steps of obtaining data generated by a business scene, wherein the types of the data comprise behavior data of a user on an APP, hardware data, merchant data and commodity data of a merchant; according to the type of the data, determining a data processing mode for processing to obtain to-be-fragmented data, the data processing mode at least comprising one of caching, verification, cleaning and encryption processing; performing fragmentation processing on the to-be-fragmented data based on the type of the data and a multi-dimensional feature to obtain a data fragment, the multi-dimensional feature including one or more of time, a geographic position and a user type; and storing the data fragments into storage nodes of the distributed storage system according to a preset data storage logic, and recording a storage relationship between the data fragments and the storage nodes. According to the method, data storage and business scenes are organically combined, so that more reliable and convenient support is provided for actual business requirements.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to the field of distributed storage technology, and in particular to a data storage method and device. Background Art

[0002] Traditional centralized data storage technology has many problems, such as concurrent performance bottlenecks, load imbalance, single point failure risk, disaster recovery difficulties, single point security risk, privacy protection mechanism, low scalability, high cost, complex permission management, data tampering risk and complex record association.

[0003] Distributed storage technology is more fault-tolerant and flexible than centralized data storage technology, but distributed storage databases use redundant storage strategies, which are more expensive. Summary of the invention

[0004] Aiming at the data storage optimization problem in a distributed storage database, a data storage method, device, electronic device and storage medium are provided.

[0005] According to a first aspect, a data storage method is provided, comprising: acquiring data generated by a business scenario, the data types including user behavior data on an APP, hardware data, merchant data, and merchant product data; determining a data processing method according to the data type for processing to obtain data to be sharded, the data processing method including at least one of the following: caching, verification, cleaning, and encryption processing; sharding the data to be sharded based on the data type and multidimensional features to obtain data shards, the multidimensional features including one or more of time, geographic location, and user type; storing the data shards in storage nodes of a distributed storage system according to a preset data storage logic, and recording the storage relationship between the data shards and the storage nodes.

[0006] In combination with the first aspect, in some examples, a data processing method is determined according to the type of the data to obtain the data to be sharded, including: in response to the type of the data being merchant data; caching the merchant data, and after integrity verification, encrypting it to obtain the data to be sharded.

[0007] In combination with the first aspect, in some examples, a data processing method is determined according to the type of the data to obtain data to be sharded, including: in response to the type of the data being behavior data, hardware data, and merchant product data; verifying the correlation between the behavior data and the hardware data; and cleaning and encrypting the verified behavior data, hardware data, and product data associated with the verified behavior data to obtain data to be sharded.

[0008] In combination with the first aspect, in some examples, the verified behavior data, hardware data, and product data associated with the verified behavior data are cleaned and encrypted to obtain data to be sharded, including: cleaning the verified behavior data, hardware data, and product data associated with the verified behavior data through a large model to obtain user portrait data and product portrait data, wherein the user portrait data includes the user's shopping preference information, and the product portrait data includes the consumer group corresponding to the product; determining the association between the user portrait data and the product portrait data and encrypting them to obtain the data to be sharded.

[0009] In combination with the first aspect, in some examples, determining the association relationship between the user portrait data and the product portrait data includes: determining the similarity between the user portrait data and the product portrait data; respectively determining the geographic location of the user in the user portrait and the geographic location of the merchant corresponding to the product in the product portrait data; and establishing the association relationship between the user portrait data and the product portrait data based on the similarity and the distance between the geographic locations.

[0010] In combination with the first aspect, in some examples, the data to be sharded is sharded based on the type and multidimensional features of the data to obtain data shards, including: sharding the data to be sharded according to the association between the user portrait data and the product portrait data to obtain data shards.

[0011] In combination with the first aspect, in some examples, the data to be sharded is sharded according to the association between the user portrait data and the product portrait data to obtain data shards, including: sharding the user portrait data and the product portrait data whose association reaches a preset threshold as a sharding dimension to obtain data shards; sharding the user portrait data and the product portrait data whose association does not reach the preset threshold according to the geographic location as a sharding dimension to obtain data shards.

[0012] According to a third aspect, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a method as in any embodiment of the data storage method.

[0013] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method of any embodiment of the data storage method is implemented.

[0014] According to the solution of the present application, various types of data generated in business scenarios are collected and processed together. By centrally processing various types of complex data, redundant data can be effectively removed. Different data processing methods are used for different types of data, and then the data is sharded and stored based on the multi-dimensional characteristics of the data, organically combining data storage and business scenarios, thereby providing more reliable and convenient support for actual business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0016] Figure 1 is an exemplary system architecture diagram to which some embodiments of the present application may be applied;

[0017] Figure 2 is a flow chart of an embodiment of a data storage method according to the present application;

[0018] Figure 3 is a schematic diagram of an application scenario of the data storage method according to the present application;

[0019] Figure 4 is a flow chart of another embodiment of the data storage method according to the present application;

[0020] Figure 5 is a schematic structural diagram of an embodiment of a data storage device according to the present application;

[0021] Figure 6 It is a block diagram of an electronic device used to implement the data storage method of an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of the data storage method or data storage device of the present application can be applied.

[0025] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0026] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as video applications, live broadcast applications, instant messaging tools, email clients, social platform software, etc.

[0027] The terminal devices 101, 102, 103 here can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0028] The server 105 may be a server that provides various services, such as a background server that provides support for the terminal devices 101, 102, and 103. The background server may analyze and process the received data generated by the business scenario.

[0029] It should be noted that the data storage method provided in the embodiment of the present application can be executed by the server 105 or the terminal devices 101, 102, 103, and accordingly, the data storage device can be set in the server 105 or the terminal devices 101, 102, 103.

[0030] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0031] Continue to refer Figure 2 , shows a process 200 of an embodiment of a data storage method according to the present application. The data storage method comprises the following steps:

[0032] Step 201, obtaining data generated by the business scenario, wherein the data types include user behavior data on the APP, hardware data, merchant data, and merchant product data.

[0033] In this embodiment, the execution subject (eg Figure 1 The server or terminal device shown) can obtain the data generated in the business scenario in a variety of ways.

[0034] In one example, data is obtained through sensors through smart hardware (such as MR glasses), including interaction data during the user's use of the smart hardware and physical quantity data of each sensor. The consumption, meeting, and social data of the user during the use of the APP are obtained through the APP; the general background server obtains merchant data and merchant product data, etc.

[0035] Step 202, determining a data processing method according to the type of the data to obtain the data to be sharded, wherein the data processing method includes at least one of the following: caching, verification, cleaning and encryption processing.

[0036] In this embodiment, the above-mentioned execution body processes different data types respectively to obtain data to be fragmented.

[0037] In some examples, in response to the data type being merchant data, the execution entity caches the merchant data, and then performs integrity verification and encryption processing to obtain the data to be sharded.

[0038] In other examples, the above-mentioned execution subject responds to the data type being behavior data, hardware data, and merchant product data, and then processes the data in a general cross-validation manner. Specifically, the above-mentioned execution subject verifies the correlation between the behavior data and the hardware data, for example, whether the verification data is within a reasonable range. According to the standby time of the smart hardware, the behavior data should not exceed the standby time limit. Finally, the verified behavior data, hardware data, and product data associated with the verified behavior data are cleaned and encrypted to obtain the data to be sharded.

[0039] Step 203: Slice the data to be sliced ​​based on the type and multi-dimensional features of the data to obtain data slices, wherein the multi-dimensional features include one or more of time, geographic location, and user type.

[0040] In this embodiment, data sharding can adopt hash sharding or range sharding, etc., and multi-dimensional data can adopt multi-level sharding processing, etc. The embodiment of the present application does not impose any special restrictions on this.

[0041] Step 204: store the data shards in storage nodes of the distributed storage system according to a preset data storage logic, and record the storage relationship between the data shards and the storage nodes.

[0042] In this embodiment, the above-mentioned execution entity will divide the data into high-frequency access data and low-frequency access data according to the access frequency and importance of the data, store the high-frequency access data on the high-performance storage node SSD, and store the low-frequency access data on the low-performance storage node HDD.

[0043] Therefore, various types of data generated in business scenarios are collected and processed together. By centrally processing various types of complex data, redundant data can be effectively removed. Different data processing methods are used for different types of data, and then the data is sharded and stored based on the multi-dimensional characteristics of the data, organically combining data storage and business scenarios, thereby providing more reliable and convenient support for actual business needs.

[0044] Continue to see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an application scenario of the data storage method according to this embodiment. Figure 3 In the application scenario, the execution subject 301 obtains all the data 302 of the business scenario and performs unified processing. The data of the business scenario includes the user's behavior data on the APP, hardware data, merchant data and merchant's product data. Then, the data processing method 303 is determined according to the type of data to obtain the data to be sharded 304. The data processing method includes at least one or more of caching, verification, cleaning and encryption processing.

[0045] Finally, the data slices are stored in the storage nodes of the distributed storage system according to the preset data storage logic, and the storage relationship between the data slices and the storage nodes is recorded.

[0046] Further references Figure 4 , which shows a process 400 of another embodiment of a data storage method. The process 400 includes the following steps:

[0047] Step 401, clean the verified behavior data, hardware data and product data associated with the verified behavior data through a large model to obtain user portrait data and product portrait data, wherein the user portrait data includes the user's shopping preference information, and the product portrait data includes the consumer group corresponding to the product.

[0048] In this embodiment, the big model may be an open source big model, such as L l ama, CodeGeeX, etc.

[0049] Step 402, determine the association relationship between the user portrait data and the product portrait data and encrypt them to obtain the data to be sharded.

[0050] In this embodiment, the above-mentioned execution subject determines the relationship between the user portrait data and the product portrait data mainly through similarity and geographical location. Specifically, the above-mentioned execution subject determines the similarity between the user portrait data and the product portrait data, and respectively determines the geographical location of the user in the user portrait and the geographical location of the merchant corresponding to the product in the product portrait data, and finally establishes the association relationship between the user portrait data and the product portrait data based on the similarity obtained above and the distance of the geographical location.

[0051] The embodiment of the present application does not impose any particular restriction on the method for calculating the similarity between user portrait data and product portrait data, which may be methods such as Euclidean distance and Manhattan distance.

[0052] Correspondingly, in step 203, processing the data to be sharded based on the type and multi-dimensional features of the data further includes sharding the data to be sharded according to the association between the user portrait data and the product portrait data to obtain data shards.

[0053] More specifically, the user portrait data and product portrait data whose association value reaches a preset threshold are sharded as a sharding dimension to obtain data shards; the user portrait data and product portrait data whose association value does not reach the preset threshold are sharded according to the geographic location as a sharding dimension to obtain data shards.

[0054] To sum up, by collecting and processing various types of data generated in business scenarios together, and by centrally processing various types of complex data, redundant data can be effectively removed; different data processing methods are used for different types of data, and then the data is sharded and stored based on the multi-dimensional characteristics of the data. By combining the data processing method with the data sharding, the acquired data is deeply processed using large model technology, and sharded based on the processed information, the data distributed storage method of this solution can better provide support for actual business.

[0055] Further references Figure 5 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a data storage device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown in FIG. 1 , in addition to the features described below, the device embodiment may also include Figure 2 The device can be applied to various electronic devices.

[0056] like Figure 5 As shown, the data storage device 500 of this embodiment includes: a first unit 501, a second unit 502, a third unit 503 and a fourth unit 504. Among them, the first unit 501 is used to obtain data generated by business scenarios, and the types of the data include user behavior data on the APP, hardware data, merchant data and merchant commodity data; the second unit 502 is used to determine the data processing method according to the type of the data to process and obtain the data to be sharded, and the data processing method includes at least one of the following: caching, verification, cleaning and encryption processing; the third unit 503 is used to shard the data to be sharded based on the type and multidimensional characteristics of the data to obtain data shards, and the multidimensional characteristics include one or more of time, geographic location and user type; the fourth unit 504 is used to store the data shards in the storage nodes of the distributed storage system according to the preset data storage logic, and record the storage relationship between the data shards and the storage nodes.

[0057] In this embodiment, the specific processing of the first unit 501, the second unit 502, the third unit 503 and the fourth unit 504 of the data storage device 500 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of step 201, step 202, step 203 and step 204 in the corresponding embodiment are not repeated here.

[0058] In some optional implementations of this embodiment, the second unit 502 is further used to respond to the type of the data being merchant data; cache the merchant data, perform encryption processing after integrity verification, and obtain the data to be sharded.

[0059] In some optional implementations of this embodiment, the second unit 502 is further used to respond to the type of the data being behavior data, hardware data, and merchant's product data; verify the correlation between the behavior data and the hardware data; clean and encrypt the verified behavior data, hardware data, and product data associated with the verified behavior data to obtain data to be sharded.

[0060] In some optional implementations of this embodiment, the second unit 502 is further used to clean the verified behavior data, hardware data, and product data associated with the verified behavior data through a large model to obtain user portrait data and product portrait data, wherein the user portrait data includes the user's shopping preference information, and the product portrait data includes the consumer group corresponding to the product; determine the association between the user portrait data and the product portrait data and encrypt them to obtain the data to be sharded.

[0061] In some optional implementations of this embodiment, the second unit 502 is further used to determine the similarity between the user portrait data and the product portrait data; respectively determine the geographic location of the user in the user portrait and the geographic location of the merchant corresponding to the product in the product portrait data; and establish an association relationship between the user portrait data and the product portrait data based on the similarity and the distance between the geographic locations.

[0062] In some optional implementations of this embodiment, the third unit 503 is further used to perform sharding processing on the data to be sharded according to the association relationship between the user portrait data and the product portrait data to obtain data shards.

[0063] In some optional implementations of this embodiment, the third unit 503 is further used to shard the user portrait data and product portrait data whose association relationship reaches a preset threshold as a sharding dimension to obtain data shards; and to shard the user portrait data and product portrait data whose association relationship does not reach the preset threshold according to geographic location as a sharding dimension to obtain data shards.

[0064] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0065] like Figure 6 , is a block diagram of an electronic device according to a data storage method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0066] like Figure 6As shown, the electronic device includes: one or more processors 601, memory 602, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as required. The processor can process the instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 601 is taken as an example.

[0067] The memory 602 is a non-transient computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable at least one processor to perform the data storage method provided in the present application. The non-transient computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the data storage method provided in the present application.

[0068] The memory 602 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the data storage method in the embodiment of the present application (for example, the attached Figure 5 The processor 601 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 602, that is, the data storage method in the above method embodiment is implemented.

[0069] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the data storage electronic device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 602 may optionally include a memory remotely arranged relative to the processor 601, and these remote memories may be connected to the data storage electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0070] The electronic device of the data storage method may further include: an input device 603 and an output device 604. The processor 601, the memory 602, the input device 603 and the output device 604 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0071] The input device 603 can receive input digital or character information, and generate key signal input related to user settings and function control of the data storage electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 604 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0072] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0073] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0074] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0075] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0076] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0077] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0078] The units involved in the embodiments described in the present application may be implemented by software or by hardware. The units described may also be provided in a processor, for example, may be described as: a processor includes a first unit, a second unit, a third unit, and a fourth unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the first unit may also be described as a "unit for obtaining data generated by a business scenario".

[0079] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by the device, the device: obtains data generated by the business scenario, and the type of the data includes user behavior data on the APP, hardware data, merchant data, and merchant product data; determines the data processing method according to the type of the data to process and obtain the data to be sharded, and the data processing method includes at least one of the following: caching, verification, cleaning and encryption processing; performs sharding processing on the data to be sharded based on the type and multidimensional features of the data to obtain data shards, and the multidimensional features include one or more of time, geographic location and user type; stores the data shards in the storage nodes of the distributed storage system according to the preset data storage logic, and records the storage relationship between the data shards and the storage nodes.

[0080] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A data storage method, characterized in that: The method comprises: Acquire data generated by business scenarios, including user behavior data on the APP, hardware data, merchant data, and merchant product data; Determine a data processing method according to the type of the data to process the data to obtain the data to be sharded, wherein the data processing method includes at least one of the following: caching, verification, cleaning and encryption processing; Slice the data to be sliced ​​based on the type and multidimensional features of the data to obtain data slices, wherein the multidimensional features include one or more of time, geographic location, and user type; The data slices are stored in the storage nodes of the distributed storage system according to the preset data storage logic, and the storage relationship between the data slices and the storage nodes is recorded.

2. The method according to claim 1, characterized in that Determine the data processing method according to the type of the data to process the data to obtain the data to be fragmented, including: In response to the type of the data being merchant data; The merchant data is cached, and after integrity verification, encryption processing is performed to obtain the data to be fragmented.

3. The method according to claim 1, characterized in that Determine the data processing method according to the type of the data to process the data to obtain the data to be fragmented, including: In response to the data type being behavior data, hardware data, and merchant product data; Verifying the correlation between the behavior data and the hardware data; The verified behavior data, hardware data, and commodity data associated with the verified behavior data are cleaned and encrypted to obtain the data to be sharded.

4. The method according to claim 1, characterized in that: The verified behavior data, hardware data, and commodity data associated with the verified behavior data are cleaned and encrypted to obtain the data to be sharded, including: The verified behavior data, hardware data, and commodity data associated with the verified behavior data are cleaned by a large model to obtain user portrait data and commodity portrait data, wherein the user portrait data includes the user's shopping preference information, and the commodity portrait data includes the consumer group corresponding to the commodity; Determine the association between the user portrait data and the product portrait data and encrypt them to obtain the data to be sharded.

5. The method according to claim 4, characterized in that Determining the association between the user portrait data and the product portrait data includes: Determining the similarity between the user portrait data and the product portrait data; Determine the geographical location of the user in the user portrait and the geographical location of the merchant corresponding to the product in the product portrait data respectively; An association relationship between the user portrait data and the product portrait data is established based on the similarity and the distance between the geographical locations.

6. The method according to claim 5, characterized in that The data to be sharded is sharded based on the type and multi-dimensional features of the data to obtain data shards, including: The data to be sharded is sharded according to the association between the user portrait data and the product portrait data to obtain data shards.

7. The method according to claim 6, characterized in that The data to be fragmented is fragmented according to the association between the user portrait data and the product portrait data to obtain data fragments, including: The user portrait data and the product portrait data whose association relationship reaches a preset threshold are processed as a sharding dimension to obtain data shards; The user portrait data and product portrait data whose association relationship does not reach the preset threshold are sharded according to the geographic location as a sharding dimension to obtain data shards.

8. A data storage device, characterized in that: The device comprises: The first unit is used to obtain data generated by business scenarios, where the types of data include user behavior data on the APP, hardware data, merchant data, and merchant product data; The second unit is used to determine a data processing method according to the type of the data to obtain the data to be fragmented, and the data processing method includes at least one of the following: caching, verification, cleaning and encryption processing; A third unit is used to perform sharding processing on the data to be sharded based on the type and multidimensional features of the data to obtain data shards, wherein the multidimensional features include one or more of time, geographic location and user type; The fourth unit is used to store the data shards in the storage nodes of the distributed storage system according to a preset data storage logic, and record the storage relationship between the data shards and the storage nodes.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images