Compression storage method, decompression method and related device for crowd data

Abstract

The embodiment of the invention provides a crowd data compression storage method, a crowd data decompression method and related devices. The crowd data comprises a plurality of personal data, and generating a personal vector representation corresponding to the personal data; performing compression coding on the personal vector representation to obtain coded data; wherein in the process of carrying out compression coding on the personal vector representations, the incidence relation among the multiple personal vector representations is identified, and information contained in coded data can express the incidence relation; and storing the coded data corresponding to the crowd data. Personal vector representation is generated through corresponding personal data, and association relationships among a plurality of personal vectors are identified in a compression coding process, so that coded data can effectively express the association relationships, and compression distortion of crowd data is weakened.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of data compression and storage, and in particular to a method for compressing and storing crowd data, a method for decompressing crowd data, and related devices. Background Art

[0002] With the rapid development of big data technology and the continuous growth of people's demand for data mining, the acquisition of crowd data has become a hot topic in the field of e-commerce marketing. Crowd data has the characteristics of high collection frequency and large data size. Since storing massive crowd data will occupy a lot of hardware resources and consume high bandwidth resources during crowd data transmission, data compression processing is usually required before storing and transmitting crowd data.

[0003] Traditional data compression methods such as Huffman coding, arithmetic coding and LZ compression algorithm mainly perform data compression based on the statistical characteristics of data to achieve efficient compression of crowd data. However, after compressing crowd data in the existing technology, there is serious compression distortion, resulting in a large amount of information loss in the compressed crowd data. Summary of the invention

[0004] In view of this, multiple embodiments of the present application are dedicated to providing a compression storage method, a decompression method and related devices for crowd data, which can reduce compression distortion to a certain extent.

[0005] One embodiment of the present application provides a method for compressing and storing crowd data, wherein the crowd data includes multiple personal data, and the method includes: generating a personal vector representation corresponding to the personal data; performing compression encoding on the personal vector representation to obtain encoded data; wherein, in the process of performing compression encoding on the personal vector representation, identifying the association relationship between the multiple personal vector representations, and making the information contained in the encoded data able to express the association relationship; and storing the encoded data corresponding to the crowd data.

[0006] One embodiment of the present application also provides a method for decompressing crowd data, comprising: receiving the encoded data as described in the aforementioned method for compressing and storing crowd data; in the process of decoding the encoded data, decoding and obtaining the multiple personal vector representations according to the association relationship between the multiple personal vector representations expressed by the encoded data.

[0007] An embodiment of the present application also provides a compression storage device for crowd data, wherein the crowd data includes multiple personal data, and the compression storage device includes: a vector generation module, used to generate a personal vector representation corresponding to the personal data; an encoding module, used to perform compression encoding on the personal vector representation to obtain encoded data; wherein, in the process of compressing and encoding the personal vector representation, the association relationship between the multiple personal vector representations is identified, and the information contained in the encoded data can express the association relationship; a storage module, used to store the encoded data corresponding to the crowd data.

[0008] One embodiment of the present application also provides a decompression device for crowd data, the decompression device comprising: a receiving module for receiving the encoded data as described in the aforementioned compression and storage method for crowd data; a decoding module for decoding the encoded data according to the association relationship between the multiple personal vector representations expressed by the encoded data to obtain the multiple personal vector representations during the decoding process of the encoded data.

[0009] One embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the aforementioned method.

[0010] One embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer program, and when the at least one computer program is executed by a processor, the method as described above can be implemented.

[0011] One embodiment of the present application further provides a computer program product, which is used to implement the aforementioned method.

[0012] In multiple implementations provided in the present application, personal vector representations are generated by corresponding personal data, and the associations between multiple personal vectors are identified during the compression encoding process, so that the encoded data can effectively express these associations, thereby reducing the compression distortion of the crowd data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram of an application scenario of a method for compressing and storing crowd data provided in one embodiment of the present application.

[0014] Figure 2 A flowchart of a method for compressing and storing crowd data provided in accordance with one embodiment of the present application.

[0015] Figure 3 A schematic diagram of generating a personal vector representation provided for one embodiment of the present application.

[0016] Figure 4 A functional schematic diagram of a multi-head attention mechanism provided for one embodiment of the present application.

[0017] Figure 5 A functional schematic diagram of a linear transformation layer provided for one embodiment of the present application.

[0018] Figure 6 A flowchart of a method for decompressing crowd data provided in accordance with one embodiment of the present application.

[0019] Figure 7 A functional schematic diagram of a crowd data decompression device provided for one embodiment of the present application.

[0020] Figure 8 A flowchart of a training method for a specified coding model provided for one embodiment of the present application.

[0021] Fig. 9 A schematic diagram of a module of a compression storage device for crowd data provided in one embodiment of the present application.

[0022] Fig.10 A schematic diagram of a module of a crowd data decompression device provided in one embodiment of the present application.

[0023] Fig.11 A schematic diagram of a computer device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0025] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0026] In the related art, in the field of crowd data compression and storage, traditional data compression methods usually rely on the statistical characteristics of data and reduce the amount of data by eliminating redundant information. Specifically, traditional data compression methods such as Huffman coding, arithmetic coding and LZ compression algorithm mainly achieve compression by analyzing the frequency distribution and repetitive patterns of data. Although these methods can reduce data storage space and optimize transmission bandwidth to a certain extent, they are mainly based on the statistical characteristics of data and often ignore the semantics and contextual relationships contained in crowd data.

[0027] With the rapid development of big data technology and the diversification of data collection methods, the scale and complexity of crowd data have shown explosive growth. When faced with massive and diverse crowd data, traditional data compression methods usually use a higher compression ratio for crowd data in order to reduce storage space. However, such compression storage methods will cause some key information to be lost during the compression process. Furthermore, there will be a certain amount of information loss during the decompression and recovery process, resulting in a large amount of key information being lost in the crowd data after decompression.

[0028] In the related technologies, deep learning technology can be used to perform data compression, especially data compression methods based on neural networks have shown better results in capturing complex data relationships and performing efficient feature extraction. However, data compression methods based on deep learning are mostly focused on specific data types such as images and texts. There is still insufficient research on the compression of multi-dimensional and multi-attribute data types such as crowd data, especially in how to effectively capture and utilize the correlation between data to improve compression efficiency and data restoration accuracy.

[0029] In summary, relevant technologies have the problem of compression distortion of crowd data when dealing with the compression and storage of large-scale, high-dimensional and complexly related crowd data.

[0030] In multiple implementations provided in the present application, the compression storage method of crowd data can be applied to a compression storage device for crowd data. The compression storage device for crowd data can be an electronic device with certain computing power and network access capabilities. The electronic device can be a desktop computer, a laptop computer, a tablet computer, a smart phone, or a server. The server can also be a distributed server, including multiple processors, memories, network communication modules, etc. that work together to achieve various functions. Alternatively, the server can also be a server cluster formed by several servers, with higher computing and data processing capabilities. With the development of science and technology, the server can also be implemented using new forms of technical means, such as a new type of "server" based on quantum computing. Of course, in some implementations, the compression storage device for crowd data can also be a program module running in an electronic device.

[0031] See also Figure 1 One embodiment of the present application provides an example of an application scenario of a method for compressing and storing crowd data. The method for compressing and storing crowd data is applied to a compression storage device for crowd data to achieve efficient compression and storage of user data on an e-commerce platform, and can reduce compression distortion of crowd data during compression and storage.

[0032] For example, in a large e-commerce platform, when users shop through the client or website, a large amount of personal data is generated. This personal data includes basic information of users (such as age, gender, region), consumption behavior (such as browsing history, purchase history, shopping cart content), and interest preferences (such as favorite product categories, evaluation content, etc.). This data needs to be effectively stored and analyzed to support accurate marketing strategies and personalized recommendation systems.

[0033] In this scenario example, the large e-commerce platform can manage the large amount of personal data generated through the marketing data management system. Specifically, the server of the marketing data management system can run a data collection service that automatically collects personal data, and the data collection service can call a compression storage device to process the large amount of personal data collected. After receiving the personal data, the compression storage device generates a personal vector representation, and identifies the association between the personal vector representations during the compression encoding process, and finally generates the encoded data and stores it in the database for subsequent data analysis, user segmentation, and formulation of precision marketing strategies.

[0034] In this scenario example, by calling a compression storage device to compress a large amount of personal data of e-commerce platform users and storing the compressed encoded data in a database, it is possible to identify complex correlations in personal data during the data compression process, thereby reducing compression distortion during the compression process and retaining more information in the compressed population data.

[0035] See also Figure 2 One embodiment of the present application provides a method for compressing and storing crowd data. The method for compressing and storing crowd data may include the following steps: Step S110: Generate a personal vector representation corresponding to the personal data.

[0036] Step S120: compress and encode the personal vector representation to obtain encoded data; wherein, in the process of compressing and encoding the personal vector representation, the association relationship between multiple personal vector representations is identified, and the information contained in the encoded data can express the association relationship.

[0037] Step S130: storing the coding data corresponding to the crowd data.

[0038] In this embodiment, the crowd data may include personal data of multiple users, and the personal data of a user may include information and statistical data related to the user. For example, the user's basic information, consumption behavior, and interest preferences. Personal data can be used to analyze, understand, and segment target customers. Alternatively, crowd data can also generate samples for training machine learning models, which are used to train machine learning models. After generating a personal vector representation corresponding to the personal data, the personal vector representation can effectively represent the relevant information of each user.

[0039] Specifically, the compression storage device can convert high-dimensional, sparse personal data into low-dimensional, dense personal vector representations through a deep learning model to facilitate subsequent compression coding processing. The compression storage device can identify the association between multiple personal vector representations through a deep learning model and compress and encode the personal vector representations to obtain encoded data, so that the encoded data occupies less storage space while retaining the complex associations and semantic messages between personal data.

[0040] In this embodiment, there will be certain association information between multiple personal data, and this association information plays an important role in the subsequent use process. In this embodiment, taking the deep learning model as a Transformer model as an example, in the process of compressing and encoding the personal vector representations of multiple personal data, the Transformer model will learn the association relationship between the personal vector representations of multiple personal data, and can form corresponding encoded data according to the learned knowledge. So that the finally obtained encoded data can carry the said association relationship, thereby reducing the compression distortion of the data. Furthermore, after decoding the encoded data, the vector representation of the personal data can be restored more accurately. In some embodiments, for example, personal data may include but is not limited to users browsing product information, purchasing products, and evaluating products through the client or website of the e-commerce platform.

[0041] In this embodiment, the compression storage device can obtain personal data such as basic information, browsing history, purchase history, and product evaluation of multiple users. Further, the compression storage device can generate a personal vector representation corresponding to the personal data. Specifically, the compression storage device can generate a personal vector representation corresponding to the personal data based on a machine learning algorithm. For example, the machine learning algorithm can include but is not limited to the BERT algorithm or the Word2Vec algorithm.

[0042] In some embodiments, the encoded data may be stored in a distributed database. Specifically, the compression storage device may obtain the personal information of the users of the e-commerce platform through the marketing data management system of the e-commerce platform, and send the encoded data generated corresponding to the user personal information to the marketing data management system. The marketing data management system may store the encoded data in the distributed databases established by the e-commerce platform in different regions, thereby storing the data in a location close to the user to reduce access latency and improve user experience. The e-commerce platform may transfer the encoded data in the distributed databases in different regions to the same marketing database for further data analysis, user segmentation, and formulation of precision marketing strategies.

[0043] In some embodiments, the encoded data may also be stored in a cloud storage system. Specifically, the compression storage device may generate vectors and compress the user's personal data in the cloud by calling a remotely deployed deep learning model server or cloud computing platform, and finally store the encoded data in the cloud storage system. Data compression and storage through the cloud platform can significantly reduce the pressure on local storage and accelerate the data compression process by utilizing the computing power of the cloud.

[0044] In multiple embodiments of the present application, by generating personal vector representations and identifying the associations between personal vectors during the compression encoding process, the encoded data can effectively express these associations, thereby improving the efficiency of data compression and the quality of data restoration after decompression.

[0045] In some embodiments, a compression storage device for crowd data can generate a primary vector representation of personal data; assign a position vector representation representing a position order to the primary vector representation; and combine the primary vector representation and the position vector representation to obtain the personal vector representation.

[0046] See also Figure 3 In this embodiment, the compression storage device can receive personal data of multiple users in the crowd data, and convert the multiple personal data (for example, the user's consumption behavior and interest preferences) contained in the crowd data into a primary vector representation through an embedding matrix. Specifically, the embedding matrix can map each word in the personal data to a low-dimensional continuous vector space, thereby capturing the user information contained in the personal data. By adopting the embedding matrix, the compression storage device can convert high-dimensional, sparse personal data into a low-dimensional, dense primary vector representation.

[0047] Next, the compression storage device may arrange the order of these primary vector representations according to the similarity between the multiple primary vector representations. Specifically, the measurement of similarity may be based on the similarity of the basic information of the users, the commonality of the behavior patterns, or other relevant features. In some embodiments, for example, the ages of user A and user B are both within the age range of 20 to 25 years old, while the age of user C is within the age range of 30 to 35 years old. Then, in the primary vector representations corresponding to the personal data of user A, user B, and user C, the similarity between the primary vector representations representing the age information of user A and user B is higher than the similarity between the primary vector representations representing the age information of user A and user C or user B and user C. According to the arrangement order, the compression storage device may assign a position vector representation representing its relative position to each primary vector representation. Specifically, the compression storage device may assign its corresponding position vector representation to each primary vector representation by means of position encoding. The position vector representation contains the relative position information of different primary vector representations, which can help the deep learning model understand the order and structure of the data.

[0048] Finally, the compression storage device can combine the primary vector representation with the corresponding position vector representation by vector addition or concatenation to generate the final personal vector representation. By combining the primary vector representation and the corresponding position vector representation, the generated personal vector representation not only contains the personal information of each user, but also reflects its relative position relationship in the data set, providing more abundant and orderly data information for the subsequent compression encoding process.

[0049] By using an embedding matrix to generate a primary vector representation of personal data, and assigning a position vector representation representing its relative position to each primary vector representation, and finally combining the primary vector representation and the corresponding position vector representation to obtain a personal vector representation, this embodiment can extract the rich information contained in the user's personal data into the personal vector representation to improve the integrity of user information during the data compression process.

[0050] In some embodiments, a compression storage device for crowd data may arrange the order of multiple primary vector representations according to the similarities between the multiple primary vector representations; and assign position vector representations expressing relative positions to the primary vector representations according to the arrangement order of the multiple primary vector representations.

[0051] In this embodiment, the compression storage device may first arrange the order of the primary vector representations according to the similarity between the multiple primary vector representations. Specifically, the compression storage device may calculate the similarity between the multiple primary vector representations based on the cosine similarity, Euclidean distance or other appropriate similarity measurement methods between vectors. In some embodiments, for example, the primary vector representations generated by the personal data of user A and user B are located in a three-dimensional vector space, and the corresponding coordinates are and ,in and Can be used to represent basic information of the user such as age, and It can be used to represent the user's consumption behavior, such as the total amount of consumption on the e-commerce platform within a week. and It can be used to represent user interest preferences, such as the number of times a user searches for a certain category of goods in a month. The similarity between the primary vector representations of user A and user B It can be obtained by the Euclidean distance calculation formula: By measuring the similarity between multiple primary vector representations, the compression storage device can identify personal data with high correlation or similar behavior patterns in the feature space, so that the primary vector representations with high similarity are closer in arrangement order.

[0052] Secondly, the compression storage device can assign a unique position vector representation to each primary vector representation by a position coding method, and the position vector representation reflects its relative position in the arrangement order. Specifically, the compression storage device can generate a corresponding position vector representation for each primary vector representation according to the order after the arrangement of multiple primary vector representations, using a fixed position coding or a trainable position coding method generated by a sine function and a cosine function. By sorting and position coding the primary vector representations, the compression storage device can assign an accurate and ordered position vector representation to each primary vector representation, and then generate a final personal vector representation containing rich position information. Specifically, the sorting based on similarity can ensure the rationality and relevance of the arrangement of the primary vector representations, and the position vector representation can provide the necessary position information support, so that the final personal vector representation not only contains the user's personal information, but also reflects its relative position relationship in the data set. This structured vector representation method can provide a more ordered and more relevant data basis for the subsequent compression coding process, thereby improving the efficiency of data compression and the quality of data restoration after decompression.

[0053] This embodiment arranges the primary vector representations in order according to their similarity and assigns position vector representations representing relative positions to the primary vector representations in this order, thereby generating a structured and information-rich personal vector representation, so that key user information and association relationships can be retained during the compression and decompression process of crowd data.

[0054] In some embodiments, a compression storage device for crowd data inputs multiple personal vector representations into a specified encoding model with a multi-head attention mechanism to identify the association relationship between the multiple personal vector representations through the multi-head attention mechanism.

[0055] See also Figure 4 . In this embodiment, the compression storage device can input multiple personal vector representations into a designated coding model with a multi-head attention mechanism. The designated coding model usually adopts a deep learning architecture (such as TransformerEncoder), and its core components include a multi-head attention mechanism and a feedforward neural network. The multi-head attention mechanism allows the designated coding model to focus on different subspace features of the input vector in parallel in different attention heads, so as to capture the multi-level and multi-dimensional correlation between personal vector representations. Specifically, the compression storage device can take multiple personal vector representations as input sequences, and after processing by the multi-head attention mechanism, each attention head can independently calculate the attention weights between the input vectors and generate corresponding attention outputs. Through the multi-head attention mechanism, the designated coding model can identify and model complex correlations between personal vector representations, such as user similarities, commonalities in behavior patterns, or other relevant features.

[0056] Next, the compression storage device can pass the output data processed by the multi-head attention mechanism to the feedforward neural network for further nonlinear transformation and enhancement of features. The feedforward neural network can improve the expressiveness of feature representation and the deep learning ability of the specified encoding model through a series of linear transformations and activation functions. The data processed by the feedforward neural network will be passed to the linear transformation layer to reduce the dimensionality of high-dimensional features and generate the final encoded data. The linear transformation layer uses a low-dimensional linear matrix to reduce the dimensionality of the individual vector representation processed by the multi-head attention mechanism and the feedforward neural network to achieve further data compression. This dimensionality reduction process can not only effectively reduce the storage space requirements, but also allow the encoded data to contain sufficient semantic and association information to support the subsequent decompression and restoration process.

[0057] This implementation method inputs multiple personal vector representations into a designated encoding model with a multi-head attention mechanism, uses the multi-head attention mechanism to identify the correlation between the personal vector representations, and implements data dimensionality reduction compression through a linear transformation layer. This not only improves the efficiency of crowd data compression, but also retains the complex correlation in the crowd data, thereby improving the quality of data restoration after decompression.

[0058] In some embodiments, the designated encoding model includes a linear transformation layer; the linear transformation layer uses a low-dimensional linear matrix to reduce the dimension of the individual vector representation processed by the multi-head attention mechanism to obtain encoded data.

[0059] See also Figure 5 . In this embodiment, the compression storage device performs dimensionality reduction processing on the personal vector representation after being processed by the multi-head attention mechanism through the linear transformation layer in the specified coding model. The linear transformation layer may include one or more low-dimensional linear matrices, which can be used to convert the high-dimensional personal vector representation into low-dimensional encoded data. The compression storage device can implement matrix multiplication operations through low-dimensional linear matrices to map the high-dimensional vector output by the multi-head attention mechanism to a lower-dimensional space, thereby effectively reducing the dimension of the data and the storage space requirements. The design of the low-dimensional linear matrix can be based on pre-trained specified coding model parameters, so as to retain the key information and association relationships in the original population data as much as possible.

[0060] Next, the compression storage device can further process the reduced-dimensional personal vector representation through activation functions and normalization processing. The activation function can use linear or nonlinear activation functions (such as Relu, Sigmoid, etc.) to introduce nonlinear features and enhance the expression ability of the encoded data and the fitting ability of the model. The compression storage device can standardize the reduced-dimensional personal vector representation through normalization processing, so that the encoded data has consistent scale and distribution characteristics between different dimensions, so as to improve the data stability and reliability in the subsequent storage and decompression process.

[0061] In some embodiments, the compression storage device may use a trainable low-dimensional linear matrix, and continuously optimize the parameters of the low-dimensional linear matrix during the training process of the specified coding model through the back-propagation algorithm to further improve the dimensionality reduction effect of the individual vector representation and the quality of the coded data. In addition, the compression storage device can also flexibly adjust the dimension of the dimensionality reduction and the structure of the low-dimensional linear matrix according to different application requirements and data characteristics to improve the compression effect of the crowd data and the restoration accuracy during decompression.

[0062] This implementation introduces a linear transformation layer into the specified coding model and uses a low-dimensional linear matrix to reduce the dimension of the individual vector representation after the multi-head attention mechanism, thereby generating information-rich coded data. This method can improve the efficiency of crowd data compression and retain the key information and associations in the original crowd data, thereby accurately restoring the key information and associations of the original crowd data during the decompression and recovery of the coded data.

[0063] See also Figure 6 The present application also provides a method for decompressing crowd data. The method for decompressing crowd data may include the following steps: Step S210: Receive the encoded data as described in any of the above.

[0064] Step S220: in the process of decoding the coded data, decoding is performed to obtain the multiple personal vector representations according to the association relationship between the multiple personal vector representations expressed by the coded data.

[0065] In this embodiment, the crowd data decompression method can be applied to the crowd data decompression device. The crowd data decompression device can be an electronic device with certain computing power and network access capabilities. The electronic device can be a desktop computer, a laptop computer, a tablet computer, a smart phone, or a server. The server can also be a distributed server, including multiple processors, memories, network communication modules, etc., which work together to achieve various functions. Alternatively, the server can also be a server cluster formed by several servers, with higher computing and data processing capabilities. With the development of science and technology, the server can also be implemented by new forms of technical means, such as a new type of "server" based on quantum computing. Of course, in some embodiments, the crowd data decompression device can also be a program module running in an electronic device.

[0066] See also Figure 7 . In this embodiment, the crowd data decompression device can receive encoded data stored in a distributed database or a cloud storage system. The encoded data refers to the personal vector representation after compression and encoding, and contains the association relationship information between multiple personal vectors. These association relationship information can be used to accurately restore the personal vector representation during the decompression process. The crowd data decompression device can decode the received encoded data based on a deep learning model (such as the Decoder of the Transformer model) according to the association relationship between multiple personal vector representations expressed by the encoded data. In some embodiments, the crowd data decompression device can use the Decoder of the Transformer model to analyze the similarities and dependencies between the personal vector representations in the encoded data, and restore the original high-dimensional personal vector representation through a multi-head attention mechanism and an inverse linear transformation process, so that the decompressed personal vector representation can accurately reflect the characteristics and association relationships of the original personal data.

[0067] In some embodiments, the crowd data decompression device can achieve efficient data decompression by calling a local or remotely deployed deep learning model server. Specifically, the enterprise can use the computing resources of the cloud service provider to download the encoded data stored in the cloud to the local, use the locally deployed deep learning model to decompress the data, and finally use the restored personal vector representation for data analysis, user segmentation and the formulation of precision marketing strategies. This method can not only make full use of the computing power of the cloud to improve the decompression efficiency of crowd data, but also improve the security of crowd data during the decompression process.

[0068] This embodiment receives coded data and decodes the received coded data based on a deep learning model according to the association relationship between multiple individual vector representations expressed by the coded data to restore the original individual vector representation. This method can retain key features and association relationships during the decompression process of crowd data, thereby improving the accuracy of crowd data decompression.

[0069] See also Figure 8 The embodiment of the present application also provides a training method for a specified coding model. The specified coding model may adopt a deep learning architecture (such as a Transformer model), and the training method for the specified coding model may include the following steps: Step S310: pre-process the collected crowd data to construct a training set.

[0070] Step S320: training a designated coding model based on the training set.

[0071] In this embodiment, the training method of the specified coding model can be applied to the training device of the specified coding model. The training device of the specified coding model can be an electronic device with certain computing power and network access capability. The electronic device can be a desktop computer, a laptop computer, a tablet computer, a smart phone, or a server. The server can also be a distributed server, including multiple processors, memories, network communication modules, etc., which work together to achieve various functions. Alternatively, the server can also be a server cluster formed by several servers, with higher computing and data processing capabilities. With the development of science and technology, the server can also be implemented by new forms of technical means, such as a new type of "server" based on quantum computing. Of course, in some embodiments, the training device of the specified coding model can also be a program module running in an electronic device.

[0072] In this embodiment, the training device of the designated coding model can widely collect a large amount of personal data generated by users when they conduct shopping, browsing and other activities on the e-commerce platform, including basic information of users (such as age, gender, region), consumption behavior (such as browsing history, purchase record, shopping cart content) and interest preferences (such as collection of commodity categories, evaluation content, etc.) as original population data. Next, the training device of the designated coding model can pre-process the collected original population data: for example, the Z-Score or IQR method can be used to clean the original population data to remove noise and outliers; then, a normalization algorithm such as Min-Max scaling or Z-score standardization can be applied to make different features in the original population data after data cleaning in the same dimension for subsequent processing; next, the missing values ​​in the normalized population data can be processed by filling (such as mean, median, etc.) or deleting strategies to improve the integrity of the population data. The training device of the designated coding model can use data enhancement techniques such as jitter, translation and rotation to expand the original population data after pre-processing to obtain a high-quality training set, thereby improving the generalization ability of the designated coding model. In addition, the training device of the specified encoding model can select a loss function (such as a cross entropy loss function) to measure the difference between the data in the constructed training set and the original population data, and use the Adam optimizer to train the specified encoding model, adjusting the learning rate and batch size to optimize the training process.

[0073] See also Fig. 9 The present application also provides a device for compressing and storing crowd data. The device for compressing and storing crowd data may include: A vector generation module, for generating a personal vector representation corresponding to personal data; an encoding module, configured to compress and encode the personal vector representation to obtain encoded data; wherein, in the process of compressing and encoding the personal vector representation, an association relationship between multiple personal vector representations is identified, and information contained in the encoded data is enabled to express the association relationship; The storage module is used to store the coded data corresponding to the crowd data.

[0074] In this embodiment, the specific functions and effects achieved by the compression storage device for crowd data can be explained with reference to other embodiments of the present application and will not be described in detail here.

[0075] See also Fig.10 The present application also provides a crowd data decompression device. The crowd data decompression device may include: A receiving module, used to receive the coded data as described above; A decoding module is used to decode and obtain the multiple personal vector representations according to the association relationship between the multiple personal vector representations expressed by the encoded data during the decoding process of the encoded data.

[0076] See also Fig.11 The embodiment of the present application further provides a computer device, the computer device comprising: a memory and a processor, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the above method.

[0077] The embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor implements the aforementioned method.

[0078] The present application also provides a computer program product comprising instructions, and when the computer program product is executed by a processor, the method as described above is implemented.

[0079] The user information or user account information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, etc.) involved in multiple implementation methods of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0080] It should be understood that the specific examples in this article are only intended to help those skilled in the art to better understand the embodiments of the present application, rather than to limit the scope of the present invention.

[0081] It can be understood that in the various implementations of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the implementation methods of the present application.

[0082] It can be understood that the various embodiments described in this application can be implemented individually or in combination, and the embodiments of this application are not limited to this.

[0083] Unless otherwise stated, all technical and scientific terms used in the embodiments of the present application have the same meaning as those generally understood by those skilled in the art of the technical field of the present application. The terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the scope of the present application. The term "and / or" used in the present application includes any and all combinations of one or more related listed items. The singular forms of "a kind of", "above" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0084] It can be understood that the processor of the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method implementation can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (DigitalSignal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiment of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0085] It is understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (programmable ROM, PROM), an erasable programmable read-only memory (erasable PROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0086] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0087] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0088] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0089] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0090] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0091] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0092] The above is only a specific implementation of the present application, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for compressing and storing crowd data, characterized in that: The crowd data includes a plurality of personal data, and the method includes: Generate a personal vector representation corresponding to the personal data; The personal vector representation is compressed and encoded to obtain encoded data, including: inputting multiple personal vector representations into a specified encoding model with a multi-head attention mechanism to identify the association relationship between the multiple personal vector representations through the multi-head attention mechanism; wherein, in the process of compressing and encoding the personal vector representation, the association relationship between the multiple personal vector representations is identified, and the information contained in the encoded data can express the association relationship; The coded data corresponding to the crowd data is stored.

2. The method according to claim 1, characterized in that The steps of generating a personal vector representation corresponding to the personal data include: Generate a primary vector representation of personal data; Assigning a position vector representation representing a position order to the primary vector representation; The primary vector representation and the position vector representation are combined to obtain the personal vector representation.

3. The method according to claim 2, characterized in that The step of assigning a position vector representation representing a position order to the primary vector representation comprises: Arranging the order of the plurality of primary vector representations according to the similarities between the plurality of primary vector representations; According to the arrangement order of the plurality of primary vector representations, position vector representations expressing relative positions are allocated to the primary vector representations.

4. The method according to claim 1, characterized in that: The designated coding model includes a linear transformation layer; the linear transformation layer uses a low-dimensional linear matrix to reduce the dimension of the individual vector representation processed by the multi-head attention mechanism to obtain encoded data.

5. A method for decompressing crowd data, characterized in that: The method comprises: Receiving the encoded data as claimed in any one of claims 1 to 4; In the process of decoding the coded data, the multiple personal vector representations are decoded according to the association relationship between the multiple personal vector representations expressed by the coded data.

6. A compression storage device for crowd data, characterized in that: The crowd data includes a plurality of personal data, and the compression storage device includes: A vector generation module, for generating a personal vector representation corresponding to personal data; An encoding module, configured to compress and encode the personal vector representation to obtain encoded data, comprising: inputting a plurality of personal vector representations into a designated encoding model having a multi-head attention mechanism, so as to identify the association relationship between the plurality of personal vector representations through the multi-head attention mechanism; wherein, in the process of compressing and encoding the personal vector representation, the association relationship between the plurality of personal vector representations is identified, and the information contained in the encoded data can express the association relationship; The storage module is used to store the coded data corresponding to the crowd data.

7. A crowd data decompression device, characterized in that: The decompression device comprises: A receiving module, used for receiving the encoded data as claimed in any one of claims 1 to 4; A decoding module is used to decode and obtain the multiple personal vector representations according to the association relationship between the multiple personal vector representations expressed by the encoded data during the decoding process of the encoded data.

8. A computer device, characterized in that: The computer device includes a memory and a processor, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and when the at least one computer program is executed by a processor, the method according to any one of claims 1 to 4 can be implemented.

10. A computer program product, characterized in that The computer program product is used to implement the method according to any one of claims 1 to 4.

Images