Data collection method and system

By semantic encoding of user behavior data and group compression transmission after calculating the cluster center offset coefficient on the edge side, the problem of retail enterprises losing key data in data transmission is solved, and the accuracy and efficiency of data analysis are improved.

CN118643375BActive Publication Date: 2024-11-19SHANGHAI WICRESOFT
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410827205.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-25
Publication Date
2024-11-19
Estimated Expiration
2044-06-25

AI Technical Summary

Technical Problem

Retail companies face the problem of losing a large amount of critical data during data transmission when collecting and analyzing user behavior data, resulting in incomplete data and inaccurate analysis results, and missing business opportunities and market insights.

Method used

User behavior data is collected on the edge side, semantic encoding is performed to generate semantic encoding vectors, calculate the cluster center offset coefficient, group these coefficients and perform principal component analysis data compression, and transmit compressed data to the retail industry big data analysis middle platform.

Benefits of technology

Through the optimization and compression of data itself information, the loss of data transmission is reduced, which is conducive to the analysis and mining of back-end user behavior information and improve the accuracy and efficiency of data transmission.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118643375B_ABST
    Figure CN118643375B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a data collection method and system thereof. The method includes: semantically encoding the set of acquired user behavior data to obtain a set of semantic coding vectors of user behavior data; calculating the initial cluster center of the sequence distribution of the set of semantic coding vectors of user behavior data, and calculating the semantic correlation between the set of semantic coding vectors of user behavior data and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients; based on the sequence of cluster center offset coefficients, grouping the set of semantic coding vectors of user behavior data, and performing data compression based on principal component analysis on the subsets in each group to obtain multiple compressed data, and transmitting them to the big data analysis middle station of the retail industry. In this way, the data's own information can be used to optimize and guide compression, reduce the loss of a large amount of key user behavior data during data transmission, and facilitate the analysis and mining of user behavior information by the back end.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of data collection, and in particular, to a data collection method and a system thereof. Background Art

[0002] In retail enterprises, data is a vital asset that can help enterprises better understand customers, optimize marketing strategies, and improve customer experience. However, due to the huge amount of user behavior data, limited transmission bandwidth, and high storage costs, retail enterprises face many challenges in collecting and analyzing user behavior data. For example, it is a common problem that a large amount of key user behavior data is lost during data transmission, which may lead to incomplete data, information loss, and inaccurate analysis results, which may lead to missed business opportunities and market insights.

[0003] Therefore, an optimized data acquisition scheme is desired. Summary of the invention

[0004] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, the present disclosure provides a data collection method, the method comprising:

[0006] After obtaining user authorization, a collection of user behavior data is collected on the edge side;

[0007] At the edge side, semantically encoding each user behavior data in the set of user behavior data to obtain a set of user behavior data semantic encoding vectors;

[0008] Calculating the initial cluster center of the sequence distribution of the set of semantic coding vectors of the user behavior data, and calculating the semantic association between each semantic coding vector of the user behavior data in the set of semantic coding vectors of the user behavior data and the initial cluster center of the sequence distribution to obtain a sequence of cluster center shift coefficients;

[0009] Based on the sequence of cluster center offset coefficients, the set of semantic encoding vectors of the user behavior data is grouped, and a subset in each group is compressed based on principal component analysis to obtain a plurality of compressed data;

[0010] The multiple compressed data are transmitted to the retail industry big data analysis platform.

[0011] Optionally, the initial cluster center of the sequence distribution of the set of user behavior data semantic coding vectors is calculated, and the semantic association between each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors and the initial cluster center of the sequence distribution is calculated to obtain a sequence of cluster center offset coefficients, including: calculating the mean vector of the set of user behavior data semantic coding vectors as the initial cluster center of the sequence distribution; calculating the semantic association between each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients.

[0012] Optionally, the semantic association between each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors and the initial cluster center of the sequence distribution is calculated to obtain a sequence of cluster center offset coefficients, including: performing modulation analysis on each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors to obtain a sequence of user behavior data semantic weighted bias adjusted feature vectors; performing activation processing on the sequence of user behavior data semantic weighted bias adjusted feature vectors to obtain a sequence of user behavior data semantic activation feature vectors; and respectively calculating the product between each user behavior data semantic activation feature vector in the sequence of user behavior data semantic activation feature vectors and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients.

[0013] Optionally, modulation analysis is performed on each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors to obtain a sequence of user behavior data semantic weighted bias adjusted feature vectors, including: respectively calculating the matrix product between each weight coefficient matrix and the corresponding user behavior data semantic coding vector to obtain a sequence of user behavior data semantic weighted feature vectors; adding the corresponding bias vector to the sequence of each user behavior data semantic weighted feature vector in the sequence of user behavior data semantic weighted feature vectors to obtain a sequence of user behavior data semantic weighted bias adjusted feature vectors.

[0014] Optionally, the sequence of semantically weighted bias-adjusted feature vectors of user behavior data is activated to obtain a sequence of semantically activated feature vectors of user behavior data, comprising: inputting the sequence of semantically weighted bias-adjusted feature vectors of user behavior data into The function performs activation processing to obtain a sequence of semantically activated feature vectors of the user behavior data.

[0015] Optionally, based on the sequence of cluster center offset coefficients, the set of user behavior data semantic coding vectors is grouped, and data compression based on principal component analysis is performed on a subset in each group to obtain a plurality of compressed data, including: based on the sequence of cluster center offset coefficients, the set of user behavior data semantic coding vectors is grouped to obtain a plurality of subsets of user behavior data semantic coding vectors; and a subset of each user behavior data semantic coding vector in the plurality of subsets of user behavior data semantic coding vectors is respectively compressed based on principal component analysis to obtain a plurality of compressed data.

[0016] In a second aspect, the present disclosure provides a data acquisition system, the system comprising:

[0017] A user behavior data collection module is used to collect a collection of user behavior data on the edge side after obtaining user authorization;

[0018] A semantic coding module, configured to perform semantic coding on each user behavior data in the set of user behavior data at the edge side to obtain a set of semantic coding vectors of the user behavior data;

[0019] A semantic association calculation module, used to calculate the initial cluster center of the sequence distribution of the set of semantic coding vectors of user behavior data, and calculate the semantic association between each user behavior data semantic coding vector in the set of semantic coding vectors of user behavior data and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients;

[0020] A data compression module, used for grouping the set of semantic coding vectors of the user behavior data based on the sequence of cluster center offset coefficients, and performing data compression based on principal component analysis on a subset in each group to obtain a plurality of compressed data;

[0021] The compressed data transmission module is used to transmit the multiple compressed data to the retail industry big data analysis platform.

[0022] Optionally, the semantic association calculation module includes: a mean vector calculation unit, used to calculate the mean vector of the set of user behavior data semantic coding vectors as the initial cluster center of the sequence distribution; a semantic association calculation unit, used to calculate the semantic association between each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients.

[0023] Optionally, the semantic association calculation unit includes: a modulation analysis subunit, used to perform modulation analysis on each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors to obtain a sequence of user behavior data semantic weighted bias adjusted feature vectors; an activation processing subunit, used to perform activation processing on the sequence of user behavior data semantic weighted bias adjusted feature vectors to obtain a sequence of user behavior data semantic activation feature vectors; and an offset coefficient calculation subunit, used to respectively calculate the product between each user behavior data semantic activation feature vector in the sequence of user behavior data semantic activation feature vectors and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients.

[0024] Optionally, the modulation analysis subunit includes: a matrix product calculation secondary subunit, used to respectively calculate the matrix products between each weight coefficient matrix and the corresponding user behavior data semantic encoding vector to obtain a sequence of user behavior data semantic weighted feature vectors; a vector addition secondary subunit, used to add the corresponding bias vector to the sequence of each user behavior data semantic weighted feature vector in the sequence of user behavior data semantic weighted feature vectors to obtain a sequence of user behavior data semantic weighted bias adjusted feature vectors.

[0025] The above technical solution is adopted to obtain a set of semantic coding vectors of user behavior data by semantically coding the set of acquired user behavior data; the initial cluster center of the sequence distribution of the set of semantic coding vectors of user behavior data is calculated, and the semantic correlation between the set of semantic coding vectors of user behavior data and the initial cluster center of the sequence distribution is calculated to obtain a sequence of cluster center offset coefficients; based on the sequence of cluster center offset coefficients, the set of semantic coding vectors of user behavior data is grouped, and the subsets in each group are compressed based on principal component analysis to obtain multiple compressed data, and transmitted to the retail industry big data analysis center. In this way, the data's own information can be used to optimize and guide compression, reduce the loss of a large amount of key user behavior data during data transmission, and facilitate the back-end analysis and mining of user behavior information.

[0026] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0028] Figure 1The present invention is a flow chart of a data collection method according to an exemplary embodiment.

[0029] Figure 2 It is a block diagram of a data acquisition system according to an exemplary embodiment.

[0030] Figure 3 It is a block diagram of an electronic device according to an exemplary embodiment.

[0031] Figure 4 It is an application scenario diagram of a data collection method according to an exemplary embodiment. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0033] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0034] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0035] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0036] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0037] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0038] In order to solve the above problems, the present disclosure provides a data collection method and system thereof, which semantically encodes the set of acquired user behavior data to obtain a set of semantic encoding vectors of user behavior data; calculates the initial cluster center of the sequence distribution of the set of semantic encoding vectors of user behavior data, and calculates the semantic association between the set of semantic encoding vectors of user behavior data and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients; based on the sequence of cluster center offset coefficients, groups the set of semantic encoding vectors of user behavior data, and performs data compression based on principal component analysis on the subsets in each group to obtain multiple compressed data, and transmits them to the big data analysis center of the retail industry. In this way, the data's own information can be used to optimize and guide compression, reduce the loss of a large amount of key user behavior data during data transmission, and facilitate the analysis and mining of user behavior information by the back end.

[0039] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0040] Figure 1 is a flow chart of a data collection method according to an exemplary embodiment. Figure 1 As shown, the method includes:

[0041] Step 101: After obtaining user authorization, collect a set of user behavior data on the edge side;

[0042] Step 102: At the edge side, semantically encode each user behavior data in the set of user behavior data to obtain a set of user behavior data semantic encoding vectors;

[0043] Step 103: Calculate the initial cluster center of the sequence distribution of the set of user behavior data semantic coding vectors, and calculate the semantic association between each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors and the initial cluster center of the sequence distribution to obtain a sequence of cluster center shift coefficients;

[0044] Step 104: grouping the set of user behavior data semantic encoding vectors based on the sequence of cluster center offset coefficients, and performing data compression based on principal component analysis on a subset in each group to obtain a plurality of compressed data;

[0045] Step 105: Transmit the multiple compressed data to the retail industry big data analysis platform.

[0046] In response to the above technical problems, a data collection method is proposed in the technical solution of the present application, which can cluster user behavior data on the edge side before data transmission, and then compress the clustered user behavior data before data transmission. In this way, the data's own information can be used to optimize and guide compression, reduce the loss of a large amount of key user behavior data during data transmission, and facilitate the back-end analysis and mining of user behavior information. In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country of residence and with the authorization of the owner of the corresponding device.

[0047] Specifically, in the technical solution of the present application, after obtaining user authorization, a set of user behavior data is collected on the edge side. Next, it is considered that the original user behavior data may be unstructured, such as logs, click streams, sensor data, etc. At the same time, user behavior data usually contains a large number of details and features, and directly using the original data may lead to the complexity of data processing and storage. Based on this, in the technical solution of the present application, on the edge side, each user behavior data in the set of user behavior data is semantically encoded to obtain a set of user behavior data semantic encoding vectors. By semantically encoding each user behavior data in the set of user behavior data, the key semantic features and patterns in each user behavior data can be captured respectively, and these unstructured user behavior data can be converted into a structured data format for further processing and analysis. This helps to reduce the complexity of the data and make each user behavior data easier to understand and use.

[0048] Furthermore, an associated clustering analysis is performed on the semantics of each user behavior data, so as to utilize the semantic information of each user behavior data itself to optimize the guided compression and reduce the situation where a large amount of key user behavior data is lost during data transmission. In the clustering algorithm, determining the initial cluster center is the first step to start the clustering process. If the initial cluster center is not selected properly, the clustering results may deviate from the actual data distribution. Based on this, in the technical solution of the present application, the mean vector of the set of semantic encoding vectors of the user behavior data is further calculated as the initial cluster center of the sequence distribution. It should be understood that the mean vector is a general description of the entire set of semantic encoding vectors of user behavior data, providing an average position based on all data points, which can represent the central trend in the data set. As the initial cluster center, the mean vector helps guide the clustering algorithm to better distinguish different data clusters, so that each cluster is representative. In addition, by using the mean vector as the initial cluster center, the propagation of errors can be reduced, the accuracy and stability of the clustering results can be improved, and the convergence process of the clustering algorithm can be accelerated.

[0049] In one embodiment of the present disclosure, the initial cluster center of the sequence distribution of the set of user behavior data semantic coding vectors is calculated, and the semantic association between each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors and the initial cluster center of the sequence distribution is calculated to obtain a sequence of cluster center offset coefficients, including: calculating the mean vector of the set of user behavior data semantic coding vectors as the initial cluster center of the sequence distribution; calculating the semantic association between each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients.

[0050] Then, in order to be able to measure the semantic association between each user behavior data semantic coding feature in the set of user behavior data semantic coding vectors and the initial cluster center of the sequence distribution, the similarity or proximity between each user behavior semantic and the initial cluster center is quantified, so as to determine to which cluster subset each user behavior data semantic belongs. Based on this, in the technical solution of the present application, the semantic association between each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors and the initial cluster center of the sequence distribution is further calculated to obtain a sequence of cluster center offset coefficients. By calculating the semantic association between each user behavior data semantic coding vector and the initial cluster center of the sequence distribution, the correlation and offset degree of each data point relative to the cluster center can be quantified. This measurement reflects the relative position of the data point in the feature space, which helps to identify data points that deviate far from the cluster center and data points in the cluster, and is also conducive to the subsequent grouping and compression processing of user behavior data semantics.

[0051] In one embodiment of the present disclosure, the semantic association between each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors and the initial cluster center of the sequence distribution is calculated to obtain a sequence of cluster center offset coefficients, including: performing modulation analysis on each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors to obtain a sequence of user behavior data semantic weighted bias adjusted feature vectors; performing activation processing on the sequence of user behavior data semantic weighted bias adjusted feature vectors to obtain a sequence of user behavior data semantic activation feature vectors; and respectively calculating the product between each user behavior data semantic activation feature vector in the sequence of user behavior data semantic activation feature vectors and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients.

[0052] Furthermore, in one embodiment of the present disclosure, modulation analysis is performed on each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors to obtain a sequence of user behavior data semantic weighted bias adjusted feature vectors, including: respectively calculating the matrix product between each weight coefficient matrix and the corresponding user behavior data semantic coding vector to obtain a sequence of user behavior data semantic weighted feature vectors; adding the corresponding bias vector to the sequence of each user behavior data semantic weighted feature vector in the sequence of user behavior data semantic weighted feature vectors to obtain a sequence of user behavior data semantic weighted bias adjusted feature vectors.

[0053] Furthermore, in one embodiment of the present disclosure, the sequence of semantically weighted bias-adjusted feature vectors of user behavior data is activated to obtain a sequence of semantically activated feature vectors of user behavior data, including: inputting the sequence of semantically weighted bias-adjusted feature vectors of user behavior data into The function performs activation processing to obtain a sequence of semantically activated feature vectors of the user behavior data.

[0054] Specifically, the semantic association between each user behavior data semantic encoding vector in the set of user behavior data semantic encoding vectors and the initial cluster center of the sequence distribution is calculated using the following association formula to obtain the sequence of cluster center offset coefficients; wherein the association formula is:

[0055] in, is the first in the set of semantic encoding vectors of the user behavior data user behavior data semantic encoding vector, is the initial cluster center of sequence distribution, represents the weight coefficient matrix, is the bias vector, express function, is the first in the sequence of cluster center offset coefficients The cluster center shift coefficient.

[0056] Furthermore, considering that each cluster center offset coefficient reflects the degree of offset between each user behavior data semantic encoding vector and the cluster center, that is, the offset coefficient sequence provides a quantitative method to determine the distance between the data point and the cluster center. Through the offset coefficient, the data points with a high correlation with the specific cluster center can be identified, and these points can be grouped together to form different clusters or subsets. Based on this, in the technical solution of the present application, the set of user behavior data semantic encoding vectors is further grouped based on the sequence of cluster center offset coefficients to obtain a subset of multiple user behavior data semantic encoding vectors. By grouping based on the cluster center offset coefficient, it is clear which data points belong to the same cluster, thereby defining the boundaries of the cluster. In other words, the offset coefficient can help identify the core members and edge members of the cluster, which is very important for understanding the characteristics of the cluster and conducting further data analysis. At the same time, through grouping processing based on the cluster center offset coefficient, the structure of the cluster can also be optimized, so that the data points in each cluster have a higher similarity, while the clusters have a higher difference. This helps to better distinguish different types of user behavior data, thereby achieving more refined data classification analysis and compressed transmission. The specific steps are as follows: set a threshold range, determine whether the offset between the semantic encoding vector of each user behavior data and the cluster center exceeds the threshold, and decide whether it needs to be divided into a new subset. When grouping, you can start from the starting point of the sequence and traverse each vector in the user behavior data semantic encoding vector set one by one; for each vector, determine whether it meets the grouping range based on its offset from the cluster center; if it meets the grouping principle, divide the vector into the corresponding subset, otherwise, continue to traverse the next vector. In this way, the entire vector set is processed step by step until all vectors are divided into corresponding subsets.

[0057] Then, the principal component analysis-based data compression is performed on each subset of the semantic coding vectors of the plurality of user behavior data in the subsets of the semantic coding vectors to obtain a plurality of compressed data. That is to say, the principal component analysis semantics between the semantic features of the plurality of user behavior data in each cluster is used to perform data compression processing, so as to obtain a plurality of compressed data reflecting the key semantic features of user behavior in the centers of the plurality of clusters, and then the plurality of compressed data are transmitted to the big data analysis center of the retail industry. In this way, the user behavior data can be clustered on the edge side before data transmission, and then the clustered user behavior data can be compressed before data transmission. In this way, the data's own information can be used to optimize and guide compression, thereby reducing the loss of a large amount of key user behavior data during data transmission, which is beneficial to the back-end analysis and mining of user behavior information.

[0058] In one embodiment of the present disclosure, based on the sequence of cluster center offset coefficients, the set of user behavior data semantic coding vectors is grouped, and data compression based on principal component analysis is performed on a subset in each group to obtain a plurality of compressed data, including: based on the sequence of cluster center offset coefficients, the set of user behavior data semantic coding vectors is grouped to obtain a plurality of subsets of user behavior data semantic coding vectors; and a subset of each user behavior data semantic coding vector in the plurality of subsets of user behavior data semantic coding vectors is respectively compressed based on principal component analysis to obtain a plurality of compressed data.

[0059] In the technical solution of the present application, each user behavior data semantic encoding vector in the set of user behavior data semantic encoding vectors respectively represents the text semantic association features of each user behavior data. Considering that each user behavior data in the set of user behavior data has source text differences in the data source domain, each user behavior data semantic encoding vector in the set of user behavior data semantic encoding vectors will also have more significant feature distribution differences due to the source text differences, which makes the initial cluster center of the sequence distribution obtained by calculating the mean vector of the set of user behavior data semantic encoding vectors have the problem of insufficient aggregation of user behavior semantic space expression, thereby affecting the accuracy of the semantic association between each user behavior data semantic encoding vector and the initial cluster center of the sequence distribution.

[0060] In a preferred embodiment, the semantic association between each user behavior data semantic encoding vector in the set of user behavior data semantic encoding vectors and the initial cluster center of the sequence distribution is calculated to obtain a sequence of cluster center offset coefficients, including the following steps: cascading each user behavior data semantic encoding vector in the set of user behavior data semantic encoding vectors into a user behavior data semantic association joint representation vector; processing the user behavior data semantic association joint representation vector according to the following formula to obtain an optimized user behavior data semantic association joint representation vector, wherein the formula is:

[0061]

[0062] in, represents the semantic association joint representation vector of the user behavior data, is the length of the semantic association joint representation vector of the user behavior data, is the norm of the semantic association joint representation vector of the user behavior data, represents the binary norm of the semantic association joint representation vector of the user behavior data, represents the balanced hyperparameter, It means point multiplication by position. Indicates adding by position point. It means to subtract by position point. Indicates the calculation of square root, Representing an optimized user behavior data semantic association joint representation vector; splitting the optimized user behavior data semantic association joint representation vector into individual optimized user behavior data semantic encoding vectors based on a cascade of individual user behavior data semantic encoding vectors in the set of user behavior data semantic encoding vectors; and calculating the semantic association between the initial cluster center of the sequence distribution and each optimized user behavior data semantic encoding vector to obtain a sequence of the semantic associations.

[0063] Accordingly, in this preferred embodiment, the present application targets the set of user behavior data semantic coding vectors as a feature set, wherein the change semantics represented by the characteristic value of the set of user behavior data semantic coding vectors is expressed in units. In order to dynamically aggregate the semantic set composed of different change semantics of the set of user behavior data semantic coding vectors without ignoring the individual semantic change information, the individual features of the set of user behavior data semantic coding vectors and the aggregate scale of the set of user behavior data semantic coding vectors are expressed as the feature full amplitude and half amplitude, and the low-rank negative correlation of different dimensions of the overall semantics of the feature set of the set of user behavior data semantic coding vectors is used as the phase and scaled to dynamically adjust the change relationship of the semantic content of the set of user behavior data semantic coding vectors, thereby improving the aggregation of the overall semantic information expression of the feature set of the set of user behavior data semantic coding vectors, and improving the calculation accuracy of the semantic correlation between each user behavior data semantic coding vector and the initial cluster center of the sequence distribution. In this way, it can help to cluster and compress the data more accurately in the future, thereby reducing the loss of a large amount of key user behavior data during data transmission, which is beneficial to the analysis and mining of user behavior information by the back end.

[0064] In summary, the above solution can cluster user behavior data on the edge before data transmission, and then compress the clustered user behavior data before data transmission. In this way, the data's own information can be used to optimize and guide compression, reducing the loss of a large amount of key user behavior data during data transmission, which is beneficial to the back-end analysis and mining of user behavior information.

[0065] Figure 2 FIG. 1 is a block diagram of a data acquisition system according to an exemplary embodiment. Figure 2 As shown, the system 200 includes:

[0066] The user behavior data collection module 201 is used to collect a set of user behavior data on the edge side after obtaining user authorization;

[0067] A semantic coding module 202, configured to perform semantic coding on each user behavior data in the set of user behavior data at the edge side to obtain a set of semantic coding vectors of user behavior data;

[0068] The semantic association calculation module 203 is used to calculate the initial cluster center of the sequence distribution of the set of semantic coding vectors of the user behavior data, and calculate the semantic association between each user behavior data semantic coding vector in the set of semantic coding vectors of the user behavior data and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients;

[0069] A data compression module 204, configured to group the set of user behavior data semantic encoding vectors based on the sequence of cluster center offset coefficients, and perform data compression based on principal component analysis on a subset in each group to obtain a plurality of compressed data;

[0070] The compressed data transmission module 205 is used to transmit the multiple compressed data to the retail industry big data analysis platform.

[0071] In one embodiment of the present disclosure, the semantic association calculation module includes: a mean vector calculation unit, used to calculate the mean vector of the set of user behavior data semantic coding vectors as the initial cluster center of the sequence distribution; a semantic association calculation unit, used to calculate the semantic association between each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients.

[0072] In one embodiment of the present disclosure, the semantic association calculation unit includes: a modulation analysis subunit, which is used to perform modulation analysis on each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors to obtain a sequence of user behavior data semantic weighted bias adjusted feature vectors; an activation processing subunit, which is used to perform activation processing on the sequence of user behavior data semantic weighted bias adjusted feature vectors to obtain a sequence of user behavior data semantic activation feature vectors; and an offset coefficient calculation subunit, which is used to respectively calculate the product between each user behavior data semantic activation feature vector in the sequence of user behavior data semantic activation feature vectors and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients.

[0073] In one embodiment of the present disclosure, the modulation analysis subunit includes: a matrix product calculation secondary subunit, which is used to respectively calculate the matrix products between each weight coefficient matrix and the corresponding user behavior data semantic encoding vector to obtain a sequence of user behavior data semantic weighted feature vectors; a vector addition secondary subunit, which is used to add the corresponding bias vector to the sequence of each user behavior data semantic weighted feature vector in the sequence of user behavior data semantic weighted feature vectors to obtain a sequence of user behavior data semantic weighted bias adjusted feature vectors.

[0074] Under the wave of big data, the retail industry's demand for self-development has become increasingly obvious. Along with the pace of Internetization, there is a need for lean analysis of massive data. The concept of "people, goods, and places" proposed for the retail industry is becoming popular. As the three core analysis dimensions of the retail industry, people, goods, and places play a more important role under the data-driven concept. In the digitalization process of the pan-retail industry, the identification of consumer needs is the core, and the use of data to guide the lean operation of enterprises and optimize the supply chain of enterprises can truly address the following long-term pain points in the retail industry.

[0075] Pain point 1: Large data granularity makes it difficult to accurately analyze users. Every retail company has accumulated data for a long time in the traditional model and has explored data to varying degrees. They have built their own databases and have monthly reports to convey information. However, due to the difficulty in data collection and the difficulty in obtaining and connecting online and offline traffic, traditional data analysis methods have poor timeliness, large data granularity, and cannot accurately analyze users. A large amount of user behavior data is lost, making it impossible to make further progress in refined operations and accurately understand the buyer's psychology, thereby increasing sales and expanding the market.

[0076] Pain point 2: Information asymmetry makes it difficult to manage sales representatives. Brands, distributors, and shopping malls and supermarkets form a sales model. Sales representatives need to regularly patrol stores, understand buyers' needs in a timely manner, reasonably control goods distribution strategies, help distributors meet manufacturers' sales targets, and achieve sales efficiency and maximize profits. As a link between manufacturers and the front line, sales representatives play a vital role, so scientific management of sales representatives has also become a major difficulty in the retail industry. How to quickly understand the work of sales representatives, standardize the performance assessment of sales representatives, and how to master the store patrol routes, the traditional model has always played a limited role.

[0077] Pain point 3: Digital transformation, e-commerce platform operation is difficult. In the process of digital transformation, major traditional retailers have built their own e-commerce apps / applets. Whether they are for consumers or dealers, they have taken exploratory steps on the road to expanding online channels. However, it is accompanied by the difficulty of connecting offline data with online data, the difficulty of compatibility between online operation models and traditional offline sales models, and the problem of insufficient online operation experience. Only by flexibly integrating e-commerce gameplay and reasonably connecting online and offline can seamless transformation be achieved.

[0078] In one embodiment of the present disclosure, a company retail industry big data intelligent mining and analysis platform is provided to design and establish a complete cloud-native enterprise data platform infrastructure and technical framework; design and implement key data models and business indicator systems for general retail businesses; design, develop and integrate the core component systems of the data platform, namely data storage, data integration, data analysis and visualization, as well as supporting data authority and security management.

[0079] Key technologies include:

[0080] Big data collection and storage technologies, including data warehouses / data lakes, distributed storage, etc., such as using data streams such as Kafka, Flume, and Flink to collect data and enter the data lake;

[0081] Data fusion and integration technologies, including ETL, data virtualization, etc., such as using Spark Streaming for real-time data computing;

[0082] Clean, filter, integrate and associate data to establish a unified data asset view;

[0083] Data analysis and visualization technologies, including third-party BI and self-service report analysis tools, as well as data dimension extraction and intelligent analysis model construction.

[0084] The main business modules to be built on the system platform include: 1) overview and analysis platform; 2) intelligent operation; 3) data management.

[0085] The related technologies based on Microsoft cloud platform Azure mainly include: Azure Kubernetes, Azure ServiceFabric, Azure Data Lake, Azure Storage Container, Microsoft PowerBI, etc., which ensure the security, stability and scalability of the system.

[0086] The construction achievements will mainly include:

[0087] 1) Build the overall facilities and architecture of the system platform to realize the collection and capture of multi-business application front-end (APP / Web / applet) and back-end log data;

[0088] 2) Data warehouse and data analysis models, indicator systems, visual analysis reports, etc. based on the inherent logic of the business and data application scenarios;

[0089] 3) Support and manage data-driven intelligent operations in the retail industry.

[0090] Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0091] like Figure 3 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0092] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0093] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0094] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0095] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0096] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0097] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, a test parameter acquisition module may also be described as a "module for acquiring device test parameters corresponding to a target device".

[0100] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0101] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0102] Figure 4 FIG. 1 is an application scenario diagram of a data collection method according to an exemplary embodiment. Figure 4 As shown, in this application scenario, first, after obtaining user authorization, a collection of user behavior data is collected on the edge side (for example, Figure 4 Then, the obtained user behavior data set is input into a server deployed with a data collection algorithm (for example, Figure 4 In S) shown in , the server is capable of processing the set of user behavior data based on a data collection algorithm to perform data compression based on principal component analysis to obtain multiple compressed data; and transmit the multiple compressed data to the retail industry big data analysis middle platform.

[0103] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0104] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0105] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A data collection method, characterized in that: include: After obtaining user authorization, a collection of user behavior data is collected on the edge side; At the edge side, semantically encoding each user behavior data in the set of user behavior data to obtain a set of user behavior data semantic encoding vectors; Calculating the initial cluster center of the sequence distribution of the set of semantic coding vectors of the user behavior data, and calculating the semantic association between each semantic coding vector of the user behavior data in the set of semantic coding vectors of the user behavior data and the initial cluster center of the sequence distribution to obtain a sequence of cluster center shift coefficients; Based on the sequence of cluster center offset coefficients, the set of semantic encoding vectors of the user behavior data is grouped, and a subset in each group is compressed based on principal component analysis to obtain a plurality of compressed data; Transmitting the plurality of compressed data to a retail industry big data analysis platform; The step of calculating the initial cluster center of the sequence distribution of the set of semantic coding vectors of the user behavior data, and calculating the semantic association between each semantic coding vector of the user behavior data in the set of semantic coding vectors of the user behavior data and the initial cluster center of the sequence distribution to obtain a sequence of cluster center shift coefficients includes: Calculating the mean vector of the set of semantic encoding vectors of the user behavior data as the initial cluster center of the sequence distribution; Calculating the semantic association between each user behavior data semantic encoding vector in the set of user behavior data semantic encoding vectors and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients; The step of calculating the semantic association between each user behavior data semantic encoding vector in the set of user behavior data semantic encoding vectors and the initial cluster center of the sequence distribution to obtain the sequence of cluster center offset coefficients includes: Performing modulation analysis on each user behavior data semantic encoding vector in the set of user behavior data semantic encoding vectors to obtain a sequence of user behavior data semantic weighted bias adjusted feature vectors; Performing activation processing on the sequence of semantically weighted bias-adjusted feature vectors of user behavior data to obtain a sequence of semantically activated feature vectors of user behavior data; The product of each user behavior data semantic activation feature vector in the sequence of user behavior data semantic activation feature vectors and the initial cluster center of the sequence distribution is calculated respectively to obtain a sequence of cluster center shift coefficients.

2. The data collection method according to claim 1, characterized in that: Performing modulation analysis on each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors to obtain a sequence of user behavior data semantic weighted bias adjusted feature vectors, including: Calculating the matrix product between each weight coefficient matrix and the corresponding user behavior data semantic encoding vector to obtain a sequence of user behavior data semantic weighted feature vectors; A corresponding bias vector is added to each sequence of semantically weighted feature vectors of user behavior data in the sequence of semantically weighted feature vectors of user behavior data to obtain a sequence of semantically weighted bias-adjusted feature vectors of user behavior data.

3. The data collection method according to claim 2, characterized in that: The sequence of semantically weighted bias-adjusted feature vectors of user behavior data is activated to obtain a sequence of semantically activated feature vectors of user behavior data, including: inputting the sequence of semantically weighted bias-adjusted feature vectors of user behavior data into The function performs activation processing to obtain a sequence of semantically activated feature vectors of the user behavior data.

4. The data collection method according to claim 3, characterized in that: Based on the sequence of cluster center offset coefficients, the set of user behavior data semantic encoding vectors is grouped, and a subset in each group is compressed based on principal component analysis to obtain a plurality of compressed data, including: Based on the sequence of cluster center offset coefficients, grouping the set of user behavior data semantic encoding vectors to obtain a plurality of subsets of user behavior data semantic encoding vectors; Data compression based on principal component analysis is performed on each subset of the user behavior data semantic encoding vectors in the subsets of the plurality of user behavior data semantic encoding vectors to obtain a plurality of compressed data.

5. A data acquisition system, characterized in that: include: A user behavior data collection module is used to collect a collection of user behavior data on the edge side after obtaining user authorization; A semantic coding module, configured to perform semantic coding on each user behavior data in the set of user behavior data at the edge side to obtain a set of semantic coding vectors of the user behavior data; A semantic association calculation module, used to calculate the initial cluster center of the sequence distribution of the set of semantic coding vectors of user behavior data, and calculate the semantic association between each user behavior data semantic coding vector in the set of semantic coding vectors of user behavior data and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients; A data compression module, used for grouping the set of semantic coding vectors of the user behavior data based on the sequence of cluster center offset coefficients, and performing data compression based on principal component analysis on a subset in each group to obtain a plurality of compressed data; A compressed data transmission module, used to transmit the plurality of compressed data to a retail industry big data analysis platform; Wherein, the semantic relevance calculation module includes: A mean vector calculation unit, used to calculate the mean vector of the set of semantic encoding vectors of the user behavior data as the initial cluster center of the sequence distribution; A semantic association calculation unit, used to calculate the semantic association between each user behavior data semantic encoding vector in the set of user behavior data semantic encoding vectors and the initial cluster center of the sequence distribution to obtain a sequence of cluster center offset coefficients; Wherein, the semantic relevance calculation unit includes: A modulation analysis subunit, configured to perform modulation analysis on each user behavior data semantic coding vector in the set of user behavior data semantic coding vectors to obtain a sequence of user behavior data semantic weighted bias adjusted feature vectors; An activation processing subunit, configured to perform activation processing on the sequence of semantically weighted bias-adjusted feature vectors of user behavior data to obtain a sequence of semantically activated feature vectors of user behavior data; The shift coefficient calculation subunit is used to respectively calculate the product between each user behavior data semantic activation feature vector in the sequence of user behavior data semantic activation feature vectors and the initial cluster center of the sequence distribution to obtain the sequence of cluster center shift coefficients.

6. The data acquisition system according to claim 5, characterized in that: The modulation analysis subunit comprises: A matrix product calculation secondary subunit is used to calculate the matrix product between each weight coefficient matrix and the corresponding user behavior data semantic encoding vector to obtain a sequence of user behavior data semantic weighted feature vectors; The vector addition secondary subunit is used to add the corresponding bias vector to each sequence of user behavior data semantic weighted feature vectors in the sequence of user behavior data semantic weighted feature vectors to obtain the sequence of user behavior data semantic weighted bias adjusted feature vectors.