A method and apparatus for data analysis

Abstract

The application discloses a data analysis method and device, and relates to the technical field of big data. A specific implementation of the method comprises the following steps: identifying public text data of a plurality of users and private text data of each user from original text data sets of the plurality of users, and obtaining a first text data set which has a first attribution relationship with a corresponding user; identifying text data in which a data theme is within a data theme range of the corresponding user from each text data except the first text data set in the original text data sets, and obtaining a second text data set which has a second attribution relationship with the corresponding user; and identifying abnormal text data in the original text data sets according to the first text data set and the second text data set. The implementation can improve the accuracy of abnormal data identification, and effectively avoid the situation that abnormal data cannot be identified by setting a threshold to analyze data and abnormal data is misjudged by public or similar features.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a method and apparatus for data analysis. Background Technology

[0002] In existing technologies, data analysis typically involves setting thresholds based on quantifiable metrics, comparing these thresholds with the data, or training neural networks using quantifiable metrics to classify the data. Because of this reliance on quantifiable metrics, the accuracy of anomaly identification is significantly affected when normal and abnormal data are difficult to distinguish through quantification. This phenomenon is particularly pronounced when identifying target users; even when uncovering relationships between users, misjudgments can occur due to identical or similar data, making it difficult to accurately identify anomalies. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a data analysis method and apparatus, which identify text data appearing in a user's original text dataset and identify text data whose data topics are within the scope of the corresponding user's data topics, thereby obtaining a text dataset that has a first attribution relationship or a second attribution relationship with the corresponding user, and identifying abnormal text data in the original text dataset based on the text dataset, thereby improving the accuracy of identifying abnormal text data.

[0004] To achieve the above objectives, according to one aspect of the present invention, a data analysis method is provided, comprising:

[0005] From the original text datasets of multiple users, the public text data of the multiple users and the private text data of each user are identified to obtain a first text dataset that has a first attribution relationship with the corresponding user;

[0006] From each text data in the original text dataset other than the first text dataset, identify the text data whose data topic is within the data topic range of the corresponding user, and obtain a second text dataset that has a second affiliation relationship with the corresponding user;

[0007] Identify anomalous text data in the original text dataset based on the first text dataset and the second text dataset.

[0008] Optionally, identifying public text data of the multiple users from their original text datasets includes:

[0009] Text data that appears in the original text datasets of the multiple users, or text data that appears in the original text datasets of users greater than or equal to a preset first threshold, shall be regarded as the public text data.

[0010] Optionally, the private text data of each user is identified from the raw text datasets of multiple users, including:

[0011] Text data that appears only in the original text dataset of one user, or text data that appears in the original text datasets of fewer than or equal to a preset second threshold of users, is considered as the private text data.

[0012] Optionally, the data analysis method further includes: using a term frequency-inverse document frequency algorithm to identify public and / or private text data in the original text datasets of the multiple users.

[0013] Optionally, text data whose data topic falls within the corresponding user's data topic range is identified, including:

[0014] Obtain the user's data topic range, determine the data topic of any text data in the user's original text dataset other than the first text dataset, and if the data topic of any text data is within the user's data topic range, treat the any text data as the user's private text data.

[0015] Optionally, the data analysis method further includes: obtaining the user's historical text dataset, and determining the user's data topic range based on the historical text dataset using topic modeling methods.

[0016] Optionally, the model for topic modeling includes one of the following: implicit Dirichlet distribution, nonnegative matrix factorization.

[0017] According to a second aspect of the present invention, an apparatus for data analysis is provided, comprising:

[0018] The first identification module is used to identify the public text data of the multiple users and the private text data of each user from the original text dataset of multiple users, so as to obtain a first text dataset with a first affiliation relationship with the corresponding user;

[0019] The second identification module is used to identify text data whose data topic is within the data topic range of the corresponding user from each text data in the original text dataset other than the first text dataset, and to obtain a second text dataset that has a second affiliation relationship with the corresponding user.

[0020] The third identification module is used to identify abnormal text data in the original text dataset based on the first text dataset and the second text dataset.

[0021] Optionally, the first identification module may use the text data that appears in the original text datasets of the multiple users, or the text data that appears in the original text datasets of users greater than or equal to a preset first threshold, as the public text data.

[0022] Optionally, the first recognition module may use text data that appears only in the original text dataset of one user, or text data that appears in the original text datasets of fewer than or equal to a preset second threshold of users, as the private text data.

[0023] Optionally, the first identification module is further configured to: identify public and / or private text data in the original text datasets of the multiple users using a term frequency-inverse document frequency algorithm.

[0024] Optionally, the second identification module obtains the user's data topic range, determines the data topic of any text data in the user's original text dataset other than the first text dataset, and if the data topic of any text data is within the user's data topic range, then the any text data is regarded as the user's private text data.

[0025] Optionally, the device further includes a pre-identification module, configured to: acquire the user's historical text dataset, and determine the user's data topic range based on the historical text dataset using a topic modeling method.

[0026] Optionally, the model for topic modeling includes one of the following: implicit Dirichlet distribution, nonnegative matrix factorization.

[0027] According to a third aspect of the present invention, an electronic device is provided, comprising:

[0028] One or more processors;

[0029] Storage device for storing one or more programs.

[0030] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0031] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0032] One embodiment of the above invention has the following advantages or beneficial effects: by identifying text data appearing in the user's original text dataset and identifying text data whose data topics fall within the corresponding user's data topic range, a text dataset with a first or second attribution relationship to the corresponding user is obtained. Based on this text dataset, abnormal text data in the original text dataset is identified, thereby accurately identifying abnormal text data. This embodiment of the invention, by analyzing the text data used by the user to form attribution relationships, can improve the accuracy of abnormal data identification without relying on quantifiable indicators, effectively avoiding situations where abnormal data cannot be identified by threshold methods when quantifiable indicators have low discriminative power, and situations where abnormal data is misjudged based on common or similar features.

[0033] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0034] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0035] Figure 1 This is a schematic diagram of the main flow of a data analysis method according to an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram illustrating the implementation idea of ​​an optional embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the main flow of a data analysis method according to a possible embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the main flow of a data analysis method according to another possible embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of the main flow of a data analysis method according to another embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of the main modules of a data analysis apparatus according to an embodiment of the present invention;

[0041] Figure 7 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0042] Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0043] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0044] Currently, anomaly identification is primarily achieved by setting thresholds or by training neural networks to cluster data. These two data analysis methods mainly target quantifiable data, such as order volume, number of order accounts, and number of shipping addresses. However, because they rely on the quality of the quantifiable data, when the differences between quantified normal and anomaly data are not significant enough, or when patterns cannot be found to differentiate them, identifying anomalies using these methods fails to reflect the true situation, resulting in inaccurate identification results and low reference value.

[0045] Furthermore, the two data analysis methods mentioned above are not suitable for identifying outliers in non-quantifiable data. For example, identifying outliers in the logistics process is still in its early stages. The volume of text data in the logistics process is enormous, but there are no unified standards, making it difficult to quantify. Existing technologies cannot accurately identify outliers in such data.

[0046] In view of this, according to one aspect of the present invention, a method for data analysis is provided.

[0047] Figure 1 This is a schematic diagram of the main flow of a data analysis method according to an embodiment of the present invention. Figure 1 As shown, the data analysis method according to an embodiment of the present invention includes:

[0048] Step S101: Identify the public text data of the multiple users and the private text data of each user from the original text dataset of multiple users to obtain a first text dataset that has a first affiliation relationship with the corresponding user.

[0049] The raw text dataset includes text data input by users or generated through user-executed commands. The text data in the raw text dataset has different dimensions; taking logistics and transportation as an example, it includes dimensions such as sender address, sender's mobile phone number, sender, product category, and order account. Public and private text data are part of the raw text dataset.

[0050] The first text dataset includes each user's public and / or private text data. The first text dataset can have data structures such as arrays, linked lists, or maps. When the first text dataset uses an array or linked list data structure, the first element of the array or the first node of the linked list stores the user's basic information. Subsequent array elements or linked list nodes store the user's public and / or private text data. By retrieving the first element of the array or the first node of the linked list, the user's basic information can be determined, thereby determining the user's public and / or private text data. When the first text dataset uses a map data structure, the key of the map is the user's basic information, and the value of the map is the user's public and / or private text data. Based on the user's basic information (i.e., the key of the map), the user's public and / or private text data (i.e., the value of the map) can be determined.

[0051] Data frequently used by a user over a past period is considered to belong to that user; this attribution can be understood as the user's data usage habits. Furthermore, the user is more likely to use data associated with that user in the future, thus reinforcing this attribution. For example, in the logistics and transportation sector, if a user repeatedly uses Province P, City Q as their shipping address and the same mobile phone number as their contact information within a month, it can be concluded that Province P, City Q, and the mobile phone number are associated with that user, and the user is more likely to frequently use Province P, City Q, and the mobile phone number in their logistics and transportation data in the future.

[0052] A user having a primary attribution relationship with text data indicates that the text data is either shared by that user and other users, or it is used solely by that user. Within the first text dataset, public and private text data that can be identified through the user's basic data have a primary attribution relationship with that user. Data with a primary attribution relationship with a user is text data input by that user or generated through the user executing related instructions within the original text dataset.

[0053] Step S102: From the original text dataset, excluding the first text dataset, identify the text data whose data topic falls within the data topic range of the corresponding user, and obtain a second text dataset that has a second attribution relationship with the corresponding user.

[0054] Text data has a data topic, and the scope of the user's data topic can be determined based on the user's original text dataset. The method of dividing the data topic can be selectively set. For example, in the logistics and transportation field, data topics divided according to the region (e.g., province, city, or county) of the sender's address can include Province R, City S, County T, etc.; data topics divided according to the sender's surname can include surnames U, V, W, etc.; data topics divided according to the user's business scope include multiple categories (e.g., clothing, food, toys, etc.). If the data topic of the text data falls within the user's data topic scope, then the text data is stored in a second text dataset as text data with a second attribution relationship to the user.

[0055] The second text dataset includes text data within each user's data topic scope. The second text dataset can have the same data structure as the first text dataset. Within the second text dataset, text data within that user's data topic scope can be determined based on the user's basic information.

[0056] A second attribution relationship between a user and text data indicates that the text data falls within the user's data topic scope. In the second text dataset, text data within the user's data topic scope, which can be determined through the user's basic data, has a second attribution relationship with that user. The data with a second attribution relationship with the user is also text data from the original text dataset that was input by the user or generated through the user executing related instructions.

[0057] Step S103: Identify abnormal text data in the original text dataset based on the first text dataset and the second text dataset.

[0058] The data in the first and second text datasets are text data that users normally use. If the text data in the user's original text dataset does not exist in either the first or second text dataset, it means that the text data has no first or second attribution relationship with the user. That is, the text data is neither the user's public nor private text data, nor is it text data within the user's data topic scope. Therefore, the text data is considered abnormal text data of the user.

[0059] In embodiments of the present invention, original text datasets from multiple users are acquired. These original text datasets may contain text data representing dimensions such as sender address, sender phone number, sender, order account, and product category. Furthermore, the original text datasets may also be composite text data formed by combining text data from at least two of the aforementioned dimensions. In practical applications, text data from multiple users across multiple dimensions can also be acquired, and for each dimension of text data, abnormal text data is identified through steps S101 to S103 of the embodiments of the present invention.

[0060] In some alternative embodiments, when identifying common text data of multiple users from their original text datasets, text data that appears in all of the original text datasets of the multiple users can be used as the common text data. The multiple users refer to two or more users.

[0061] In this embodiment, when identifying the common text data of multiple users from their original text datasets, if multiple users have the same text data under the same data dimension, then the same text data is the common text data of the multiple users. For example, when performing data analysis on 10 users, if all 10 users use the same shipping address X to ship goods, then the shipping address X will be identified as the common text data of the 10 users, and a first attribution relationship will be formed between the shipping address X and these 10 users, and it will be stored in the first text dataset.

[0062] In some alternative embodiments, when identifying the common text data of multiple users from their original text datasets, text data appearing in the original text datasets of users greater than or equal to a preset first threshold can be considered as the common text data. The multiple users refer to two or more users. In this embodiment, when identifying the common text data of multiple users from their original text datasets, if the number of users with identical text data in the same data dimension is greater than or equal to a preset first threshold, the identical text data is considered the common text data of the aforementioned users. The first threshold can be selectively set or modified according to actual needs. For example, when analyzing data from 10 users, if the preset first threshold is 7, then when 7 or more users use shipping address X to send goods, shipping address X will be identified as the common text data of these 7 or more users. A first attribution relationship is formed between shipping address X and the users using shipping address X, and the data is stored in the first text dataset.

[0063] Many legitimate users utilize public or similar data, such as shipping goods through public addresses. This is normal operational practice and should not be identified as anomalous. Identifying public text data can prevent it from being misjudged as anomalous, ensuring the accuracy of data analysis.

[0064] In some alternative embodiments, when identifying the private text data of each user from the original text datasets of multiple users, the text data that appears only in the original text dataset of one user can be used as the private text data.

[0065] In this embodiment, when identifying the private text data of each user from the original text datasets of multiple users, if a certain text data appears only in the original text dataset of one user, then this text data is identified as the private text data of that user. For example, when performing data analysis on 15 users, if only one user uses sender Y to send a package, then sender Y is identified as the private text data of that user, and a first attribution relationship is formed between sender Y and the user using sender Y, and stored in the first text dataset.

[0066] In some alternative embodiments, when identifying the private text data of each user from the original text datasets of multiple users, text data that appears in the original text datasets of users less than or equal to a preset second threshold can be considered as the private text data. In this embodiment, when identifying the private text data of each user from the original text datasets of multiple users, if the number of users using the text data does not exceed the preset second threshold, then the text data is identified as the private text data of the user using the text data. The second threshold can be selectively set or modified according to actual needs. For example, when performing data analysis on 15 users, if the preset second threshold is 5, then when 5 or fewer users use sender Y to send goods, sender Y will be identified as the private text data of these 5 or fewer users, and a first attribution relationship is formed between sender Y and the users using sender Y, and the data is stored in the first text dataset.

[0067] Users possess private data belonging to themselves or their groups, intended for internal use only. This method allows for the accurate identification of private text data without being affected by the quality of quantifiable data, ensuring the accuracy of data analysis.

[0068] Optionally, step S101 further includes: using a term frequency-inverse document frequency algorithm to identify public and / or private text data in the original text datasets of the multiple users.

[0069] For example, Figure 2This is a schematic diagram illustrating the implementation idea of ​​an optional embodiment of the present invention. For example... Figure 2 As shown, the original text datasets of multiple users are obtained, including text data such as sender address, sender mobile number, sender, order account, and product category. The term frequency-inverse document frequency algorithm is used to identify the text data in the original text dataset that has a first attribution relationship with the user.

[0070] Taking the sender address as an example, a user-sender address detail table is generated based on the sender address, as shown in Table 1.

[0071] Table 1

[0072]

[0073]

[0074] In this context, the sender's address for each waybill of each user is treated as a word, and the sender's addresses corresponding to all waybills of each user constitute a document.

[0075] Then, based on the user-sender address details, the term frequency (TF), inverse document frequency (IDF), and term frequency-inverse document frequency index (TF-IDF) of each sender address in documents composed of sender addresses from different users are calculated. The calculation methods for term frequency, inverse document frequency, and term frequency-inverse document frequency index are as follows:

[0076] Word frequency = the number of times a particular mailing address appears in the document / the total number of words in the document

[0077] Inverse document frequency = log(total number of documents / (number of documents containing this sender address + 1))

[0078] Term Frequency-Inverse Document Frequency Index = Term Frequency * Inverse Document Frequency

[0079] The calculation results are shown in Tables 2 and 3:

[0080] Table 2

[0081] Sending address TF(A) TF(B) TF(C) IDF add1 0.40 0.20 0.60 -0.12 add2 0.00 0.20 0.40 0.00 add3 0.40 0.20 0.00 0.00 add4 0.20 0.20 0.00 0.00 add5 0.00 0.20 0.00 0.18

[0082] Table 3

[0083] Sending address TF-IDF(A) TF-IDF(B) TF-IDF(C) add1 -0.05 -0.02 -0.07 add2 0.00 0.00 0.00 add3 0.00 0.00 0.00 add4 0.00 0.00 0.00 add5 0.00 0.04 0.00

[0084] As can be seen from Tables 2 and 3, the sender address add1 appears in all three users' datasets, and its term frequency and inverse document frequency are both non-zero. Therefore, its term frequency-inverse document frequency index is non-zero, and add1 can be considered a public sender address, which should be stored in the first text dataset of the three users. The sender address add5 appears only in user B's dataset, and its term frequency and inverse document frequency are both non-zero. Therefore, its term frequency-inverse document frequency index is non-zero, and add5 can be considered user B's private sender address, which should be stored in user B's first text dataset. As for the sender addresses add2, add3, and add4, since they appear in multiple users' datasets and appear frequently, their term frequency-inverse document frequency index is zero. Therefore, it is impossible to determine whether these sender addresses are public or private text data, and further analysis is needed in conjunction with step S102.

[0085] This invention analyzes mailing address data to identify users' public and private mailing addresses, obtaining a first text dataset with a first attribution relationship to the user. It should be noted that this invention can also be applied to studies of first attribution relationships between users and other dimensions, such as sender, sender's mobile number, order account, product category, etc., and can also be applied to studies of first attribution relationships between users and dimensions formed by combining multiple of the above dimensions.

[0086] Optionally, identifying text data whose data topic falls within the corresponding user's data topic range includes: obtaining the user's data topic range, determining the data topic of any text data in the user's original text dataset other than the first text dataset, and, if the data topic of any text data falls within the user's data topic range, treating that text data as the user's private text data. When it is impossible to identify whether certain text data is public or private, the user's data topic range can be obtained, and the unidentifiable data can be compared with the user's data topic range to further identify a second attribution relationship between this data and the user.

[0087] Optionally, identifying text data whose data topic falls within the corresponding user's data topic range includes: converting any text data in the user's original text dataset other than the first text dataset into a word-document matrix, where the word-document matrix represents the probability relevance of each word in the document; then decomposing the word-document matrix into a topic-document matrix and a word-topic matrix, where these two matrices represent the probability relevance of the document to the topic and the probability relevance of the topic to the word, respectively, thereby obtaining the data topic range of any text data in the user's original text dataset other than the first text dataset; using the same method to obtain the user's data topic range, if the data topic of any text data falls within the user's data topic range, then treating that text data as the user's private text data, further identifying the second attribution relationship between this data and the user, which helps to accurately identify abnormal data and reduce misjudgments.

[0088] Optionally, identifying text data whose data topic falls within the corresponding user's data topic range includes: converting any text data in the user's original text dataset other than the first text dataset into a word-document matrix; performing singular value decomposition on the word-document matrix; reducing the dimensionality of the matrix after singular value decomposition; and constructing a latent semantic space using the reduced-dimensional matrix. This latent semantic space can be used as the data topic range of any text data in the user's original text dataset other than the first text dataset. Using the same method, the user's data topic range is obtained. If the data topic of any text data falls within the user's data topic range, the any text data is considered the user's private text data. This further identifies a second attribution relationship between this data and the user, which helps to accurately identify abnormal data and reduce misjudgments.

[0089] Optionally, step S102 further includes: obtaining the user's historical text dataset, and determining the user's data topic range based on the historical text dataset using a topic modeling method. Before comparing the text data with the user's data topic range, the user's data topic range can be determined in advance using a topic modeling method. The topic modeling model can be a Latent Dirichlet Distribution (LDA), Non-negative Matrix Factorization (NMF), etc.

[0090] like Figure 2 As shown, LDA (Local Theme Analysis) can be used to further analyze the data and identify the attribution relationship between the data and users. LDA is a document topic generation model that includes a three-layer structure of words, topics, and documents. The documents represent a probability distribution of topics, and the topics represent a probability distribution of words. Topic modeling can be used to identify latent topic information in document sets or corpora.

[0091] Taking a sender's address and user A as an example, the distribution of user A in the dimension of sender's address, obtained through LDA topic modeling, is as follows: add1 (0.4), add2 (0.2), add3 (0.4), where the numbers in parentheses represent the probability of the user using the corresponding sender's address. Comparing the above topic range with the sender's address details corresponding to user A in Table 1, and removing add1 which has already been identified as a public sender's address for user A, it can be seen that add3 is within the data topic range of user A. Therefore, add3 can be stored as the user's private text data in the second text dataset.

[0092] Step S103, based on the first text dataset obtained in step S101 and the second text dataset obtained in step S102, identifies the text data that is not present in either the first or second text dataset as abnormal text data used by the user and which has no affiliation with it.

[0093] Taking a sender's address and user A as an example, user A's sender's address details include: add1, add3, and add4. User A's first text dataset includes add1, and the second text dataset includes add3. Therefore, add4 will be identified as abnormal text data in user A's sender's address details.

[0094] It should be noted that the embodiments of the present invention can be applied to identify target users using anomalous data, and also to identify other users. For example, based on text data of a certain dimension, users who have an attribution relationship with anomalous text data of that dimension are identified as target users; or, based on text data of multiple dimensions, users who have an attribution relationship with anomalous text data of any one or more dimensions are identified as target users. By analyzing data of relevant dimensions and combinations of data, other target user behaviors that may substitute for actions can be identified. The embodiments of the present invention have high reusability in user behavior research.

[0095] Based on the various embodiments described above, it can be seen that the embodiments of the present invention identify text data appearing in the user's original text dataset and text data whose data topics fall within the corresponding user's data topic range, thereby obtaining a text dataset with a first or second attribution relationship with the corresponding user. Based on this text dataset, abnormal text data in the original text dataset is identified, thus enabling accurate identification of abnormal text data. By analyzing the text data used by the user to form attribution relationships, the embodiments of the present invention can improve the accuracy of abnormal data identification without relying on quantifiable indicators, effectively avoiding situations where threshold methods cannot identify abnormal data when quantifiable indicators have low discriminative power, and where abnormal data is misjudged based on common or similar features.

[0096] Figure 3 This is a schematic diagram of the main flow of a data analysis method according to a possible embodiment of the present invention. As another embodiment of the present invention, such as... Figure 3 As shown, the data analysis method may include:

[0097] Step 301: Identify text data that appears in all of the original text datasets of multiple users from the original text datasets of multiple users, and store the text data as public text data in the first text dataset.

[0098] Step 302: Identify text data that appears only in the original text dataset of one user from the original text datasets of multiple users, and store the text data as private text data in the first text dataset;

[0099] The term "multiple users" refers to two or more users. Text data that appears in two or more users is considered public text data. In contrast, private text data refers to data that appears only in one user.

[0100] Step S303: Obtain the data topic range of the user and determine the data topic of any text data in the user's original text dataset other than the first text dataset;

[0101] Step S304: Determine whether the data topic of any text data is within the range of the user's data topic; if the data topic of any text data is within the range of the user's data topic, proceed to step S305; otherwise, proceed to step S306.

[0102] Step S305: Store any of the text data as the user's private text data in the second text dataset;

[0103] Step S306: Identify abnormal text data in the original text dataset based on the first text dataset and the second text dataset.

[0104] The specific implementation details of the data analysis method of the present invention as described above have been explained in detail in the data analysis method described above, so the content will not be repeated here.

[0105] Figure 4 This is a schematic diagram of the main flow of a data analysis method according to another possible embodiment of the present invention. As yet another embodiment of the present invention, such as... Figure 4 As shown, the data analysis method may include:

[0106] Step S401: Identify text data that appears in the original text datasets of multiple users from the original text datasets of multiple users, and preset a first threshold and / or a second threshold.

[0107] Step S402: Determine whether the number of multiple users is greater than or equal to a preset first threshold; if the number of multiple users is greater than or equal to the preset first threshold, proceed to step S403; otherwise, proceed to step S404.

[0108] Step S403: Store the text data as public text data in the first text dataset;

[0109] Step S404: Determine whether the number of multiple users is less than or equal to a preset second threshold; if the number of multiple users is less than or equal to a preset first threshold, proceed to step S405; otherwise, proceed to step S406.

[0110] Step S405: Store the text data as private text data in the first text dataset;

[0111] Text data that appears in more than one number of users than the first preset threshold (less than or equal to the total number of users) is defined as public text data. Text data that appears in more than one number of users than the first preset threshold (more than or equal to the second preset threshold) is defined as private text data. In practice, both the first and second preset thresholds can be set simultaneously, or only the first or only the second preset threshold can be set.

[0112] Step S406: Obtain the data topic range of the user and determine the data topic of any text data in the user's original text dataset other than the first text dataset;

[0113] Step S407: Determine whether the data topic of any text data is within the range of the user's data topics. If the data topic of any text data is within the range of the user's data topics, proceed to step S408; otherwise, proceed to step S409.

[0114] Step S408: Store any text data as the user's private text data in the second text dataset;

[0115] Step S409: Identify abnormal text data in the original text dataset based on the first text dataset and the second text dataset.

[0116] The specific implementation details of the data analysis method of another applicable embodiment of the present invention have been described in detail in the data analysis method described above, so the details will not be repeated here.

[0117] Figure 5 This is a schematic diagram of the main flow of a data analysis method according to another embodiment of the present invention. As yet another embodiment of the present invention, such as... Figure 5 As shown, the data analysis method may include:

[0118] Step S501: From the original text datasets of multiple users, the term frequency-inverse document frequency algorithm is used to identify the public and private text data in the original text datasets of multiple users, and this is used as the first text dataset;

[0119] Step S502: Obtain the user's historical text dataset, and determine the user's data topic range using topic modeling based on the historical text dataset;

[0120] Based on the user's previously used historical text dataset, the user's data topic range is determined in advance using topic modeling methods, including one of the following: Latent Dirichlet Distribution, Non-negative Matrix Factorization;

[0121] Step S503: Determine the data topic of any text data in the user's original text dataset other than the first text dataset;

[0122] Step S504: Determine whether the data topic of any text data is within the range of the user's data topics. If the data topic of any text data is within the range of the user's data topics, proceed to step S505; otherwise, proceed to step S506.

[0123] Step S505: Store any of the text data as the user's private text data in the second text dataset;

[0124] Step S506: Identify abnormal text data in the original text dataset based on the first text dataset and the second text dataset.

[0125] The specific implementation details of the data analysis method of another applicable embodiment of the present invention have been described in detail in the data analysis method described above, so the details will not be repeated here.

[0126] According to a second aspect of the present invention, an apparatus for implementing the above-described method is provided.

[0127] Figure 6 This is a schematic diagram of the main modules of a data analysis apparatus according to an embodiment of the present invention, such as... Figure 6 As shown, the data analysis device 600 includes a first identification module 601, a second identification module 602, and a third identification module 603. The first identification module 601 is used to identify the public text data of the multiple users and the private text data of each user from the original text datasets of multiple users, obtaining a first text dataset with a first attribution relationship to the corresponding user. The second identification module 602 is used to identify text data whose data topics fall within the data topic range of the corresponding user from each text data in the original text dataset other than the first text dataset, obtaining a second text dataset with a second attribution relationship to the corresponding user. The third identification module 603 is used to identify abnormal text data in the original text dataset based on the first text dataset and the second text dataset.

[0128] Optionally, the first identification module 601 is further configured to:

[0129] Text data that appears in the original text datasets of the multiple users, or text data that appears in the original text datasets of users greater than or equal to a preset first threshold, shall be regarded as the public text data.

[0130] Text data that appears only in the original text dataset of one user, or text data that appears in the original text datasets of fewer than or equal to a preset second threshold of users, is considered as the private text data.

[0131] Optionally, the first identification module 601 is further configured to:

[0132] The term frequency-inverse document frequency algorithm is used to identify public and / or private text data in the original text datasets of the multiple users.

[0133] Optionally, the second identification module 602 is further configured to:

[0134] Obtain the user's data topic range, determine the data topic of any text data in the user's original text dataset other than the first text dataset, and if the data topic of any text data is within the user's data topic range, treat the any text data as the user's private text data.

[0135] Optionally, the device further includes a pre-identification module, configured to: acquire the user's historical text dataset, and determine the user's data topic range based on the historical text dataset using a topic modeling method.

[0136] Optionally, the model for topic modeling includes one of the following: implicit Dirichlet distribution, nonnegative matrix factorization.

[0137] It should be noted that the specific implementation details of the data analysis device described in this invention have been described in detail in the data analysis method described above, so the details will not be repeated here.

[0138] According to a third aspect of the present invention, an electronic device for data analysis is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the present invention.

[0139] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0140] Figure 7 An exemplary system architecture 700 is shown, in which a data analysis method or apparatus for applying embodiments of the present invention can be used.

[0141] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, and 703, a network 704, and a server 705. Network 704 serves as the medium for providing communication links between terminal devices 701, 702, and 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0142] Users can use terminal devices 701, 702, and 703 to interact with server 705 via network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as data analysis applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0143] Terminal devices 701, 702, and 703 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0144] Server 705 can be a server that provides various services, such as a backend management server (for example only) that supports data analysis performed by users using terminal devices 701, 702, and 703. The backend management server can analyze and process data such as received user information query requests, and feed back the processing results (for example only) to the terminal devices.

[0145] It should be noted that the data analysis method provided in this embodiment of the invention is generally executed by server 705, and correspondingly, the data analysis device is generally installed in server 705. The data analysis method provided in this embodiment of the invention can also be executed by terminal devices 701, 702, and 703, and correspondingly, the data analysis device can be installed in terminal devices 701, 702, and 703.

[0146] It should be understood that Figure 7 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0147] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing a terminal device of the present invention. Figure 8 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0148] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 608 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0149] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0150] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this invention.

[0151] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer programs according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0153] The modules described in the embodiments of the present invention can be implemented in software or in hardware. The described modules can also be housed in a processor. For example, a processor can be described as including a first identification module, a second identification module, and a third identification module. The names of these modules do not necessarily limit the module itself; for example, the first identification module can also be described as a "module for identifying public text data and private text data."

[0154] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, implement the following method: identifying public text data of the multiple users and private text data of each user from original text datasets of multiple users, to obtain a first text dataset with a first attribution relationship to the corresponding user; identifying text data whose data topics fall within the data topic range of the corresponding user from each text data in the original text dataset other than the first text dataset, to obtain a second text dataset with a second attribution relationship to the corresponding user; and identifying abnormal text data in the original text dataset based on the first text dataset and the second text dataset.

[0155] According to the technical solution of the present invention, by identifying text data appearing in a user's original text dataset and identifying text data whose data topics fall within the scope of the corresponding user's data topics, a text dataset with a first or second attribution relationship with the corresponding user is obtained. Abnormal text data in the original text dataset is then identified based on this text dataset, thereby improving the accuracy of identifying abnormal text data. The present invention, by analyzing the text data used by the user to form attribution relationships, can improve the accuracy of abnormal data identification without relying on quantifiable indicators, effectively avoiding situations where abnormal data cannot be identified by threshold methods when the quantifiable indicators have low discriminative power, and situations where abnormal data is misjudged based on common or similar features.

[0156] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0157] It should be noted that the acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

Claims

1. A data analysis method, characterized in that, include: From the original text datasets of multiple users, the public text data of the multiple users and the private text data of each user are identified to obtain a first text dataset that has a first attribution relationship with the corresponding user; From each text data in the original text dataset other than the first text dataset, identify the text data whose data topic is within the data topic range of the corresponding user, and obtain a second text dataset that has a second affiliation relationship with the corresponding user; Identify anomalous text data in the original text dataset based on the first text dataset and the second text dataset.

2. The method according to claim 1, characterized in that, Identify the public text data of the multiple users from their original text datasets, including: Text data that appears in the original text datasets of the multiple users, or text data that appears in the original text datasets of users greater than or equal to a preset first threshold, shall be regarded as the public text data.

3. The method according to claim 1, characterized in that, Identify the private text data of each user from the raw text datasets of multiple users, including: Text data that appears only in the original text dataset of one user, or text data that appears in the original text datasets of fewer than or equal to a preset second threshold of users, is considered as the private text data.

4. The method according to claim 1, characterized in that, Also includes: The term frequency-inverse document frequency algorithm is used to identify public and / or private text data in the original text datasets of the multiple users.

5. The method according to claim 1, characterized in that, Identify text data whose data topic falls within the corresponding user's data topic range, including: Obtain the user's data topic range, determine the data topic of any text data in the user's original text dataset other than the first text dataset, and if the data topic of any text data is within the user's data topic range, treat the any text data as the user's private text data.

6. The method according to claim 1, characterized in that, Also includes: Obtain the user's historical text dataset, and based on the historical text dataset, use topic modeling methods to determine the user's data topic range.

7. The method according to claim 6, characterized in that, The topic modeling model includes one of the following: implicit Dirichlet distribution, nonnegative matrix factorization.

8. A data analysis apparatus, characterized in that, include: The first identification module is used to identify the public text data of the multiple users and the private text data of each user from the original text dataset of multiple users, so as to obtain a first text dataset with a first affiliation relationship with the corresponding user; The second identification module is used to identify text data whose data topic is within the data topic range of the corresponding user from each text data in the original text dataset other than the first text dataset, and to obtain a second text dataset that has a second affiliation relationship with the corresponding user. The third identification module is used to identify abnormal text data in the original text dataset based on the first text dataset and the second text dataset.

9. An electronic device, characterized in that, include: One or more processors; Storage device, used to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images