Big data information acquisition method

Abstract

The invention relates to a big data information collection method. The method comprises the following steps: collecting retrieval data, browsing data, consulting data and auxiliary data of multiple users in an application system; invalid data removal and noise data filtering are carried out on the collected data, invalid data removal comprises null value and meaningless data removal, duplicate data de-duplication and format standardization processing, and noise data filtering comprises regular filtering and statistical filtering; and constructing an invalid and noise data judgment model based on the processed valid data and invalid and noise data. According to the method, through data preprocessing and double screening of invalid and noise data judgment models, invalid data and noise data can be effectively eliminated, so that the accuracy of data information collection is improved; through automatic judgment of invalid and noise data judgment models and an incremental updating mechanism, the models are continuously optimized along with data accumulation, the manual screening workload of data information collection can be reduced, and the misjudgment rate of automatic judgment is reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically a method for collecting big data information. Background Technology

[0002] With the widespread use of internet applications, a large amount of user behavior data (such as search, browsing, and inquiry data) has been accumulated in application systems. This data is the core basis for analyzing user needs and optimizing service experience.

[0003] However, the current data collection and processing process has the following problems: traditional methods rely on fixed rule filtering. When business scenarios or user input information change, the rules need to be adjusted frequently by humans, which makes the operation cumbersome and the response slow. User operations such as accidental clicks, duplicate submissions, and meaningless input will generate a lot of invalid data. At the same time, non-business-related search keywords, short browsing records and other noise information are mixed in, resulting in low utilization of effective data.

[0004] To address the aforementioned issues, there is an urgent need for a big data information collection method that can collect user behavior data, eliminate invalid and noisy data, and continuously improve the accuracy of judgments through model updates. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] This invention provides a method for collecting big data information, which solves the problems mentioned in the background section.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution: a big data information collection method, comprising the following steps:

[0009] Step 1: Collect search data, browsing data, consultation data, and auxiliary data from multiple users in the application system. The auxiliary data includes user identity tags.

[0010] Step 2: Perform invalid data removal and noise data filtering on the data collected in Step 1. The invalid data removal includes removing null values ​​and meaningless data, deduplicating duplicate data, and standardizing the format. The noise data filtering includes rule-based filtering and statistical filtering.

[0011] Step 3: Construct an invalid and noisy data determination model based on the valid data processed in Step 2, as well as the invalid and noisy data;

[0012] Step 4: The newly collected data is judged by the invalid and noisy data judgment model and the judgment results of valid data, invalid data or noisy data are output. The data in the preset doubtful range is triggered for manual review. The review results are used to correct the judgment conclusion and are fed back to Step 3 as a supervision signal to drive the model update.

[0013] Preferably, the method for establishing the invalid and noisy data determination model in step 3 is as follows:

[0014] Step 31: Extract the quantitative features of the effective data processed in Step 2. The quantitative features include at least three of the following: keyword relevance, normalized browsing time, text topic matching degree, attachment integrity coefficient, and user identity matching degree.

[0015] Step 32: Complete the model initialization training using manually labeled sample sets. The calculation formula for determining invalid and noisy data is as follows:

[0016]

[0017] In the formula, for The model predicts the value at any given time. This is the activation function, used to weight and map features to the probability interval [0,1]. for The feature weight vector at time step, For sample feature vectors, for The time bias term.

[0018] In a further preferred embodiment, step 4, driving the model update, includes updating the feature weight vector and the bias term. The formula for driving the model update is as follows:

[0019]

[0020]

[0021] In the formula, and These are the updated feature weight vector and the bias term, respectively. For learning rate, The true labels for the samples.

[0022] In a further preferred embodiment, the retrieval data includes retrieval keywords, number of retrievals, number of clicks on retrieval results, and retrieval time window; the browsing data includes browsing page type, single-page dwell time, page interaction behavior, and browsing path; and the consultation data includes consultation text content, consultation attachments, consultation channels, consultation time, and reply interaction records.

[0023] In a further preferred embodiment, the removal of null values ​​and meaningless data is set to remove data with empty search keywords, data with consultation text length less than a certain number of characters, and data with invalid user identity tags on the browsing page. The deduplication of duplicate data is based on a combination of user identity tags, behavior type, and behavior time to remove completely duplicate behavior records. The format standardization processing is based on a unified text encoding format to remove garbled characters and meaningless special symbols.

[0024] In a further preferred embodiment, the rule-based filtering removes data with browsing time less than a certain duration, low relevance between search keywords and business topics, and a large proportion of irrelevant keywords in the consultation text by setting thresholds. The statistical filtering removes extreme outliers that exceed the range of 99% of user behavior by calculating the normal distribution of behavioral data.

[0025] (III) Beneficial Effects

[0026] Compared with existing technologies, the present invention provides a method for collecting big data information, which has the following beneficial effects:

[0027] In this invention, through data preprocessing and a dual screening method using an invalid and noisy data identification model, invalid and noisy data can be effectively removed, thereby improving the accuracy of data information collection.

[0028] In this invention, by automating the determination of invalid and noisy data and by using an incremental update mechanism to continuously optimize the model as data accumulates, the workload of manually screening collected data can be reduced, and the false judgment rate of automatic determination can be lowered. Attached Figure Description

[0029] Figure 1 This is a flowchart of the big data information collection method according to the implementation plan. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Please see Figure 1 A method for collecting big data information includes the following steps:

[0032] Step 1: Collect search data, browsing data, consultation data, and auxiliary data from multiple users within the application system through the application system's data output interface and behavior logs. The search data includes search keywords, search frequency, search result click count, and search time window; the browsing data includes page type, single-page dwell time, page interaction behavior, and browsing path; the consultation data includes consultation text content, consultation attachments, consultation channels, consultation time, and reply interaction records; and the auxiliary data includes user identity tags. During data collection, highly immediate behavioral data such as consultation submissions and search result clicks can be acquired immediately after user actions are completed; while non-immediate data such as browsing logs and search history can be collected in batches at fixed daily time periods.

[0033] Step 2: Perform invalid data removal and noise filtering on the data collected in Step 1, i.e., preprocess the collected data. Invalid data removal can include removing null and meaningless data, deduplicating duplicate data, and standardizing the format. Null and meaningless data removal can eliminate data with empty search keywords, consultation texts shorter than a certain number of characters, and data with invalid user identity tags on the browsing page. Deduplicating duplicate data can eliminate completely repeated behavior records (e.g., the same user submitting the same consultation repeatedly within 30 minutes) based on a combination of user identity tags, behavior type, and behavior time. Format standardization can remove garbled characters and meaningless special symbols based on a unified text encoding format to ensure consistent data format. Noise filtering can be performed using both rule-based and statistical methods. Rule-based filtering removes data with browsing time shorter than a certain duration, search keywords with low relevance to the business topic, and consultation texts with a high proportion of irrelevant keywords by setting thresholds. Statistical filtering removes extreme outlier data that exceeds 99% of user behavior range by calculating the normal distribution of the behavioral data.

[0034] Step 3: Based on the valid data processed in Step 2, as well as the invalid and noisy data, construct an invalid and noisy data judgment model. When constructing this model, first extract the quantitative features of the valid data processed in Step 2. These quantitative features include, but are not limited to, at least three of the following: keyword relevance (the degree of matching between search or consultation keywords and the business topic), normalized browsing time value (mapping the browsing time of different users to the [0-1] interval), text topic matching degree (calculating the fit between consultation text and the core business topic through semantic analysis), attachment completeness coefficient (whether business-related attachments are uploaded; 1 for uploaded, 0 for not uploaded), and user identity matching degree (the degree of fit between user identity tags and the core service objects of the business). Then, select a manually labeled sample set (including valid data samples and invalid and noisy data samples), input the sample feature vector into the model, and train it using machine learning algorithms (such as logistic regression) to obtain the initial feature weight vector. With bias term The model initialization is completed, and the calculation formula for determining invalid and noisy data is as follows:

[0035]

[0036] In the formula, for The time-based model prediction value ranges from [0,1]. The closer it is to 1, the more likely the sample is to be valid data; the closer it is to 0, the more likely it is to be invalid or noisy data. It can be used to compare with a preset threshold.

[0037] The activation function is used to weight and sum the features. Mapping to the [0,1] interval enables binary classification probabilities of valid or invalid data, avoiding the inconsistency of judgment criteria caused by unbounded output values. In this embodiment, the activation function can be the Sigmoid function.

[0038] for The feature weight vector at time step, It is a one-dimensional vector, and its dimension is the same as that of the feature vector. Consistent, Each element in the table represents the contribution of that feature to the determination of valid data. The larger the value of each element in the matrix, the stronger the influence of that feature on the judgment result. The positive or negative value of each element in the matrix indicates whether the feature is positively or negatively correlated with the valid data.

[0039] The sample feature vector (a one-dimensional vector) is a set of quantized features extracted from the preprocessed data. Each element is a standardized quantized value (range [0,1]) of a single feature, ensuring that features of different dimensions can be weighted and summed. For example, the sample feature vector... Keyword relevance Used to quantify the degree of matching between search keywords or consultation keywords and the business theme of the application system; normalized value of browsing time. Used to quantify the time users spend browsing business-related pages, eliminating time differences among different users and page types; text topic matching degree. Used to quantify the semantic fit between the consultation text or retrieval description text and the core business topic; Attachment completeness coefficient Used to quantify whether a user uploads business-related attachments; user identity matching degree. Used to quantify the degree of fit between user identity tags and the core service objects of the application system. The values ​​of each element are [0,1]: 1 represents a perfect match, and 0 represents a complete mismatch. Feature Vector During the assignment process, an expandable core keyword library is first preset based on the application scenario. Each core keyword can also be assigned synonyms and near-synonyms to expand the matching range. Then, effective keywords are extracted from user search behavior or consultation text. Finally, the ratio of the number of effective user keywords belonging to the core keyword library to the total number of effective user keywords is used to obtain the value. Mapping the user's "actual dwell time on business-related pages" to the [0,1] range eliminates the time differences between different users and different pages; the longer the dwell time, the clearer the demand. The closer the value is to 1, the better. Semantic analysis quantifies the relevance between the consultation text or retrieval description text and the core business theme; the stronger the relevance to the core theme, the better. The closer the value is to 1, and Quantization can be achieved using existing technologies and algorithms (such as the TF-IDF algorithm). The value is assigned a binary value based on whether the user uploaded business-related attachments. If a valid attachment is uploaded, the value is 1; otherwise, the value is 0. Users can be assigned values ​​based on the degree of fit between their identity tags and the core service objects of the application system. The application system can pre-set an identity tag system (which can be automatically labeled based on user registration information and behavioral characteristics). During the assignment process, identity tags are extracted from user information and assigned values ​​according to preset categories. Assign a value. for The bias term at time step is used as an offset adjustment parameter for the model, which can correct the baseline value of the feature weighted sum and avoid drift in the model's decision threshold due to feature weight initialization bias. For example, when all feature quantization values ​​are 0, ,at this time By adjusting The default effective probability can be set when there is no feature contribution.

[0040] Step 4: After constructing the invalid and noisy data identification model, the newly collected data undergoes data preprocessing and feature vector extraction. Subsequently, the invalid and noisy data judgment model can be used to determine and output the judgment results of valid data, invalid data, or noisy data. Data falling within the preset doubtful interval judgment threshold range is subject to manual review. The review results are used to correct the judgment conclusion and serve as a supervisory signal fed back to step 3 to drive model updates. Driving model updates includes updating the feature weight vector and bias terms. The formula for driving model updates is as follows:

[0041]

[0042]

[0043] In the formula, and These are the updated feature weight vector and the bias term, respectively. and Dimensionality is consistent by fusing the supervision signals of new samples (i.e., the true labels). The adjustments resulted in an improved adaptability of the model to new business scenarios and new user behavior patterns. and Synchronous updates ensure that the model's benchmark judgment threshold is dynamically adjusted as the data distribution changes.

[0044] The learning rate is used to control the update step size of the model weights and biases and to balance the model's convergence speed and stability. Its value ranges from [0,1].

[0045] The sample labels are genuine and were manually verified and labeled. This indicates that the sample data has been manually verified and confirmed as valid. This indicates that the sample data has been manually verified as invalid or noisy.

[0046] To make the technical solution of this invention clearer and easier to understand, the implementation process of this invention will be described in detail below in conjunction with the specific application scenario of the "Technology Achievement Transformation Consulting Platform".

[0047] 1. Scene setting

[0048] A technology transfer consulting platform provides services such as technology transfer matching, policy consultation, and process guidance to enterprises, research institutions, and individual inventors. It needs to collect user search, browsing, and consultation data on the platform, filter out effective data with genuine transfer needs, eliminate invalid and noisy data (such as misoperation and consultations unrelated to transfer), and optimize the allocation of service resources.

[0049] 2. Data Collection

[0050] The following data was collected by parsing the platform's log files and API interfaces:

[0051] Search data includes user search keywords (e.g., "artificial intelligence patent transformation", "new energy achievement matching enterprises", "transformation policy query"), number of searches (e.g., a user searches 3 times in a single day), number of clicks on search results (e.g., clicks on 2 achievement detail pages), and search time window (e.g., 14:30-14:35 on October 15, 2024).

[0052] Browsing data includes: page type (e.g., "Results Library", "Transformation Policy", "Successful Cases"), single-page dwell time (e.g., 8 minutes on a policy page, 3 minutes on a case study page), page interaction behavior (e.g., downloading the "Transformation Materials List", saving a results page), and browsing path (e.g., "Homepage → Results Library → Transformation Policy → Consultation Portal").

[0053] Consultation data includes: consultation text content, consultation attachments (such as business license, requirements specification), consultation channel (PC, mobile), consultation time, and response interaction records (subsequent consultant replies).

[0054] Supporting data: User identity tags (such as "corporate technical procurement officer", "individual inventor", "ordinary user").

[0055] 3. Data Preprocessing

[0056] Invalid information removal includes eliminating records where the consultation text is "none", "?", or purely numerical, and deleting data with empty search keywords or page IDs of "0000" (invalid IDs); based on the combination of "user ID + behavior type + timestamp", duplicate consultations submitted by a user between 14:00 and 14:30 on October 15, 2024, are removed; the unified text encoding is UTF-8, and meaningless special characters such as "￥%&*" are removed from the consultation text.

[0057] Noise data filtering was performed to remove data that showed the user spent 5 seconds browsing the "Conversion Policy" page (accidental click), searched for the keyword "platform customer service phone number" (irrelevant to conversion, relevance score 0.1), and inquiries about "system login failure" (irrelevant keywords accounted for 100%). The normal distribution of the number of inquiries per user per day was calculated, and it was determined that the maximum number of inquiries per day for 99% of users was 5. Extreme anomalies such as a user having 20 inquiries per day were removed.

[0058] 4. Model Building and Updating

[0059] Five core quantization features were extracted from the collected data to construct a feature vector. The meanings of each element are as described above. 1000 manually labeled samples (600 valid data points and 400 invalid and noisy data points) are selected and input into the logistic regression model for training to obtain the initial weights. Bias term Set a threshold for data validity. .

[0060] After preprocessing, the feature vector of a new user's inquiry is... The data was manually verified and confirmed as valid. Set the learning rate .

[0061] Calculate the initial predicted value Weight update: Bias item update: .

[0062] 5. New data judgment

[0063] The following data was collected: New users (enterprise technical procurement personnel) submitted consultation texts such as "Our company needs to connect with the core algorithm achievements of industrial robots from universities, and plans to complete the technology transfer within 3 months. Does the platform have a relevant achievement database and connection process?", spent an average of 12 minutes browsing the "Achievement Database" and "Technology Transfer Process" pages, searched for the keyword "industrial robot algorithm technology transfer", and uploaded the company's business license.

[0064] Feature vectors are extracted after preprocessing. Substitute the updated model values ​​into the calculation: At this point, the newly collected data can be determined to be valid data.

[0065] In the description of this invention, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art will understand the specific meaning of these terms in this invention based on the specific circumstances. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0066] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for collecting big data information, characterized in that, Includes the following steps: Step 1: Collect search data, browsing data, consultation data, and auxiliary data from multiple users in the application system. The auxiliary data includes user identity tags. Step 2: Perform invalid data removal and noise data filtering on the data collected in Step 1. The invalid data removal includes removing null values ​​and meaningless data, deduplicating duplicate data, and standardizing the format. The noise data filtering includes rule-based filtering and statistical filtering. Step 3: Construct an invalid and noisy data determination model based on the valid data processed in Step 2, as well as the invalid and noisy data; Step 4: The newly collected data is judged by the invalid and noisy data judgment model and the judgment results of valid data, invalid data or noisy data are output. The data in the preset doubtful range is triggered for manual review. The review results are used to correct the judgment conclusion and are fed back to Step 3 as a supervision signal to drive the model update.

2. The big data information collection method according to claim 1, characterized in that: The method for establishing the invalid and noisy data determination model in step 3 is as follows: Step 31: Extract the quantitative features of the effective data processed in Step 2. The quantitative features include at least three of the following: keyword relevance, normalized browsing time, text topic matching degree, attachment integrity coefficient, and user identity matching degree. Step 32: Complete the model initialization training using manually labeled sample sets. The calculation formula for determining invalid and noisy data is as follows: In the formula, for The model predicts the value at any given time. This is the activation function, used to weight and map features to the probability interval [0,1]. for The feature weight vector at time step, For sample feature vectors, for The time bias term.

3. The big data information collection method according to claim 2, characterized in that: Step 4, driving the model update, includes updating the feature weight vector and the bias term. The formula for driving the model update is as follows: In the formula, and These are the updated feature weight vector and the bias term, respectively. For learning rate, The true labels for the samples.

4. The big data information collection method according to claim 1, characterized in that: The search data includes search keywords, number of searches, number of clicks on search results, and search time window.

5. The big data information collection method according to claim 4, characterized in that: The browsing data includes the type of page viewed, the duration of stay on a single page, page interaction behavior, and the browsing path.

6. The big data information collection method according to claim 5, characterized in that: The consultation data includes consultation text content, consultation attachments, consultation channels, consultation time, and response interaction records.

7. The big data information collection method according to claim 1, characterized in that: The null and meaningless data removal settings are configured to remove data with empty search keywords, data with consultation text length less than a certain number of characters, and data with invalid user identity tags on the browsing page.

8. The big data information collection method according to claim 7, characterized in that: The deduplication of data is based on a combination of user identity tags, behavior type, and behavior time to remove completely duplicate behavior records.

9. A big data information collection method according to claim 8, characterized in that: The format standardization process is based on a unified text encoding format, removing garbled characters and meaningless special symbols.

10. A big data information collection method according to claim 1, characterized in that: The rule-based filtering removes data with browsing time less than a certain duration, low relevance of search keywords to the business topic, and a large proportion of irrelevant keywords in the consultation text by setting thresholds; the statistical filtering removes extreme outlier data that exceeds 99% of user behavior range by calculating the normal distribution of behavioral data.

Images