Data aging identification method and device

Abstract

The present disclosure relates to a data timeliness identification method and device, a content pushing method and device, an electronic device and a computer readable storage medium. The data timeliness identification method comprises: obtaining to-be-processed data; obtaining associated data by using a web crawler according to the to-be-processed data; and determining timeliness information of the to-be-processed data based on a semantic relationship between the to-be-processed data and the associated data, wherein the timeliness information comprises old news or non-old news. The real publication time of news content can be determined by using Internet information through the web crawler, the timeliness of the news content is further determined through the determined real publication time, whether the publication time of the news content is modified by a content cooperation party can be identified, and the news content in the database is marked with timeliness, so that the timeliness problem can be fully considered during subsequent pushing.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, and in particular to methods and apparatus for identifying data timeliness, methods and apparatus for content delivery, electronic devices, and computer-readable storage media. Background Technology

[0002] With the rapid development of information technology and the internet industry, information overload has become one of the negative impacts of the information age's abundance of information. How users can quickly and accurately find the information they need from the exponentially growing sea of ​​data has become a major challenge. The emergence of recommendation systems has greatly alleviated this difficulty. Personalized systems are a product of the information age; they are advanced intelligent platforms built on massive amounts of data. These platforms connect massive amounts of information content with users, enabling personalized information services tailored to each individual. In recent years, with the research and development of recommendation technology, its application areas have become increasingly widespread, permeating all aspects of our lives, such as the news recommendations in MIUI browser and Toutiao, product recommendations on Taobao, and music recommendations on NetEase Cloud Music. The development trend of deep learning has also been very rapid in recent years, achieving significant results in various fields such as the internet, healthcare, and finance, attracting widespread attention both domestically and internationally. Deep learning technology has been able to effectively solve most problems in fields such as natural language processing and computer vision, achieving leading levels and bringing technological innovation to various fields. Benefiting from the powerful computing power and strong algorithms of the big data era, recommendation systems have also leveraged deep learning to achieve exciting results.

[0003] Currently, common content push systems, such as the MIUI browser, connect content and users through personalized recommendation systems. With a single user request, the system filters dozens of high-quality, relevant content items from millions of submissions and pushes them to the user in real time. The recommended content primarily includes text, images, and videos. Various news apps have become essential tools for users to access trending content and are widely popular. While these news feed products integrate content from numerous high-quality websites, allowing users to easily access valuable reading material, a problem arises when integrating content. Content partners may change the publication date of their content to a recent date, while these articles might describe past hot topics. Such articles need to be filtered in advance to prevent them from being pushed to users. Current push systems cannot accurately determine the timeliness of pushed content and can only filter content based on the time of publication by content partners, often resulting in the push of outdated news and a poor user experience. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a data timeliness identification method and apparatus, a content push method and apparatus, an electronic device, and a computer-readable storage medium.

[0005] According to a first aspect of the present disclosure, a data timeliness identification method is provided, the method comprising: acquiring data to be processed; acquiring related data using a web crawler based on the data to be processed; and determining timeliness information of the data to be processed based on the semantic relationship between the data to be processed and the related data, wherein the timeliness information includes old news or non-old news.

[0006] In one embodiment, before obtaining related data using a web crawler based on the data to be processed, the method further includes: in response to the data to be processed containing date information, determining the timeliness information of the data to be processed based on the date information; and in response to the data to be processed not containing date information, performing the step of obtaining related data using a web crawler based on the data to be processed.

[0007] In one embodiment, determining the timeliness information of the data to be processed based on date information includes: determining the entry time, where the entry time is the time when the data to be processed is obtained; in response to the time difference between the date information and the entry time being greater than a first time threshold, determining that the data to be processed is old news or performing the step of obtaining related data using a web crawler based on the data to be processed; in response to the time difference between the date information and the entry time being less than or equal to the first time threshold, determining that the data to be processed is not old news.

[0008] In one embodiment, the data to be processed includes first title information; based on the data to be processed, related data is obtained using a web crawler, including: obtaining search results through a search engine based on the first title information; and obtaining second title information and publication time as related data based on the search results.

[0009] In one embodiment, determining the timeliness information of the data to be processed based on the semantic relationship between the data to be processed and the associated data includes: preprocessing the first title information and the second title information; vectorizing the preprocessed first title information and the second title information into a first vector and a second vector, respectively; determining the semantic similarity between the first title information and each second title information based on the first vector and the second vector; determining the timeliness information of the data to be processed based on a second time threshold in response to the maximum semantic similarity being greater than or equal to a text similarity threshold; and determining that the data to be processed is not old news in response to the maximum semantic similarity being less than a text similarity threshold.

[0010] In one embodiment, the method further includes: in response to the timeliness information of the data to be processed being determined to be old news, determining the timeliness of the data to be processed whose timeliness information is old news; in response to the timeliness being no timeliness, re-determining the timeliness information of the data to be processed whose timeliness information is old news as non-old news; and in response to the timeliness being timely, maintaining the timeliness information of the data to be processed whose timeliness information is old news.

[0011] In one embodiment, the timeliness of data to be processed that is considered old news is determined by a timeliness classification model. The timeliness classification model is obtained through the following training method: acquiring a training set, which includes multiple training data sets and corresponding labels for the training data, wherein the labels include at least timeliness and no timeliness, and / or strong timeliness and weak timeliness; and training the timeliness classification model using the training set. According to a second aspect of this disclosure, a content push method is provided, comprising: retrieving push content from a database based on user personalized information, wherein the database data is stored after the timeliness information of the data to be processed is determined by the data timeliness identification method of the first aspect; and pushing the push content to the user.

[0012] According to a third aspect of the present disclosure, a data timeliness identification device is provided. The device includes: a data acquisition unit for acquiring data to be processed; a crawler unit for acquiring related data using a web crawler based on the data to be processed; and a timeliness determination unit for determining the timeliness information of the data to be processed based on the semantic relationship between the data to be processed and the related data, wherein the timeliness information includes old news or non-old news.

[0013] In one embodiment, the apparatus further includes: a date determination unit, configured to determine the timeliness information of the data to be processed based on the date information in response to the data to be processed containing date information; and to perform the step of obtaining related data using a web crawler based on the data to be processed in response to the data to be processed not containing date information.

[0014] In one embodiment, the date determination unit further includes: determining the entry time, where the entry time is the time when the data to be processed is acquired; in response to the time difference between the date information and the entry time being greater than a first time threshold, determining that the data to be processed is old news or performing the step of acquiring related data using a web crawler based on the data to be processed; in response to the time difference between the date information and the entry time being less than or equal to the first time threshold, determining that the data to be processed is not old news.

[0015] In one embodiment, the data to be processed includes first title information; the crawler unit includes: a search unit, used to obtain search results through a search engine based on the first title information; and a crawling unit, used to obtain second title information and publication time as associated data based on the search results.

[0016] In one embodiment, the timeliness determination unit includes: preprocessing first title information and second title information; vectorizing the preprocessed first title information and second title information into a first vector and a second vector, respectively; determining the semantic similarity between the first title information and each second title information based on the first vector and the second vector; determining the timeliness information of the data to be processed according to a second time threshold in response to the maximum semantic similarity being greater than or equal to a text similarity threshold; and determining that the data to be processed is not old news in response to the maximum semantic similarity being less than a text similarity threshold.

[0017] In one embodiment, the apparatus further includes: an old news timeliness re-determination unit, configured to, in response to the timeliness information of the data to be processed being determined to be old news, determine the timeliness of the data to be processed whose timeliness information is old news: in response to the timeliness being no timeliness, re-determine the timeliness information of the data to be processed whose timeliness information is old news to be non-old news; in response to the timeliness being timely, maintain the timeliness information of the data to be processed whose timeliness information is old news.

[0018] In one embodiment, the timeliness of the data to be processed, which is old news, is determined by a timeliness classification model. The timeliness classification model is obtained by training the data in the following way: obtaining a training set, which includes multiple training data and corresponding labels for the training data, wherein the labels include at least timeliness and no timeliness, and / or strong timeliness and weak timeliness; and training the timeliness classification model using the training set.

[0019] According to a fourth aspect of the present disclosure, a content push device is provided, the device comprising: a content acquisition unit, configured to acquire push content from data in a database based on user personalized information, wherein the data in the database is stored in the database after the timeliness information of the data to be processed is determined by the data timeliness identification device as described in the third aspect; and a push unit, configured to push the push content to the user.

[0020] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a memory for storing instructions; and a processor for invoking the instructions stored in the memory to execute the data timeliness identification method of the first aspect.

[0021] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, which stores instructions that, when executed by a processor, perform the data timeliness identification method of the first aspect.

[0022] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: This disclosure proposes a data timeliness identification method, which can use web crawlers to determine the real release time of news content with the help of Internet information. The timeliness of the news content can be further determined by the determined real release time, and it can identify whether the release time of the news content is missing or has been modified to a recent time by the content partner. At the same time, the timeliness of the news content in the database is marked, so that the timeliness issue can be fully considered in subsequent push. At the same time, a content push method is proposed based on the data timeliness identification method. Based on timeliness and user personalization analysis, the content push can effectively identify old news and avoid pushing hot events that happened a long time ago to users. Old news can be reliably identified from the content entering the database every day, which significantly improves the user experience of information flow products.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0025] Figure 1 This is a flowchart illustrating a data timeliness identification method according to an exemplary embodiment;

[0026] Figure 2 This is a flowchart illustrating another data timeliness identification method according to an exemplary embodiment;

[0027] Figure 3 This is a flowchart illustrating another data timeliness identification method according to an exemplary embodiment;

[0028] Figure 4 This is a schematic diagram illustrating a content push process according to an exemplary embodiment;

[0029] Figure 5 This is a schematic block diagram illustrating a data timeliness identification device according to an exemplary embodiment;

[0030] Figure 6 This is a schematic block diagram illustrating another data timeliness identification device according to an exemplary embodiment;

[0031] Figure 7 This is a schematic block diagram illustrating a content push device according to an exemplary embodiment;

[0032] Figure 8 This is a schematic block diagram illustrating an apparatus according to an exemplary embodiment.

[0033] Figure 9 This is a schematic block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0035] Current push notification systems cannot accurately determine the timeliness of pushed content; they can only filter content based on the time it was posted by content partners. However, partners often modify the posting time to be recent, and common recommendation systems have low accuracy in identifying old news, frequently resulting in push notifications of outdated news and a poor user experience. The related application in this disclosure proposes an article content recognition method, but this method only utilizes a deep learning model, and its accuracy in judging the timeliness of news still has room for improvement.

[0036] To address the aforementioned problems, this disclosure provides a data timeliness identification method 10, see [link to relevant documentation]. Figure 1 The method includes steps S11-S13, which are described in detail below:

[0037] Step S11: Obtain the data to be processed;

[0038] The data to be processed comes from partners and can be in various formats such as text, images, graphic text, and video, primarily news content. This disclosed technical solution is not limited to text data; images and videos, whose file information includes titles and dates, are also suitable for timeliness detection.

[0039] Step S12: Based on the data to be processed, use a web crawler to obtain related data.

[0040] With the development of the internet, the World Wide Web has become a carrier of vast amounts of information. Traditional search engines such as Google and Baidu serve as tools to assist people in retrieving information and act as entry points for users to access the World Wide Web. Web crawlers, also known as web spiders, are programs or scripts that automatically retrieve information from the World Wide Web according to certain rules. They are widely used in internet search engines or other similar websites to obtain their content. For example, through crawling, we can obtain news titles, publication times, and article content. There are many technologies for implementing web crawlers. They can be implemented using programming languages ​​such as Java, Python, and Scala. Even with Java, there are many options. We can use open-source crawling frameworks such as Nutch, Crawler4j, WebMagic, and WebCollector, or we can use basic tools like HttpClient and Jsoup to crawl web page content. In short, implementing web crawlers is very convenient and fast. Web crawlers also obtain content that is the same as or similar to the data to be processed—i.e., related data—so that the timeliness of the data to be processed can be determined later based on the information in the related data.

[0041] In one embodiment of this disclosure, the data to be processed includes first title information; step S12 may include: obtaining search results through a search engine based on the first title information; and obtaining second title information and publication time as associated data based on the search results.

[0042] In this embodiment, the first title information can be the title of the data to be processed, such as the main title, a subtitle, or, in some cases, the first sentence of the text. The first title information generally represents the main content of the data to be processed. Searching using the first title allows for quick retrieval of content related to the data, reducing computational load compared to full-text search and avoiding the influence of irrelevant content in the full text on the search results. The second title information can be the title of the search results, i.e., the title of the related data. This can be the title displayed on the search results page, or it can include the subtitle or the first sentence of the related data. The second title information can summarize the content of the related data. For example, searching for title information in search engines like Baidu returns k search results, where k can be chosen according to actual needs. After obtaining the search results, a web crawler is used to retrieve the titles and publication times of the first k results returned by the search page. This information represents the related data similar to the data to be processed. The crawler can extract the title information and publication time from page snapshots, webpage source code, etc. Firstly, using title information for retrieval allows for quick retrieval of the corresponding search results, and the search engine will sort by similarity, displaying the most similar results first. On the internet, the publication time of a news article is usually the actual publication time, which can be found in information such as the webpage source code and webpage snapshots. Web crawlers can be used to quickly search for and obtain the actual publication time.

[0043] Step S13: Determine the timeliness information of the data to be processed based on the semantic relationship between the data to be processed and the related data. The timeliness information includes old news or non-old news.

[0044] News content is defined as old news based on its timeliness. Content meeting the following two conditions is considered old news: First, the actual publication time of the news is x days prior to its entry into the database (x can be chosen based on specific circumstances) (the content can be pushed to users immediately after being entered into the database); Second, the news content has timeliness. Based on the semantic information of the data to be processed and its associated data, it can be determined whether the news content represented by the data to be processed and its associated data are consistent. The timeliness of the associated data is used to further determine the timeliness of the data to be processed.

[0045] In one embodiment of this disclosure, step S13 may include: determining reference data in the associated data based on the semantic relationship between the data to be processed and the associated data; and determining the timeliness information of the data to be processed based on the reference data.

[0046] Reference data can be the most relevant and closest in meaning to the data to be processed among the related data. In the K related data retrieved during the retrieval process, the closest corresponding result exists. Semantic information can be used to determine the data closest to the data to be processed, and this data is used as reference data. Firstly, identifying the closest content avoids noise from the K results affecting the judgment result. Secondly, selecting reference data for comparison effectively reduces computational complexity.

[0047] In one embodiment of this disclosure, determining reference data in the associated data based on the semantic relationship between the data to be processed and the associated data may include: preprocessing the first title information and the second title information; vectorizing the preprocessed first title information and the second title information into a first vector and a second vector, respectively; determining the semantic similarity between the first title information and each second title information based on the first vector and the second vector; and using the associated data corresponding to the second title information with the highest semantic similarity as the reference data.

[0048] First, semantic information is processed. Given news titles and crawled news titles, open-source technologies are used for word segmentation and stop word removal. Then, pre-trained word vectors, such as word2vec, are used to calculate the semantic vector of the title. This pre-trained word vector model can quickly vectorize the titles of the data to be processed and related data. Furthermore, the semantic similarity between the data to be processed and each related data item is calculated, and the related data with the highest semantic similarity is used as the reference data. Cosine similarity can be calculated and used as the semantic similarity. When news occurs, multiple media outlets may use different words in their content reports, but their titles are mostly the same or similar. Therefore, choosing the title as the semantic vector for comparison allows for quick location of relevant news, while avoiding the need to calculate word vectors for all news content, thus improving the speed of semantic calculation.

[0049] In one embodiment of this disclosure, determining the timeliness information of the data to be processed based on reference data may include: determining whether the semantic similarity between the reference data and the data to be processed is greater than a text similarity threshold; in response to the maximum semantic similarity being greater than or equal to the text similarity threshold, determining the timeliness information of the data to be processed based on a second time threshold; and in response to the maximum semantic similarity being less than the text similarity threshold, determining that the data to be processed is not old news. By setting a semantic similarity threshold, it is possible to determine whether there is relevant news, especially old news, which generally has corresponding news data on the network. When a search engine contains text with high similarity, it is necessary to further compare its actual publication time. When there is no relevant news in the system, that is, its semantic similarity is less than the threshold, it proves that the corresponding news cannot be found in the search engine, indicating that the content is highly time-sensitive and has not yet been published, and the data can be determined as not old news.

[0050] In one embodiment of this disclosure, determining the timeliness information of data based on a second time threshold includes: if the difference between the publication time of the reference data and the entry time of the data to be processed is greater than or equal to the second time threshold, then the data to be processed is determined to be old news, wherein the entry time is the time when the data to be processed is acquired; if the difference between the publication time of the reference data and the entry time of the data to be processed is less than the second time threshold, then the data to be processed is determined to be non-old news.

[0051] Specifically, for example, the content with the highest similarity among crawled similar news articles is considered reference data, and is categorized into three cases: Case 1: If the cosine similarity between the reference data and the given news article's title is greater than or equal to the text similarity threshold (the text similarity threshold is set according to specific circumstances), and the publication time of the reference data is more than x days earlier than its entry time into the database, it is defined as old news; Case 2: If the cosine similarity between the title of the reference data and the title of the given news article is greater than or equal to the text similarity threshold, and the publication time of the reference data is less than x days earlier than its entry time into the database, the given news is determined to be non-old news; Case 3: If the cosine similarity is less than the text similarity threshold, it is impossible to determine whether the given news is old news through crawling, and it is directly defined as non-old news. When news information exists in a search engine, comparing the actual publication time and the entry time into the database, if the difference is greater than a certain time, it proves that the content belongs to old news. Using a time threshold can conveniently and quickly calculate the timeliness of news. Furthermore, if the cosine similarity between the reference data and the given news article's title is less than the text similarity threshold, it may be because the given news is very new, and therefore no similar reference data can be found, hence it is marked as non-old news.

[0052] In one embodiment of this disclosure, the data timeliness identification method 10 further includes: step S14, as follows: Figure 2 As shown, step S14 includes determining the timeliness of the data to be processed as old news in response to the determination that the timeliness information of the data to be processed is old news; in response to the timeliness being no timeliness, re-determining the timeliness information of the data to be processed as old news as non-old news; and in response to the timeliness being timely, maintaining the timeliness information of the data to be processed as old news.

[0053] When judging old news by time, some articles are easily overlooked, such as advertisements, public service announcements, life tips, and health and wellness content. These articles may have been published a long time ago, and using the above-mentioned technical solutions, they are easily classified as old news, affecting their ranking in subsequent push notifications. These articles are content that users will not perceive as "outdated" no matter how long they are pushed to them; therefore, they should not all be treated as old news, reducing their weight in push notifications. Therefore, this disclosure further filters old news, identifying this portion as non-time-sensitive content and classifying it as non-old news. During the filtering process, if a given news item is identified as non-time-sensitive content, it is considered non-old news; otherwise, it is considered old news. By further confirming old news information, non-time-sensitive content can be filtered out. In personalized recommendations, relevant user groups will prefer to read this type of non-time-sensitive content. This avoids the ranking of related content being lowered due to timeliness issues, thus affecting the results of personalized push notifications.

[0054] In one embodiment of this disclosure, the timeliness of the data to be processed, which is old news, is determined by a timeliness classification model. The timeliness classification model is obtained by the following training method: obtaining a training set, which includes multiple training data and corresponding labels for the training data, wherein the labels include at least timeliness and no timeliness; and training the timeliness classification model using the training set.

[0055] The timeliness classification model can be a classification model from the field of NLP (Natural Language Processing). It obtains the classification result of text information through input text information and feature extraction, such as the BERT (Bidirectional Encoder Representation from Transformers) model. In this embodiment, the training set for training the timeliness classification model can include various types of news samples, which are manually labeled as "having timely content" or "not having timely content". A supervised classification model, such as the aforementioned BERT model, is trained using a sample set with category labels. The data to be processed is input into the trained timeliness classification model to obtain the timeliness category of the data. The output results are then supervised through labeling, and the parameters of the timeliness classification model are adjusted. The trained model achieves an industrially usable precision and recall of over 80% on new samples. Through a supervised classification model, the timeliness of text can be quickly distinguished, and non-timely content can be accurately labeled.

[0056] In one embodiment of this disclosure, in addition to time-sensitive content, the labeling also includes strong timeliness and weak timeliness. That is, the training set is labeled as "strongly time-sensitive content," "weakly time-sensitive content," or "no timeliness content." A supervised classification model is trained using a sample set with category labels to obtain a timeliness classification model.

[0057] If an article is pushed to users the day after its publication, giving users the perception of "outdated content," it will be classified as highly time-sensitive content. If an article is pushed to users no longer and still doesn't give users the perception of being "outdated," it will be classified as timeless content. All other content is classified as less time-sensitive. By classifying highly and less time-sensitive content in just one step, on top of existing time-sensitive content, the distinction between timeliness and non-timeliness is more explicit. In subsequent personalized recommendations, the weight of tags can be adjusted, making the timeliness score more flexible and the push ratio adjusted accordingly, better aligning with actual reading habits.

[0058] The solution provided in this disclosure can effectively identify old news, and can stably identify about three thousand old news items from the content added to the database every day. Filtering old news significantly improves the user experience of information flow products.

[0059] This disclosure also provides a data timeliness identification method 20, such as Figure 4 As shown, the data timeliness identification method 20 includes steps S21-S24, which are described in detail below:

[0060] Step S21: Obtain the data to be processed.

[0061] Step S22: In response to the data to be processed containing date information, determine the timeliness information of the data to be processed based on the date information; in response to the data to be processed not containing date information, execute the step of obtaining related data using a web crawler based on the data to be processed.

[0062] Some push notifications contain date information, such as news releases describing the specific date of an event or image files including the image's creation time. The timeliness of this information can be directly determined based on the included date, eliminating the need for further web scraping. When the push notification does not include a corresponding date, then web scraping is used to retrieve the relevant data. This improves the efficiency of timeliness assessment, allowing for direct determination of timeliness based on the date information in the data to be processed, thereby reducing the workload of subsequent matching steps.

[0063] In one embodiment of this disclosure, determining the timeliness information of the data to be processed based on date information includes: determining the entry time, where the entry time is the time when the data to be processed is acquired; in response to the time difference between the date information and the entry time being greater than a first time threshold, determining that the data to be processed is old news or performing the step of acquiring related data using a web crawler based on the data to be processed; and in response to the time difference between the date information and the entry time being less than or equal to the first time threshold, determining that the data to be processed is not old news.

[0064] Specifically, date pattern matching is performed on the title and body of a given news article. Specifically, if a date pattern such as "Year Month Day", "Month Day", or "Day" is matched, and the latest date matched in the article is less than x days from the time the news was entered into the database, the given news is determined to be non-old news; otherwise, proceed to step S23. By comparing the entry time and the occurrence date, the timeliness of the news can be quickly and intuitively determined.

[0065] Step S23: Based on the data to be processed, use a web crawler to obtain related data.

[0066] Step S24: Determine the timeliness information of the data to be processed based on the semantic relationship between the data to be processed and the related data, wherein the timeliness information includes old news or non-old news.

[0067] Web crawlers can use internet information to determine the true publication time of news content. By determining the true publication time, the timeliness of the news content can be further determined. It is possible to identify whether the publication time of the news content has been modified to a recent time by the content partner. At the same time, the timeliness of the news content entering the database can be marked so that the timeliness issue can be fully considered when pushing it out.

[0068] According to a second aspect of the present disclosure, a content push method is provided, the method comprising: retrieving push content from data in a database based on user personalized information, wherein the data in the database is stored in the database after the timeliness information of the data to be processed is determined by the data timeliness identification method of the first aspect; and pushing the push content to the user. The push method may be sending to a push interface or displaying on a terminal, such as through a browser or other application on the terminal.

[0069] For details on the content push process, please refer to [link / reference]. Figure 4This includes: content access, where a content partner pushes news content for access; content understanding, where the accessed content is identified as old news, marked as old news and added to the database if identified as old news, and marked as non-old news and added to the database if identified as non-old news; and content recommendation, where information flow recommendation involves steps such as content retrieval, sorting, scoring, and reordering, selecting dozens of high-quality content from the content pool to recommend to users. During the content retrieval stage, old news is filtered out to avoid pushing descriptions of hot events that occurred a long time ago to users. Based on the data timeliness identification method disclosed herein, a content push method is proposed, which pushes content based on timeliness and user personalization analysis. After timeliness labeling of the data, both timeliness and user personalization information are considered, significantly improving the user experience of information flow products.

[0070] Based on the same inventive concept Figure 5 A data timeliness identification device 100 is shown, comprising: a data acquisition unit 110 for acquiring data to be processed; a crawler unit 120 for acquiring related data using a web crawler based on the data to be processed; and a timeliness determination unit 130 for determining the timeliness information of the data to be processed based on the semantic relationship between the data to be processed and the related data, wherein the timeliness information includes old news or non-old news.

[0071] In one embodiment, such as Figure 6 As shown, the data timeliness identification device 100 further includes: a date judgment unit 140, which is used to determine the timeliness information of the data to be processed based on the date information in response to the data to be processed containing date information; and to perform the step of obtaining related data based on the data to be processed using a web crawler in response to the data to be processed not containing date information.

[0072] In one embodiment, the date determination unit 140 is further configured to: determine the entry time, wherein the entry time is the time when the data to be processed is acquired; in response to the time difference between the date information and the entry time being greater than a first time threshold, determine that the data to be processed is old news or perform the step of acquiring related data using a web crawler based on the data to be processed; in response to the time difference between the date information and the entry time being less than or equal to the first time threshold, determine that the data to be processed is not old news.

[0073] In one embodiment, the data to be processed includes first title information; the crawler unit 120 includes: a search unit, used to obtain search results through a search engine based on the first title information; and a crawling unit, used to obtain second title information and publication time as associated data based on the search results.

[0074] In one embodiment, the timeliness determination unit 130 includes: preprocessing the first title information and the second title information; vectorizing the preprocessed first title information and the second title information into a first vector and a second vector, respectively; determining the semantic similarity between the first title information and each second title information based on the first vector and the second vector; determining the timeliness information of the data to be processed according to a second time threshold in response to the maximum semantic similarity being greater than or equal to a text similarity threshold; and determining that the data to be processed is not old news in response to the maximum semantic similarity being less than a text similarity threshold.

[0075] In one embodiment, the apparatus further includes: an old news timeliness re-determination unit, configured to, in response to the timeliness information of the data to be processed being determined to be old news, determine the timeliness of the data to be processed whose timeliness information is old news: in response to the timeliness being no timeliness, re-determine the timeliness information of the data to be processed whose timeliness information is old news to be non-old news; in response to the timeliness being timely, maintain the timeliness information of the data to be processed whose timeliness information is old news.

[0076] In one embodiment, the timeliness of data to be processed that is classified as old news is determined by a timeliness classification model. The timeliness classification model is obtained through the following training method: acquiring a training set, which includes multiple training data sets and corresponding labels for the training data sets, wherein the labels include at least timeliness and no timeliness, and / or strong timeliness and weak timeliness; and training the timeliness classification model using the training set. According to a fourth aspect of the embodiments of this disclosure, as... Figure 7 As shown, a content push device 200 is provided, including: a content acquisition unit 210, used to acquire push content from data in a database based on user personalized information, wherein the data in the database is stored in the database after the timeliness information of the data to be processed is determined by a data timeliness identification device as described in the third aspect; and a push unit 220, used to push the push content to the user.

[0077] Figure 8 This is a schematic block diagram illustrating an apparatus according to an exemplary embodiment of any of the foregoing embodiments. For example, apparatus 300 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0078] Reference Figure 8 The device 300 may include one or more of the following components: a processing component 302, a memory 304, a power component 306, a multimedia component 308, an audio component 310, an input / output (I / O) interface 312, a sensor component 314, and a communication component 316.

[0079] Processing component 302 typically controls the overall operation of device 300, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 302 may include one or more processors 320 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 302 may include one or more modules to facilitate interaction between processing component 302 and other components. For example, processing component 302 may include a multimedia module to facilitate interaction between multimedia component 308 and processing component 302.

[0080] Memory 304 is configured to store various types of data to support the operation of device 300. Examples of such data include instructions for any application or method operating on device 300, contact data, phonebook data, messages, pictures, videos, etc. Memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0081] The power supply component 306 provides power to the various components of the device 300. The power supply component 306 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 300.

[0082] Multimedia component 308 includes a screen that provides an output interface between the device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 308 includes a front-facing camera and / or a rear-facing camera. When the device 300 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0083] Audio component 310 is configured to output and / or input audio signals. For example, audio component 310 includes a microphone (MIC) configured to receive external audio signals when device 300 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 304 or transmitted via communication component 316. In some embodiments, audio component 310 also includes a speaker for outputting audio signals.

[0084] I / O interface 312 provides an interface between processing component 302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0085] Sensor assembly 314 includes one or more sensors for providing status assessments of various aspects of device 300. For example, sensor assembly 314 may detect the on / off state of device 300, the relative positioning of components such as the display and keypad of device 300, changes in the position of device 300 or a component of device 300, the presence or absence of user contact with device 300, the orientation or acceleration / deceleration of device 300, and temperature changes of device 300. Sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 314 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 314 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0086] Communication component 316 is configured to facilitate wired or wireless communication between device 300 and other devices. Device 300 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 316 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 316 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0087] In an exemplary embodiment, the apparatus 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0088] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, which can be executed by a processor 320 of the device 300 to perform the above-described method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0089] Figure 9 This is a block diagram illustrating an electronic device 400 according to an exemplary embodiment. For example, device 400 may be provided as a server. (Refer to...) Figure 9 The apparatus 400 includes a processing component 422, which further includes one or more processors, and memory resources represented by memory 442 for storing instructions, such as application programs, that can be executed by the processing component 422. The application programs stored in memory 442 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 422 is configured to execute instructions to perform the methods described above.

[0090] Device 400 may also include a power supply component 426 configured to perform power management of device 300, a wired or wireless network interface 450 configured to connect device 400 to a network, and an input / output (I / O) interface 458. Device 400 may operate on an operating system stored in memory 442, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0091] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0092] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for identifying data timeliness, characterized in that, The method includes: Obtain the data to be processed, wherein the data to be processed includes first title information; Based on the first title information, search results are obtained through a search engine, and based on the search results, the second title information of the related data and the publication time of the related data are obtained; The timeliness information of the data to be processed is determined based on the semantic relationship between the data to be processed and the associated data, wherein the timeliness information includes old news or non-old news, and the semantic relationship includes semantic similarity. In response to the determination that the timeliness information of the data to be processed is old news, the timeliness of the data to be processed is determined as follows: In response to the condition that the timeliness is no timeliness, the timeliness information of the data to be processed, which is old news, is redefined as not old news; In response to the timeliness being time-sensitive, the timeliness information of the pending data is maintained as old news; The step of determining the timeliness information of the data to be processed based on the semantic relationship between the data to be processed and the associated data includes: The first title information and the second title information are preprocessed; The preprocessed first title information and second title information are vectorized respectively, and transformed into a first vector and a second vector; Based on the first vector and the second vector, determine the semantic similarity between the first title information and each piece of second title information; In response to the maximum semantic similarity being greater than or equal to the text similarity threshold, the timeliness information of the data to be processed is determined according to the second time threshold. In response to the maximum semantic similarity being less than the text similarity threshold, the data to be processed is determined to be non-old news.

2. The data timeliness identification method according to claim 1, characterized in that, Before obtaining related data using a web crawler based on the data to be processed, the method further includes: In response to the fact that the data to be processed contains date information, the timeliness information of the data to be processed is determined based on the date information; In response to the fact that the data to be processed does not contain date information, the step of obtaining related data using a web crawler based on the data to be processed is performed.

3. The data timeliness identification method according to claim 2, characterized in that, The step of determining the timeliness information of the data to be processed based on the date information includes: Determine the entry time, where the entry time is the time when the data to be processed is acquired; If the time difference between the date information and the entry time is greater than a first time threshold, the data to be processed is determined to be old news or the step of obtaining related data using a web crawler based on the data to be processed is executed. In response to the date information and the entry time being less than or equal to a first time threshold, the data to be processed is determined to be non-old news.

4. The data timeliness identification method according to claim 1, characterized in that, The timeliness of the data to be processed, which is classified as old news, is determined using a timeliness classification model. This timeliness classification model is obtained through the following training method: Obtain a training set, which includes multiple training data and corresponding annotations for the training data, wherein the annotations include at least time-sensitive and non-time-sensitive, and / or strong time-sensitive and weak time-sensitive; The time-sensitive classification model is trained using the training set.

5. A content push method, characterized in that, The method includes: Based on user personalized information, push content is obtained from data in the database, wherein the data in the database is stored in the data after the timeliness information of the data to be processed is determined by the data timeliness identification method as described in any one of claims 1-4; The push content is pushed to the user.

6. A data timeliness identification device, characterized in that, The device includes: A data acquisition unit is used to acquire data to be processed, wherein the data to be processed includes first title information; The crawler unit is used to obtain search results through a search engine based on the first title information, and to obtain the second title information of the associated data and the publication time of the associated data based on the search results; A timeliness determination unit is used to determine the timeliness information of the data to be processed based on the semantic relationship between the data to be processed and the associated data, wherein the timeliness information includes old news or non-old news, and the semantic relationship includes semantic similarity. The old news timeliness re-determination unit is used to determine the timeliness of the data to be processed in response to the timeliness information of the data to be processed being determined to be old news. In response to the condition that the timeliness is no timeliness, the timeliness information of the data to be processed, which is old news, is redefined as not old news; In response to the timeliness being time-sensitive, the timeliness information of the pending data is maintained as old news; The timeliness determination unit determines the timeliness information of the data to be processed based on the semantic relationship between the data to be processed and the associated data in the following manner: The first title information and the second title information are preprocessed; The preprocessed first title information and second title information are vectorized respectively, and transformed into a first vector and a second vector; Based on the first vector and the second vector, determine the semantic similarity between the first title information and each piece of second title information; In response to the maximum semantic similarity being greater than or equal to the text similarity threshold, the timeliness information of the data to be processed is determined according to the second time threshold. In response to the maximum semantic similarity being less than the text similarity threshold, the data to be processed is determined to be non-old news.

7. The data timeliness identification device according to claim 6, characterized in that, The device further includes: A date determination unit is used to determine the timeliness information of the data to be processed based on the date information in response to the data to be processed containing date information. In response to the fact that the data to be processed does not contain date information, the step of obtaining related data using a web crawler based on the data to be processed is performed.

8. The data timeliness identification device according to claim 7, characterized in that, The date determination unit further includes: Determine the entry time, where the entry time is the time when the data to be processed is acquired; If the time difference between the date information and the entry time is greater than a first time threshold, the data to be processed is determined to be old news or the step of obtaining related data using a web crawler based on the data to be processed is executed. If the time difference between the date information and the entry time is less than or equal to a first time threshold, then the data to be processed is determined to be non-old news.

9. The data timeliness identification device according to claim 6, characterized in that, The timeliness of the data to be processed, which is classified as old news, is determined using a timeliness classification model. This timeliness classification model is obtained through the following training method: Obtain a training set, which includes multiple training data and corresponding annotations for the training data, wherein the annotations include at least time-sensitive and non-time-sensitive, and / or strong time-sensitive and weak time-sensitive; The time-sensitive classification model is trained using the training set.

10. A content push device, characterized in that, The device includes: The content acquisition unit is used to acquire push content from the database based on the user's personalized information, wherein the data in the database is stored in the data after the timeliness information of the data to be processed is determined by the data timeliness identification device as described in any one of claims 6-9; The push unit is used to push the push content to the user.

11. An electronic device, characterized in that, include: Memory, used to store instructions; as well as A processor is configured to invoke instructions stored in the memory to execute the data timeliness identification method as described in any one of claims 1 to 4.

12. A computer-readable storage medium, characterized in that, The device stores instructions that, when executed by a processor, perform the data timeliness identification method as described in any one of claims 1 to 4.

Images