A method and system for intelligent data collection and information push based on big data

Abstract

The present invention discloses a method and system for intelligent data collection and information push based on big data. The method comprises the following steps: intelligently collect historical video data and preprocess it; deeply analyze and mine the preprocessed historical video data; analyze the comprehensive video push index, sort the comprehensive video push index from large to small, and push information to the top k; obtain the video push change index, and change the video data information push result according to the video push change index. The present invention analyzes the comprehensive video push index, and then sorts the comprehensive video push index of each video from high to low, obtains a video push list, selects the top k in the video push list for information push, and then improves the accuracy of information push, thereby solving the problem of low accuracy of information push in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of information push technology, and in particular to a method and system for intelligent data collection and information push based on big data. Background Art

[0002] With the continuous development of information technology, big data technology has made breakthrough progress in the intelligent acquisition and storage of video data, and the push of video data information. Big data technology refers to the method of using advanced computer technology and tools to process, store and analyze large-scale data sets. Big data technology has been widely used in finance, medical care, online video push and other fields, bringing more efficient decision-making and management methods to all walks of life; intelligent data collection refers to the process of acquiring, integrating and processing various video data using automation technology and algorithms to provide efficient and accurate data support; information push is based on the user's demand and preference for video data, through intelligent algorithms and personalized video push systems, to push relevant video data information to users in a timely manner.

[0003] At present, there are many aspects of research on intelligent data collection and information push. For example, Chinese invention patent CN117556151A discloses an intelligent data collection and information push method and system based on big data, through the enterprise user information confirmation step, which includes: the enterprise user registers an account through the user terminal, the administrator reviews the enterprise user account registration information through the administrator server and sends the enterprise user account registration information to the cloud server after the information is reviewed and qualified; the cloud server screens the keywords of the enterprise's business scope, matches them with the industry classification preset by the cloud server, and feeds back the matched industry classification to the user terminal; the enterprise user selects the enterprise user account on the user terminal It is necessary to pay attention to the industry classification and send it to the cloud server; the cloud server associates the enterprise user with the industry classification database preset by the cloud server; for example, Chinese invention patent CN108282541A discloses an information push method based on big data and its push system, through the following steps, step one, establish browsing interest points according to the browsing volume of different types of information, and push information according to the browsing interest points; step two, estimate the bandwidth of the receiving terminal according to the loading rate of the receiving terminal, and adapt the bit rate information according to the bandwidth feedback; step three, monitor the browsing information of the receiving terminal, update the browsing interest points, and exchange data between the sending terminal and the receiving terminal through a data transmission module.

[0004] However, in the current intelligent data collection and information push methods, users' basic behaviors are complex and changeable, and traditional algorithms find it difficult to accurately capture users' real needs for video data. In addition, information filtering bias causes the system to tend to push video content with known preferences, while ignoring other video data information that may be of interest. At the same time, the lack of real-time performance will also make the system unable to obtain the latest users' basic behaviors and feedback evaluation data on video data information in a timely manner, which will lead to a reduction in the accuracy of information push.

[0005] Therefore, in summary, the prior art has the problem of low accuracy of information push. Summary of the invention

[0006] The embodiments of the present application solve the problem of low accuracy of information push in the prior art by providing an intelligent data collection and information push method and system based on big data, thereby improving the accuracy of information push.

[0007] The embodiment of the present application provides an intelligent data collection and information push method based on big data, comprising the following steps: intelligently collecting historical video data, and preprocessing the collected original historical video data; deeply analyzing and mining the preprocessed historical video data to analyze the video data value index, the user basic behavior analysis index, and the user feedback evaluation analysis index, wherein the video data value index is used to analyze the quality of the video data content, the user basic behavior analysis index is used to analyze the basic behavior of the user, and the user feedback evaluation analysis index is used to analyze the user's feedback and evaluation on the video data content; according to the video value index, the user basic behavior analysis index, and the user feedback evaluation analysis index, a video comprehensive push index is analyzed, and the video comprehensive push index of each video is sorted from high to low according to the video comprehensive push index to obtain a video push list, and the top k in the video push list are selected for information push, and the video comprehensive push index is used to evaluate and sort the comprehensive push priority of the video data; based on the video push change index, real-time monitoring of the user basic behavior and user feedback evaluation changes of the current video data information push, and changing the next video data information push result according to the video push change index, and the video push change index is used to reflect the degree to which the current video data is popular with users.

[0008] Furthermore, the specific method for intelligently collecting historical video data is as follows: determining the specific needs of the historical video data that needs to be intelligently collected, determining the data source of the historical video data according to the specific needs of the historical video data, and selecting a suitable access method for different types of video data sources; designing a process for intelligent data collection, and intelligently collecting historical video data from various data sources through a suitable access method according to the designed process; and pre-processing the collected original historical video data according to the specific needs of the intelligently collected historical video data.

[0009] Furthermore, the specific method for preprocessing the collected original historical video data is: setting screening conditions according to the specific needs of the intelligently collected historical video data, and removing data that does not meet the requirements; standardizing the historical video data, setting a unique identifier for the historical video data, and performing deduplication operations based on the unique identifier; completing missing values ​​in the historical video data; storing the preprocessed historical video data in a database, and establishing a data backup mechanism.

[0010] Furthermore, the specific method for conducting in-depth analysis and mining of the pre-processed historical video data is: conducting in-depth analysis and mining of the pre-processed historical video data in the database; performing aggregate counting based on the unique identifier of the video, performing statistical analysis on the video viewing records to obtain the total number of video viewing completions, and performing statistical analysis on the video push records to obtain the total number of video pushes; performing statistical analysis on user sharing records to obtain the number of video sharing times; performing statistical analysis on user collection records to obtain the number of video collection times; performing statistical analysis on user viewing records to identify the number of people who play the same video to obtain the number of users who repeatedly play the video; performing statistical analysis on video comment records to obtain the number of video comments; performing statistical analysis on video like records to obtain the number of video likes.

[0011] Furthermore, the specific method for analyzing the video comprehensive push index is: numbering each video of the historical video data; obtaining the video data value index, user basic behavior analysis index and user feedback evaluation analysis index of the historical video data, and constructing a video comprehensive push index calculation formula based on them; calculating the video comprehensive push index according to the video comprehensive push index calculation formula; the video comprehensive push index calculation formula is: TS i =-sech(JZ i *α1+XW i *α2+FK i *α3)+1, where TS i is the video comprehensive push index of the i-th historical video, i = 1, 2, 3, ..., N, i is the historical video number, N is the total number of historical videos, JZ i is the video data value index of the i-th historical video, XW i is the user basic behavior analysis index of the i-th historical video, FK i is the user feedback evaluation analysis index of the ith historical video, α1 is the weight ratio of the video data value index to the video comprehensive push index, α2 is the weight ratio of the user basic behavior analysis index to the video comprehensive push index, and α3 is the weight ratio of the user feedback evaluation analysis index to the video comprehensive push index.

[0012] Furthermore, the specific method for analyzing the video data value index is: obtaining the total number of video viewing completions and the total number of video pushes for each video in the historical video data, and constructing a video data value index analysis formula based thereon; and analyzing the video data value index according to the video data value index analysis formula.

[0013] Furthermore, the specific analysis method of the user basic behavior analysis index is: obtain the number of video sharing, video collection and repeated video playback times of each video in the historical video data, and construct a user basic behavior analysis index analysis formula based on them; analyze the user basic behavior analysis index according to the user basic behavior analysis index analysis formula.

[0014] Furthermore, the specific analysis method of the user feedback evaluation analysis index is: obtaining the number of video comments and the number of video likes for each video in the historical video data, and constructing a user feedback evaluation analysis index analysis formula based thereon; and analyzing the user feedback evaluation analysis index according to the user feedback evaluation analysis index analysis formula.

[0015] Furthermore, the specific method for real-time monitoring of basic user behaviors and changes in user feedback evaluations on the current video data information push based on the video push change trigger index is: numbering each video in the current video push list; obtaining the number of people who do not like the video, the number of people who have not finished playing the video, and the number of people who have removed the video from the favorites list by analyzing the current video data, and constructing a video push change index analysis formula based on the number; analyzing the video push change trigger index according to the video push change trigger index analysis formula; the higher the video push change trigger index, the higher the probability that the current video will be removed from the video push list; when the video push change trigger index is higher than the first threshold, the video is removed from the video push list, and the video push list is updated at the same time, and the top s videos in the video recommendation list are reselected for information push.

[0016] The embodiment of the present application provides an intelligent data collection and information push system based on big data, which includes: an intelligent data collection module, a data analysis module, an information push module and a push change module; wherein the intelligent data collection module is used to collect historical video data and pre-process the collected original historical video data; the data analysis module is used to conduct in-depth analysis and mining of the pre-processed historical video data, and analyze the video data value index, the user basic behavior analysis index and the user feedback evaluation analysis index, wherein the video data value index is used to analyze the quality of the video data content, the user basic behavior analysis index is used to analyze the user's basic behavior, and the user feedback evaluation analysis index is used to analyze the user's evaluation of the video data content. feedback and evaluation; the information push module is used to analyze the video comprehensive push index according to the video value index, the user basic behavior analysis index and the user feedback evaluation analysis index, sort the videos from high to low according to the video comprehensive push index of each video, obtain the video push list, select the top k in the video push list for information push, and the video comprehensive push index is used to evaluate and sort the comprehensive push priority of video data; the push change module is used to monitor the user basic behavior and user feedback evaluation changes of the current video data information push in real time based on the video push change index, and change the next video data information push result according to the video push change index, and the video push change index is used to reflect the popularity of the current video data among users.

[0017] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0018] 1. By obtaining the video value index, user basic behavior analysis index and user feedback evaluation analysis index, the comprehensive video push index is analyzed, and then the comprehensive video push index of each video is sorted from high to low, a video push list is obtained, and the top k videos in the video push list are selected for information push, thereby improving the accuracy of information push and solving the problem of low accuracy of information push in the prior art.

[0019] 2. The video push change index is used to monitor the basic user behavior and user feedback changes of the current video data information push in real time, so as to change the next video data information push result according to the video push change index, thereby realizing real-time update of video push results to meet users' video viewing needs.

[0020] 3. By determining the specific needs and goals of video data for intelligent data collection and information push, the data source of the video data that needs to be intelligently collected is determined, so that historical video data from each data source can be intelligently collected, thereby enriching video data resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flow chart of the intelligent data collection and information push method based on big data provided in the embodiment of the present application;

[0022] Figure 2 A schematic diagram of the structure of the intelligent data collection and information push system based on big data provided in the embodiment of the present application. DETAILED DESCRIPTION

[0023] The embodiment of the present application solves the problem of low accuracy of information push in the prior art by providing an intelligent data collection and information push method and system based on big data. The comprehensive video push index is analyzed by obtaining the video value index, the user basic behavior analysis index and the user feedback evaluation analysis index. Then, the comprehensive video push index of each video is sorted from high to low, a video push list is obtained, and the top k videos in the video push list are selected for information push, thereby improving the accuracy of information push.

[0024] The technical solution in the embodiment of the present application is to solve the problem of low accuracy of the above information push, and the overall idea is as follows:

[0025] By determining the specific needs and goals of video data for intelligent data collection and information push, the data source of the video data that needs to be intelligently collected is determined. For the data sources of different types of video data, the corresponding access methods are used, and then the historical video data of each data source is intelligently collected, where the historical video data includes the historical video push data, the historical basic behavior data of the user, and the historical feedback evaluation data of the user; the collected original historical video data is preprocessed and standardized to ensure the quality of the collected historical video data, and the preprocessed historical video data is stored in the database; the preprocessed historical video data stored in the database is deeply analyzed and mined to analyze the video The value index extracts useful information to provide support for subsequent information push; based on the user's historical basic behavior data and user's historical feedback evaluation data of historical video data, the user's basic behavior analysis index and user feedback evaluation analysis index are analyzed; based on the video value index, the user's basic behavior analysis index and the user's feedback evaluation analysis index, the video comprehensive push index is analyzed, and the top k videos in the comprehensive push index are selected for information push; based on the video push change index, the user's basic behavior and user feedback evaluation changes on the current video data information push are monitored in real time, and the results of the video data information push are continuously changed according to the video push change index, so as to improve the user experience and improve the accuracy of information push.

[0026] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0027] like Figure 1 As shown, it is a flow chart of the intelligent data collection and information push method based on big data provided by an embodiment of the present application. The method is applied to an intelligent data collection and information push system based on big data. The method includes the following steps: intelligently collect historical video data and preprocess it: intelligently collect historical video data and preprocess the collected original historical video data; deeply analyze and mine the preprocessed historical video data: deeply analyze and mine the preprocessed historical video data to analyze the video data value index, user basic behavior analysis index and user feedback evaluation analysis index. The video data value index is used to analyze the quality of the video data content, the user basic behavior analysis index is used to analyze the user's basic behavior, and the user feedback evaluation analysis index is used to analyze the user's feedback and evaluation on the video data content; analyze the video comprehensive push index, so as to Get the video push list based on the comprehensive push index, and select the top k in the video push list for information push: Analyze the video comprehensive push index based on the video value index, user basic behavior analysis index and user feedback evaluation analysis index, sort the videos from high to low based on the video comprehensive push index, get the video push list, and select the top k in the video push list for information push. The video comprehensive push index is used to evaluate and sort the comprehensive push priority of video data; get the video push change index, and change the video data information push result based on the video push change index: monitor the user basic behavior and user feedback evaluation changes of the current video data information push in real time based on the video push change index, and change the next video data information push result based on the video push change index. The video push change index is used to reflect the popularity of the current video data among users.

[0028] In this embodiment, historical video data includes video history recommendation data, user historical basic behavior data and user historical feedback evaluation data; video history recommendation data includes video viewing record data and video push record data; user historical basic behavior data includes user collection record data, user sharing record data and user viewing record data; user historical feedback evaluation data includes video like record data and video comment record data.

[0029] Furthermore, the specific method for intelligently collecting historical video data is as follows: determining the specific requirements for the historical video data that needs to be intelligently collected, determining the data source of the historical video data according to the specific requirements of the historical video data, and selecting a suitable access method for different types of video data sources; designing a process for intelligent data collection, and intelligently collecting historical video data from various data sources through a suitable access method according to the designed process; and pre-processing the collected original historical video data according to the specific requirements for the intelligently collected historical video data.

[0030] In this embodiment, the specific needs of the historical video data that needs to be intelligently collected are determined, including determining the type of video data to be collected and pushed, the video data source, the video data frequency and other information; according to the specific needs of the historical video data for intelligent collection, the data source to be collected is determined, which may involve databases, API interfaces, web page crawling and other methods to ensure that enough data can be obtained; using corresponding tools or programming languages ​​to write a collection program to intelligently collect video data, obtain video data from various video data sources, and perform preliminary cleaning and integration. Crawler technology, API interfaces and other methods can be used to crawl and extract data.

[0031] Furthermore, the specific method for preprocessing the collected original historical video data is: setting screening conditions according to the specific needs of the intelligently collected historical video data, and removing data that does not meet the requirements; standardizing the historical video data, setting a unique identifier for the historical video data, and performing deduplication operations based on the unique identifier; completing missing values ​​in the historical video data; storing the preprocessed historical video data in a database, and establishing a data backup mechanism.

[0032] In this embodiment, according to the specific needs of the historical video data collected intelligently, the screening conditions are determined, and the programming language can be used in combination with conditional statements and loops to screen the data, retain the data that meets the conditions, and remove the data that does not meet the conditions. Removing the data that does not meet the requirements can improve the quality and accuracy of the data and reduce the workload of subsequent processing; setting a unique identifier for deduplication operations can ensure the consistency and uniqueness of the data, avoid interference of duplicate data on analysis and push, and improve the efficiency of data processing and analysis; detecting missing values ​​in historical video data, data processing tools can be used to process missing values. Common methods include using the mean and median for filling, deleting rows containing missing values, etc., processing missing values ​​in historical video data, and filling in missing information, which can make the data more complete and accurate, and improve the reliability and effectiveness of subsequent analysis and mining; storing the pre-processed historical video data in a database and establishing a data backup mechanism can ensure the security and persistence of the data, avoid data loss or damage, and also provide reliable data support for subsequent data analysis and application.

[0033] Furthermore, the specific method for conducting in-depth analysis and mining of the pre-processed historical video data is as follows: conducting in-depth analysis and mining of the pre-processed historical video data in the database; performing aggregate counting based on the unique identifier of the video, performing statistical analysis on the video viewing records to obtain the total number of video viewing completions, and performing statistical analysis on the video push records to obtain the total number of video pushes; performing statistical analysis on user sharing records to obtain the number of video shares; performing statistical analysis on user collection records to obtain the number of video collections; performing statistical analysis on user viewing records to identify the number of people who play the same video to obtain the number of users who play the video repeatedly; performing statistical analysis on video comment records to obtain the number of video comments; performing statistical analysis on video like records to obtain the number of video likes.

[0034] In this embodiment, the pre-processed historical video data in the database is deeply analyzed and mined to obtain the total number of video viewing completions and the total number of video pushes to analyze the video data value index, the number of video sharing, the number of video collections and the number of users who repeatedly play the video to analyze the user basic behavior analysis index, the number of video comments and the number of video likes to analyze the user feedback evaluation analysis index, and the video data value index, the user basic behavior analysis index and the user feedback evaluation analysis index are used to analyze the comprehensive video push index.

[0035] Furthermore, the specific method for analyzing the video comprehensive push index is as follows: number each video of the historical video data; obtain the video data value index, user basic behavior analysis index and user feedback evaluation analysis index of the historical video data, and construct a video comprehensive push index calculation formula based on them; calculate the video comprehensive push index according to the video comprehensive push index calculation formula; the video comprehensive push index calculation formula is: TS i =-sech(JZ i *α1+XW i *α2+FK i *α3)+1, where TS i is the video comprehensive push index of the i-th historical video, i = 1, 2, 3, ..., N, i is the historical video number, N is the total number of historical videos, JZ i is the video data value index of the i-th historical video, XW i is the user basic behavior analysis index of the i-th historical video, FK i is the user feedback evaluation analysis index of the ith historical video, α1 is the weight ratio of the video data value index to the video comprehensive push index, α2 is the weight ratio of the user basic behavior analysis index to the video comprehensive push index, and α3 is the weight ratio of the user feedback evaluation analysis index to the video comprehensive push index.

[0036] In this embodiment, the video comprehensive push index is calculated according to the video comprehensive push index calculation formula. The higher the video comprehensive push index is, the more likely the video is to be pushed to the user. The video comprehensive push index refers to generating an index or score for each video through comprehensive evaluation and analysis of the video content, which is used to measure indicators such as the quality, popularity or relevance of the video. Introducing the video comprehensive push index into the video information push process helps to more accurately evaluate the quality and relevance of each video, thereby accurately recommending video content that users are interested in, improving user experience, and providing personalized video recommendations for different users based on user preferences and behavioral habits, thereby increasing user acceptance and click-through rate of recommended content.

[0037] Furthermore, a specific method for analyzing the video data value index is as follows: obtaining the total number of video viewing completions and the total number of video pushes for each video in the historical video data, and constructing a video data value index analysis formula based thereon; and analyzing the video data value index according to the video data value index analysis formula.

[0038] In this embodiment, the video data value index analysis formula is: In the formula, GK i is the total number of completed video views of the i-th historical video, i = 1, 2, 3, ..., N, i is the historical video number, N is the total number of historical videos, TS i is the total number of video pushes for the i-th historical video, β1 is the weighted ratio of the total number of video viewing completions to the video data value index, and β2 is the weighted ratio of the total number of video pushes to the video data value index.

[0039] Furthermore, a specific analysis method for the user basic behavior analysis index is as follows: obtaining the number of video shares, video collections, and users who repeatedly play the video for each video in the historical video data, and constructing an analysis formula for the user basic behavior analysis index based on the number of shares; and analyzing the user basic behavior analysis index according to the analysis formula for the user basic behavior analysis index.

[0040] In this embodiment, the analysis formula of the user basic behavior analysis index is: In the formula, FX i is the number of times the i-th historical video is shared, i = 1, 2, 3, ..., N, i is the historical video number, N is the total number of historical videos, SC i is the number of video collections of the i-th historical video, CF iis the number of users who repeatedly play the i-th historical video, χ1 is the weighted ratio of the number of video sharing times to the user’s basic behavior analysis index, χ2 is the weighted ratio of the number of video collection times to the user’s basic behavior analysis index, and χ3 is the weighted ratio of the number of users who repeatedly play the video to the user’s basic behavior analysis index.

[0041] Furthermore, a specific analysis method of the user feedback evaluation analysis index is: obtaining the number of video comments and the number of video likes for each video in the historical video data, and constructing a user feedback evaluation analysis index analysis formula based thereon; and analyzing the user feedback evaluation analysis index according to the user feedback evaluation analysis index analysis formula.

[0042] In this embodiment, the analysis formula of the user feedback evaluation analysis index is: FK i =-sech(PL i *δ1+DZ i *δ2)+1, where PL i is the number of video comments on the i-th historical video, i = 1, 2, 3, ..., N, i is the historical video number, N is the total number of historical videos, DZ i is the number of likes for the i-th historical video, δ1 is the weighted ratio of the number of video comments to the user feedback evaluation analysis index, and δ2 is the weighted ratio of the number of video likes to the user feedback evaluation analysis index.

[0043] Furthermore, a specific method for real-time monitoring of basic user behaviors and changes in user feedback evaluations on the current video data information push based on the video push change trigger index is as follows: numbering each video in the current video push list; obtaining the number of people who do not like the video, the number of people who have not finished playing the video, and the number of people who have removed the video from the favorites list by analyzing the current video data, and constructing a video push change index analysis formula based on the number; analyzing the video push change trigger index according to the video push change trigger index analysis formula; the higher the video push change trigger index, the higher the probability that the current video will be removed from the video push list; when the video push change trigger index is higher than the first threshold, the video is removed from the video push list, and the video push list is updated at the same time, and the top s videos in the video recommendation list are reselected for information push.

[0044] In this embodiment, the video push list is obtained by sorting the video from high to low according to the video comprehensive push index of each video, and the top k videos in the video push list are selected for information push. At the same time, the video push list is changed based on the video push change trigger index. When the video push change trigger index is higher than the first threshold, the video is removed from the video push list, and the video push list is updated, and the top s videos in the video recommendation list are reselected for information push, and the above k and s are the same. The number of people who have not finished watching the video is obtained by aggregating and counting the unique identifier of the current video data, performing statistical analysis on the video viewing record, and obtaining the number of people who have removed the video from the favorite list by performing statistical analysis on the user's favorite list. The number of people who do not like the video is obtained by performing statistical analysis on the user's dislike list. The video push change trigger index analysis formula is: GB j = sin(BXH j *ε1+WB j *ε2+YC j *ε3), where GB j is the video push change trigger index of the jth video, j = 1, 2, 3, ..., M, j is the current video push number, M is the total number of current video pushes, BXH j is the number of people who dislike the jth video, WB j is the number of people who have not finished watching the jth video, YC j is the number of people who removed the j-th video from their favorites list, ε1 is the weighted ratio of the number of people who disliked the video to the index that triggered the video push change, ε2 is the weighted ratio of the number of people who did not finish watching the video to the index that triggered the video push change, and ε3 is the weighted ratio of the number of people who removed the video from their favorites list to the index that triggered the video push change.

[0045] like Figure 2As shown, it is a structural schematic diagram of the intelligent data collection and information push system based on big data provided by the embodiment of the present application. The intelligent data collection and information push system based on big data provided by the embodiment of the present application includes: an intelligent data collection module, a data analysis module, an information push module and a push change module; wherein the intelligent data collection module is used to collect historical video data and pre-process the collected original historical video data; the data analysis module is used to conduct in-depth analysis and mining of the pre-processed historical video data, and analyze the video data value index, the user basic behavior analysis index and the user feedback evaluation analysis index, the video data value index is used to analyze the quality of the video data content, the user basic behavior analysis index is used to analyze the user's basic behavior, and the user feedback evaluation analysis index is used to analyze the video data value index. The information push module is used to analyze the user's feedback and evaluation on the video data content; the information push module is used to analyze the video comprehensive push index according to the video value index, the user basic behavior analysis index and the user feedback evaluation analysis index, sort the comprehensive push index of each video from high to low, obtain the video push list, select the top k in the video push list for information push, and the video comprehensive push index is used to evaluate and sort the comprehensive push priority of video data; the push change module is used to monitor the user's basic behavior and user feedback evaluation changes of the current video data information push in real time based on the video push change index, and change the next video data information push result according to the video push change index. The video push change index is used to reflect the popularity of the current video data among users.

[0046] In summary, with the help of the above-mentioned technical scheme of the present invention, the comprehensive video push index is analyzed by obtaining the video value index, the user basic behavior analysis index and the user feedback evaluation analysis index, so as to sort the comprehensive video push index of each video from high to low, obtain the video push list, and select the top k videos in the video push list for information push, thereby improving the accuracy of information push.

[0047] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the specific device can be divided into different functional modules to complete all or part of the functions described above.

[0048] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0049] In several embodiments covered by the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.

[0050] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0051] In addition, each functional module in each embodiment of the present application can be implemented in the form of hardware or in the form of software functional modules. If these functional modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or part of the prior art that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a computer program product, which includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer program product is stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, a computer instruction can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)).

[0052] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A method for intelligent data collection and information push based on big data, characterized in that: The following steps are involved: Intelligently collect historical video data and pre-process the collected original historical video data; Conduct in-depth analysis and mining of the pre-processed historical video data to obtain a video data value index, a user basic behavior analysis index, and a user feedback evaluation analysis index. The video data value index is used to analyze the quality of the video data content, the user basic behavior analysis index is used to analyze the user's basic behavior, and the user feedback evaluation analysis index is used to analyze the user's feedback and evaluation of the video data content. Analyze the video comprehensive push index according to the video value index, user basic behavior analysis index and user feedback evaluation analysis index, sort the videos from high to low according to the video comprehensive push index, obtain the video push list, and select the top k videos in the video push list for information push. The video comprehensive push index is used to evaluate and sort the comprehensive push priority of video data; Based on the video push change index, real-time monitoring of the user basic behavior and user feedback evaluation changes of the current video data information push is carried out, and the next video data information push result is changed according to the video push change index, wherein the video push change index is used to reflect the popularity of the current video data among users; The specific method for analyzing the comprehensive video push index is: Number each video of the historical video data; Obtain the video data value index, user basic behavior analysis index and user feedback evaluation analysis index of historical video data, and construct a video comprehensive push index calculation formula based on them; Calculate the video comprehensive push index according to the video comprehensive push index calculation formula; The calculation formula of the video comprehensive push index is: TS i =-self(JZ i *α1+XW i *α2+FK i *α3)+1, Where, TS i is the video comprehensive push index of the i-th historical video, i = 1, 2, 3, ..., N, i is the historical video number, N is the total number of historical videos, JZ i is the video data value index of the i-th historical video, XW i is the user basic behavior analysis index of the i-th historical video, FK i is the user feedback evaluation analysis index of the i-th historical video, α1 is the weight ratio of the video data value index to the video comprehensive push index, α2 is the weight ratio of the user basic behavior analysis index to the video comprehensive push index, and α3 is the weight ratio of the user feedback evaluation analysis index to the video comprehensive push index; The video data value index analysis formula is: In the formula, GK i is the total number of completed video views of the i-th historical video, i = 1, 2, 3, ..., N, i is the historical video number, N is the total number of historical videos, TS i is the total number of video pushes for the i-th historical video, β1 is the weighted ratio of the total number of video viewing completions to the video data value index, and β2 is the weighted ratio of the total number of video pushes to the video data value index; The analysis formula of the user basic behavior analysis index is: In the formula, FX i is the number of times the i-th historical video is shared, i = 1, 2, 3, ..., N, i is the historical video number, N is the total number of historical videos, SC i is the number of video collections of the i-th historical video, CF i is the number of users who repeatedly played the i-th historical video, χ1 is the weighted ratio of the number of video sharing times to the user's basic behavior analysis index, χ2 is the weighted ratio of the number of video collection times to the user's basic behavior analysis index, and χ3 is the weighted ratio of the number of users who repeatedly played the video to the user's basic behavior analysis index; The analysis formula of the user feedback evaluation analysis index is: FK i =-sech(PL i *δ1+DZ i *δ2)+1, where PL i is the video of the i-th historical video The number of comments, i = 1, 2, 3, ..., N, i is the historical video number, N is the total number of historical videos, DZ i is the number of likes for the i-th historical video, δ1 is the weighted ratio of the number of video comments to the user feedback evaluation analysis index, and δ2 is the weighted ratio of the number of video likes to the user feedback evaluation analysis index.

2. The intelligent data collection and information push method based on big data as claimed in claim 1, characterized in that; The specific method of intelligently collecting historical video data is as follows: Determine the specific needs of historical video data that need to be intelligently collected, determine the data source of historical video data based on the specific needs of historical video data, and select appropriate access methods for different types of video data sources; Design the process of intelligent data collection, and intelligently collect historical video data from various data sources through appropriate access methods according to the designed process; According to the specific needs of the intelligently collected historical video data, the collected original historical video data is preprocessed.

3. The intelligent data collection and information push method based on big data as claimed in claim 2, characterized in that: The specific method for preprocessing the collected original historical video data is as follows: Set filtering conditions based on the specific needs of intelligently collected historical video data to remove data that does not meet the requirements; Standardize the historical video data, set a unique identifier for the historical video data, and perform deduplication operations based on the unique identifier; Complete the missing values ​​in historical video data; The pre-processed historical video data is stored in the database and a data backup mechanism is established.

4. The intelligent data collection and information push method based on big data as claimed in claim 1, characterized in that: The specific method for in-depth analysis and mining of the pre-processed historical video data is as follows: Conduct in-depth analysis and mining of pre-processed historical video data in the database; By performing aggregate counting based on the unique identifier of the video, statistical analysis is performed on the video viewing records to obtain the total number of video viewing completions, and statistical analysis is performed on the video push records to obtain the total number of video pushes; Obtain the number of video sharing times by statistically analyzing user sharing records; Obtain the number of video collections by conducting statistical analysis on user collection records; By performing statistical analysis on user viewing records, we can identify the number of people who play the same video and obtain the number of users who play the video repeatedly. Obtain the number of video comments by statistically analyzing the video comment records; The number of video likes is obtained by performing statistical analysis on the video like records.

5. The intelligent data collection and information push method based on big data as claimed in claim 1, characterized in that: The specific method for analyzing the video data value index is: Obtain the total number of video viewing completions and video pushes for each video in the historical video data, and construct a video data value index analysis formula based on them; The video data value index is analyzed according to the video data value index analysis formula.

6. The intelligent data collection and information push method based on big data as claimed in claim 1, characterized in that: The specific analysis method of the user basic behavior analysis index is: Obtain the number of times each video is shared, collected, and played again by each user in the historical video data, and use this to construct a user basic behavior analysis index analysis formula; The user basic behavior analysis index is analyzed according to the user basic behavior analysis index analysis formula.

7. The intelligent data collection and information push method based on big data as claimed in claim 1, characterized in that: The specific analysis method of the user feedback evaluation analysis index is: Obtain the number of video comments and video likes for each video in the historical video data, and construct a user feedback evaluation analysis index analysis formula based on them; The user feedback evaluation analysis index is analyzed according to the user feedback evaluation analysis index analysis formula.

8. The intelligent data collection and information push method based on big data as claimed in claim 1, characterized in that: The specific method of real-time monitoring the basic user behavior and user feedback evaluation changes of the current video data information push based on the video push change trigger index is: Number each video in the current video push list; By analyzing the current video data, the number of people who dislike the video, the number of people who have not finished watching the video, and the number of people who have removed the video from the favorites list are obtained, and a video push change index analysis formula is constructed based on the data; Analyze the video push change trigger index according to the video push change trigger index analysis formula; When the video push change trigger index is higher, it means the probability of the current video being removed from the video push list is higher; When the video push change trigger index is higher than the first threshold, the video is removed from the video push list, and the video push list is updated at the same time, and the top s videos in the video recommendation list are reselected for information push.

9. An intelligent data collection and information push system based on big data using any one of the methods of claims 1 to 8, characterized in that: include: Intelligent data collection module, data analysis module, information push module and push change module; Wherein, the intelligent data collection module is used to collect historical video data and pre-process the collected original historical video data; The data analysis module is used to conduct in-depth analysis and mining of the pre-processed historical video data to analyze the video data value index, the user basic behavior analysis index and the user feedback evaluation analysis index. The video data value index is used to analyze the quality of the video data content, the user basic behavior analysis index is used to analyze the basic behavior of the user, and the user feedback evaluation analysis index is used to analyze the user's feedback and evaluation on the video data content; The information push module is used to analyze the video comprehensive push index according to the video value index, the user basic behavior analysis index and the user feedback evaluation analysis index, sort the videos from high to low according to the video comprehensive push index, obtain the video push list, and select the top k videos in the video push list for information push. The video comprehensive push index is used to evaluate and sort the comprehensive push priority of video data; The push change module is used to monitor the user basic behavior and user feedback evaluation changes of the current video data information push in real time based on the video push change index, and change the next video data information push result according to the video push change index. The video push change index is used to reflect the degree to which the current video data is popular among users.

Images