A multi-level data optimization method and system based on big data

Abstract

The present invention relates to the field of data processing technology, specifically a multi-level data optimization method and system based on big data, the method comprising: obtaining pre-selected data and browsing data of a user; screening out a plurality of first video data from the browsing data; fusing and screening the first video data to obtain video data to be played; when the type of the browsed video is a negative type, determining whether the duration of browsing the negative type video is greater than a second preset duration, and determining whether the number of negative type videos browsed is greater than a preset number; if all are negative, playing the video data to be played; if one of them is yes, using a video screening model to screen out video data that meets preset rules from a video information library and playing the video data. The present invention can solve the problem that existing video intelligent recommendation algorithms do not take into account user interests, but continue to recommend negative videos to users, causing users to continue to fall into negative emotions and affecting their viewing mood.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multi-level data optimization method and system based on big data. Background Art

[0002] Most existing short video platforms have a video intelligent recommendation system. After a user watches a video on the short video platform, the intelligent recommendation system of the short video platform will automatically recommend a series of other videos related to the video the user watched. It can quickly help users search for other videos, saving users time in manual searching, facilitating users' operations and continuously attracting users' attention.

[0003] Most existing intelligent video recommendation algorithms search for related videos, determine similarities between them, and then recommend videos to users based on this similarity. This similarity is typically determined based on video content. However, this approach doesn't factor in user interests, resulting in minimal changes in related video recommendations. This leads to users constantly watching the same type of videos, which in turn causes them to remain emotionally receptive to those videos. Consistently watching negative videos can lead to persistent negative emotions, severely impacting their viewing experience.

[0004] Therefore, the present invention provides a multi-level data optimization method and system based on big data to solve the above problems. Summary of the Invention

[0005] In view of the above situation, in order to overcome the defects of the existing technology, the present invention provides a multi-level data optimization method and system based on big data to solve the problem that the above-mentioned video intelligent recommendation algorithm does not take into account user interests, but continues to recommend the same type of videos to users. When users continue to watch negative videos, they continue to fall into negative emotions, which seriously affects the user's viewing mood.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] In a first aspect, the present invention provides a multi-level data optimization method based on big data, comprising: obtaining pre-selected data and browsing data of a user, wherein the pre-selected data is a plurality of interest types selected by the user, and the browsing data includes video data browsed by the user, duration of the video data, video type, duration of browsing of similar videos, and number of negative type videos; screening out first browsing data with a browsing duration greater than a first preset duration from the browsing data, and obtaining a plurality of first video data from the first browsing data; obtaining first feature information of the first video data from a preset feature information library, and fusing each first feature information to obtain fused feature information; based on the fused feature information, Filter out a plurality of second feature information including the fusion feature information from a preset feature information library; obtain second video data based on the second feature information; adopt a random sampling method to select a set number of video data to be played from each second video data; when the browsed video type is a negative type, determine whether the duration of browsing the negative type video is greater than a second preset duration, and determine whether the number of negative type videos browsed is greater than a preset number; if all are no, play the video data to be played in the order of update time; if one of them is yes, use a video screening model to filter out video data that meets the preset rules from the video information library and play them.

[0008] Preferably, the process of obtaining fused feature information includes: judging whether each first feature information contains the same feature data, and if so, taking the same feature data as the first fused feature data; obtaining a first preset number of third feature information close to the current viewing time in each first feature information; obtaining feature data with a repetition degree in the set repetition interval in the overall data of each third feature information as the second fused feature data; integrating the first fused feature data and the second fused feature data to obtain initial fused feature information; judging whether the data amount of the initial fused feature information is greater than the set data amount, and if so, taking the feature data in the second fused feature data that is less than or equal to the set data amount as the fused feature information; if not, taking the initial fused feature information as the fused feature information.

[0009] Preferably, the process of obtaining the fused feature information includes: determining the repetition of each feature data in the overall data of each first feature information, removing the feature data whose repetition is greater than the maximum value in the first preset repetition to obtain the feature data to be arranged; adding the feature data to be arranged to the fused feature information in order from high to low in terms of the repetition of the feature data to be arranged; when the data amount of the fused feature information reaches the set data amount, the final fused feature information is obtained.

[0010] Preferably, the processing process of the video screening model includes: obtaining the positive type of interest in the pre-selected data; and obtaining the video information library corresponding to the interest type; screening the video data in the video information library at least in accordance with the video length, video upload time, and video activity to obtain multiple initial screening video data; randomly extracting a second preset number of video data from each initial screening video data, and adding them to the to-be-played list in the order of upload time.

[0011] Preferably, the processing process of the video screening model includes: obtaining a third preset number of active video data that is currently ranked at the top in terms of activity; obtaining positive type video data from the active video data to obtain second initial video data; screening out a second preset number of video data from the second initial video data in order of activity from high to low, and adding them to the to-be-played list in order of activity from high to low.

[0012] Preferably, a multi-level data optimization method based on big data also includes: after obtaining the initial video data uploaded by the user, obtaining all video frame image data and audio data of the initial video data, using a feature extraction model constructed based on the attention mechanism to perform feature extraction on each video frame data to obtain image feature information corresponding to each video frame image data, and using an audio feature extraction model to perform feature extraction on the audio data to obtain audio feature information; using a clustering analysis model to perform cluster analysis on each image feature information and audio feature information to obtain the video type of the initial video data; and integrating each image feature information and audio feature information to obtain video feature information, and adding the video feature information to a preset feature information library.

[0013] Preferably, the storage and analysis process of the feature information in the preset feature information library includes: according to the video type of the initial video data, storing the video feature information of the initial video data in a to-be-inserted list in the sub-feature information library corresponding to the video type; according to the similarity between the video feature information of the initial video data and the existing video feature information in the sub-feature information library, inserting the video feature information of the initial video data into the next position of the feature information with the maximum similarity.

[0014] Preferably, the method of extracting features from audio data using an audio feature extraction model to obtain audio feature information includes: preprocessing the audio data; converting the preprocessed audio data into text data using a speech recognition model constructed using a recurrent neural network; and performing keyword analysis on the text data to extract key information from the text data to form audio feature information.

[0015] Preferably, a multi-level data optimization method based on big data also includes: when there is no preset feature information containing fused feature information in the preset feature information library, the ratio of repeated feature data in the preset feature information and the fused feature information to the total data of the fused feature information is used as the similarity between the preset feature information and the fused feature information, and multiple second feature information are screened out in order of similarity from high to low.

[0016] In a second aspect, the present invention provides a multi-level data optimization system based on big data, comprising:

[0017] The data acquisition module obtains the user's pre-selected data and browsing data. The pre-selected data is multiple interest types selected by the user. The browsing data includes the video data browsed by the user, the length of the video data, the video type, the duration of browsing similar videos and the number of negative type videos.

[0018] The data processing module selects first browsing data having a browsing time greater than a first preset time from the browsing data, and obtains a plurality of first video data from the first browsing data; obtains first feature information of the first video data from a preset feature information library, and fuses each first feature information to obtain fused feature information; selects a plurality of second feature information including the fused feature information from the preset feature information library based on the fused feature information; obtains second video data based on the second feature information; and selects a set number of video data to be played from each second video data using a random sampling method.

[0019] The video determination module, when the browsed video type is a negative type, determines whether the duration of browsing the negative type video is greater than a second preset duration, and determines whether the number of negative type videos browsed is greater than a preset number; if all are no, the video data to be played is played in the order of update time; if one of them is yes, the video screening model is used to screen out video data that meets the preset rules from the video information library and play it.

[0020] The beneficial effects of the present invention are:

[0021] 1. The present invention adopts a technical means of judging whether the duration of browsing negative type videos is greater than a second preset duration, and judging whether the number of negative type videos browsed is greater than a preset number when the type of video being browsed is negative; if all are negative, then the video data to be played is played in the order of update time; if one of them is yes, then a video screening model is used to screen out video data that meets the preset rules from the video information library and play it, playing positive type videos to alleviate the user's viewing mood. Through the above technical means, the present invention can solve the problem that the existing video intelligent recommendation algorithm does not take into account the user's interests, but continues to recommend the same type of videos to the user. When the user continues to watch negative type videos, the user continues to fall into negative emotions, which seriously affects the user's viewing mood.

[0022] 2. When screening video data to be played based on browsing data, the present invention adopts the method of screening out first browsing data whose browsing duration is greater than a first preset duration from the browsing data, and obtaining multiple first video data from the first browsing data; fusing the first feature information of the first video data to obtain fused feature information; based on the fused feature information, screening out multiple second feature information containing the fused feature information from a preset feature information library; obtaining second video data based on the second feature information; and selecting a set number of video data to be played from each second video data using a random sampling method. When screening multiple second feature information, the second feature information is screened according to the ratio of the fused feature information to the second feature information and at a set ratio interval, so that there is a large difference between the screened second feature information, thereby allowing users to view different videos of the same type.

[0023] 3. When screening positive video data, the present invention adopts two screening methods. The first is based on the positive interest type in the user's pre-selected data; based on the video information library corresponding to the positive type, the video data in the video information library is screened at least in accordance with the video length, video upload time, and video activity, to obtain a plurality of initial screened video data, and then a second preset number of video data is randomly selected and added to the list to be played. The second is based on the third preset number of active video data that ranks at the top of the current activity level, to screen the positive video data in the active video data to obtain the second initial video data; then, in order of activity from high to low, a second preset number of video data is screened from the second initial video data, and added to the list to be played. The present invention can screen out positive video data for users through any of the above screening methods, so that the user's negative emotions are gradually relieved and gradually restored to a happy mood. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1The present invention is a schematic flow chart of a multi-level data optimization method based on big data.

[0025] Figure 2 This is a module schematic diagram of a multi-level data optimization system based on big data in the present invention. DETAILED DESCRIPTION

[0026] The following will refer to the attached Figure 1 To the attached Figure 2 The embodiments of the present invention are described in detail. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0027] A multi-level data optimization method based on big data, as shown in the attached Figure 1 As shown, the following steps are included:

[0028] Step S11: Acquire the user's pre-selected data and browsing data.

[0029] In step S11, the pre-selected data are multiple interest types selected by the user, and the browsing data includes the video data browsed by the user, the duration of the video data, the video type, the duration of browsing similar videos, and the number of negative type videos.

[0030] Preferably, negative categories are displayed in ways that include, but are not limited to, color labels, numbers, or character codes. Video categories can be categorized by subject matter into emotion, law, society, finance, sports, education, entertainment, diplomacy, and more. Video categories, based on pre-defined evaluation criteria, include positive, negative, and neutral categories. User-uploaded videos are categorized into corresponding video categories using the keyword library for positive and negative categories.

[0031] Step S12: Filter out first browsing data with a browsing duration greater than a first preset duration from the browsing data, and obtain a plurality of first video data from the first browsing data.

[0032] Preferably, the first preset duration is 10s, 15s, 30s, or the duration corresponding to when the user's browsing time accounts for 20% of the total video duration, so as to filter out multiple first video data that the user is interested in from the browsing data, which is convenient for subsequent analysis; and filtering multiple second video data can avoid contingency.

[0033] Step S13: obtaining first feature information of the first video data from a preset feature information library, and performing fusion processing on each piece of first feature information to obtain fused feature information.

[0034] In step S13, by fusing a plurality of first feature information, common features among the first video data can be found, so that the video data to be played can be found to be more in line with the user's viewing needs based on the fused feature information.

[0035] Step S14: based on the fused feature information, a plurality of second feature information including the fused feature information is screened out from a preset feature information library; and second video data is obtained based on the second feature information.

[0036] Step S15: a set number of video data to be played are selected from each second video data by a random sampling method.

[0037] Step S16: When the browsed video type is a negative type, determine whether the duration of browsing the negative type video is greater than a second preset duration; if not, execute step S17; if so, execute step S19.

[0038] Step S17: Determine whether the number of negative type videos browsed is greater than a preset number, if so, execute step S19, if not, execute step S18.

[0039] Step S18: Play the video data to be played in the order of update time.

[0040] Step S19: Use the video screening model to screen out video data that meets the preset rules from the video information library and play it.

[0041] Preferably, the screening method in this embodiment includes but is not limited to a conditional screening algorithm, a keyword screening algorithm, a date screening algorithm, and a specified data range screening algorithm. The preset rules are the screening conditions of the video screening model.

[0042] In this embodiment, the present invention adopts the following method: when the type of video being browsed is negative, it is judged whether the duration of browsing negative type videos is greater than a second preset duration, and whether the number of negative type videos being browsed is greater than a preset number; if all are negative, the video data to be played is played in the order of update time; if one of them is yes, a video screening model is used to screen out video data that meets the preset rules from the video information library and play the video, playing positive type videos to alleviate the user's viewing mood. Through the above technical means, the present invention can solve the problem that the existing video intelligent recommendation algorithm does not take into account the user's interests, but continues to recommend the same type of videos to the user. When the user continues to watch negative type videos, the user continues to fall into negative emotions, which seriously affects the user's viewing mood.

[0043] Furthermore, when filtering the video data to be played based on the browsing data, the present invention adopts the method of filtering out first browsing data whose browsing duration is greater than a first preset duration from the browsing data, and obtaining multiple first video data from the first browsing data; fusing the first feature information of the first video data to obtain fused feature information; based on the fused feature information, filtering out multiple second feature information containing the fused feature information from a preset feature information library; obtaining second video data based on the second feature information; and selecting a set number of video data to be played from each second video data using a random sampling method. When filtering multiple second feature information, the second feature information is filtered according to the ratio of the fused feature information to the second feature information and at a set ratio interval, so that there is a large difference between the filtered second feature information, thereby allowing users to view different videos of the same type.

[0044] In an optional embodiment of the present invention, a processing process for obtaining fused feature information in step S13 includes: determining whether each first feature information contains the same feature data, and if so, using the same feature data as first fused feature data; obtaining a first preset number of third feature information close to the current viewing time from each first feature information; obtaining feature data with a repetition rate within a set repetition interval from the overall data of each third feature information as second fused feature data; integrating the first fused feature data and the second fused feature data to obtain initial fused feature information;

[0045] Determine whether the data amount of the initial fused feature information is greater than the set data amount. If it is greater, the feature data in the second fused feature data that is less than or equal to the set data amount is used as the fused feature information; if it is not greater, the initial fused feature information is used as the fused feature information.

[0046] Preferably, the first feature information includes multiple feature data. Similarly, the second feature information, the fused feature information, and the third feature information also include multiple feature data.

[0047] In this embodiment, since the first feature information corresponding to the obtained first video data is large and the analysis time is long, after obtaining the first fused feature data, the feature information of the first preset number close to the current viewing time in the multiple first feature information is searched according to the nearest principle to obtain the third feature information; then the proportion of repeated feature data in the overall data of each third feature information is compared to obtain the repetition degree of each feature data; and then the feature data with repetition degree in the set repetition interval is obtained as the second fused feature data; the method of obtaining the second fused feature data in this embodiment avoids duplication with the first fused feature data and improves work efficiency.

[0048] Preferably, the set data volume is the maximum upper limit of the initial fusion feature information, and the specific value is determined according to the actual situation to avoid affecting work efficiency when the data volume of the fusion feature information exceeds the set data volume.

[0049] When the data amount of the second fused feature data is greater than the set data amount, some feature data is randomly removed. When the data amount of the second fused feature data is reduced to be equal to or less than the set data amount, the removal is stopped to obtain fused feature information.

[0050] Through the method of this embodiment, the present invention can obtain the fusion feature data of the user watching the video, and then find the video data that fits the user's viewing needs based on the fusion feature data, thereby meeting the user's video viewing needs.

[0051] In an optional embodiment of the present invention, another processing process for obtaining the fused feature information in step S13 includes: determining the repetition of each feature data in the overall data of each first feature information, removing the feature data corresponding to the maximum value in the repetition to obtain the feature data to be arranged; adding the feature data to be arranged to the fused feature information in order from high to low in terms of the repetition of the feature data to be arranged; when the data amount of the fused feature information reaches the set data amount, the final fused feature information is obtained.

[0052] By obtaining the fused feature information in this embodiment, when subsequently screening multiple second feature information containing the fused feature information from a preset feature information library based on the fused feature information, the second feature information in the feature data corresponding to the maximum value of the recorded repetition degree among the multiple second feature information is excluded at the same time, so that the obtained multiple second feature information are constantly changing, and thus the present invention can make the screened video data gradually change, so that users can watch different video data.

[0053] In an optional embodiment of the present invention, the processing process of the video screening model includes: obtaining the positive type of interest in the pre-selected data; and obtaining the video information library corresponding to the interest type; screening the video data in the video information library at least in sequence according to the video length, video upload time, and video activity to obtain multiple initial screening video data; randomly extracting a second preset number of video data from each initial screening video data, and adding them to the to-be-played list in the order of upload time.

[0054] Preferably, in the video type of each video, there are at least a theme type and a judgment type, and the judgment type includes a positive type, a negative type and a neutral type, which are classified according to the keyword library. When it does not belong to the positive type and the negative type keyword library, it is classified as a neutral type.

[0055] Preferably, in the video information library, the preset feature information library is a part thereof, and the video information library at least further includes a video database storing video data corresponding to each preset feature information in the preset feature information library.

[0056] Preferably, the second preset number is any value between 5 and 10, and can be determined randomly.

[0057] In an optional embodiment of the present invention, the processing process of the video screening model includes: obtaining a third preset number of active video data that is currently ranked at the top in terms of activity; obtaining positive type video data from the active video data to obtain second initial video data; screening out a second preset number of video data from the second initial video data in order of activity from high to low, and adding them to a to-be-played list in order of activity from high to low.

[0058] Preferably, the value of the third preset number is 200 or 300, which can be adjusted according to actual conditions.

[0059] Preferably, the activity evaluation method includes but is not limited to evaluation based on the number of user visits, evaluation based on the number of user comments, or evaluation based on a method in which the number of user visits and user statistics each account for half of the weight.

[0060] When screening positive video data, the present invention adopts the screening methods of the above two optional embodiments. The first is based on the interest type of the positive type in the user's pre-selected data; based on the video information library corresponding to the positive type, the video data in the video information library is screened at least in accordance with the video length, video upload time, and video activity, to obtain a plurality of initial screened video data, and then a second preset number of video data is randomly selected and added to the list to be played. The second is based on the third preset number of active video data that ranks at the top of the current activity level, to screen the positive video data in the active video data to obtain the second initial video data; then, in order of activity from high to low, a second preset number of video data is screened from the second initial video data, and added to the list to be played. The present invention can screen out positive video data for users through any of the above screening methods, so that the user's negative emotions are gradually relieved and gradually restored to a happy mood.

[0061] In one embodiment of the present invention, a multi-level data optimization method based on big data also includes: after obtaining the initial video data uploaded by the user, obtaining all video frame image data and audio data of the initial video data, using a feature extraction model constructed based on the attention mechanism to perform feature extraction on each video frame data to obtain image feature information corresponding to each video frame image data, and using an audio feature extraction model to perform feature extraction on the audio data to obtain audio feature information; using a clustering analysis model to perform cluster analysis on each image feature information and audio feature information to obtain the video type of the initial video data; and integrating each image feature information and audio feature information to obtain video feature information, and adding the video feature information to a preset feature information library.

[0062] In this embodiment, when the video frame data is text data, the feature extraction model extracts the text information of the video frame data and then uses a keyword extraction algorithm to process and integrate the integrated text information to form image feature information.

[0063] When the video frame data is image data, the feature extraction model processes the image data by using an image feature extraction model constructed based on the attention mechanism and convolutional neural network. The processing flow is: first, the image data is converted into pixel data, and then the pixel data is analyzed and processed through the input layer, multiple convolution layers and output layer to obtain the image feature information of the video frame image.

[0064] Methods for constructing a cluster analysis model include, but are not limited to, using a k-means method, a hierarchical analysis method, a mixed Gaussian model, etc., to implement cluster analysis and determine the video type of the initial video.

[0065] Through the implementation of this embodiment, the present invention can analyze and process the initial video data uploaded by the user to obtain the video type and video feature information of the initial video data; and then add the video feature information to the appropriate position in the preset feature information library according to the video type and video feature information for subsequent analysis and processing, thereby improving the data processing speed.

[0066] In one embodiment of the present invention, the storage and analysis process of feature information in a preset feature information library includes: storing the video feature information of the initial video data in a to-be-inserted list in a sub-feature information library corresponding to the video type according to the video type of the initial video data; and inserting the video feature information of the initial video data in the to-be-inserted list into the next position of the feature information with the maximum similarity in the sub-feature information library according to the similarity between the video feature information of the initial video data and the existing video feature information in the sub-feature information library.

[0067] Through the storage method of this embodiment, the present invention can insert the video feature information of the initial video data into the appropriate position in the sub-feature information library, so as to perform batch processing on video data of the same video type, such as batch screening, thereby improving data processing efficiency.

[0068] In one embodiment of the present invention, an audio feature extraction model is used to perform feature extraction on audio data to obtain audio feature information, including: preprocessing the audio data; converting the preprocessed audio data into text data using a speech recognition model constructed using a recurrent neural network; performing keyword analysis on the text data, and extracting key information from the text data based on a keyword library to form audio feature information.

[0069] Preferably, during preprocessing, the audio data will be subjected to denoising, smoothing, etc. to improve the quality of the audio data so that the speech recognition model can perform recognition processing and improve the accuracy of speech recognition.

[0070] Through the implementation in this embodiment, the present invention can extract features from audio data through an audio feature extraction model to obtain audio feature information, so as to subsequently process and analyze the audio feature information and improve the speed of subsequent data processing.

[0071] In one embodiment of the present invention, a multi-level data optimization method based on big data also includes: when there is no preset feature information containing fused feature information in the preset feature information library, the ratio of repeated feature data in the preset feature information and the fused feature information to the total data of the fused feature information is used as the similarity between the preset feature information and the fused feature information, and multiple second feature information are screened out in order of similarity from high to low.

[0072] In this embodiment, when the preset feature information including the fused feature information is not found in the preset feature information library, the similarity between the preset feature information and the fused feature information is calculated, i.e., the ratio of repeated feature data in the preset feature information and the fused feature information to the total data in the fused feature information is used as the similarity. Then, multiple pieces of preset feature information are filtered from the preset feature information library in descending order of similarity to obtain multiple pieces of second feature information. Then, corresponding second video data is found based on the multiple pieces of second feature information, and subsequent selection processing is performed to obtain a set number of video data to be played.

[0073] Preferably, the number is set to any value between 5 and 10, which can be determined randomly.

[0074] Through the setting method of this embodiment, the present invention can match the appropriate second feature information when the preset feature information containing the fusion feature information cannot be found, and then obtain the appropriate video data to be played, which meets the user's viewing needs.

[0075] In one embodiment of the present invention, as shown in the attached Figure 2 As shown, the present invention also provides a multi-level data optimization system based on big data, which includes a data acquisition module, a data processing module and a video determination module.

[0076] Among them, the data acquisition module is used to obtain the user's pre-selected data and browsing data. The pre-selected data is multiple interest types selected by the user, and the browsing data includes the video data browsed by the user, the length of the video data, the video type, the duration of browsing similar videos and the number of negative type videos.

[0077] The data processing module is used to filter out first browsing data with a browsing time longer than a first preset time from the browsing data, and obtain multiple first video data from the first browsing data; obtain first feature information of the first video data from a preset feature information library, and fuse each first feature information to obtain fused feature information; based on the fused feature information, filter out multiple second feature information containing the fused feature information from the preset feature information library; obtain second video data based on the second feature information; and use a random sampling method to select a set number of video data to be played from each second video data.

[0078] The video determination module is used to determine whether the duration of browsing negative type videos is greater than a second preset duration, and whether the number of negative type videos browsed is greater than a preset number when the browsed video type is a negative type; if all are no, the video data to be played is played in the order of update time; if one of them is yes, the video screening model is used to screen out video data that meets the preset rules from the video information library and play it.

[0079] In this embodiment, the present invention can solve the problem that the existing video intelligent recommendation algorithm does not take into account user interests, but continues to recommend the same type of videos to users through the mutual cooperation between the above-mentioned modules. When users continue to watch negative videos, they continue to fall into negative emotions, which seriously affects the user's viewing mood.

[0080] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0081] It should be noted that in the description of the present invention, terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. These are merely for ease of description and do not indicate or imply that the device or component described must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0082] Furthermore, it should be noted that, in the description of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0083] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0084] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0085] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0086] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0087] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0088] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A multi-level data optimization method based on big data, characterized in that: include: Obtaining the user's pre-selected data and browsing data. The pre-selected data includes multiple interest categories selected by the user. The browsing data includes the video data browsed by the user, the duration of the video data, the video type, the duration of browsing similar videos, and the number of negative videos. Filtering out first browsing data having a browsing duration greater than a first preset duration from the browsing data, and acquiring a plurality of first video data in the first browsing data; Obtaining first feature information of the first video data from a preset feature information library, and fusing each first feature information to obtain fused feature information; including: determining whether each first feature information contains the same feature data, and if so, using the same feature data as the first fused feature data; Obtaining a first preset number of third feature information close to the current viewing time from each first feature information; obtaining feature data with a repetition rate within a set repetition interval from the overall data of each third feature information as second fused feature data; integrating the first fused feature data and the second fused feature data to obtain initial fused feature information; Determine whether the data amount of the initial fused feature information is greater than the set data amount. If so, use the feature data in the second fused feature data that is not greater than the set data amount as the fused feature information; if not, use the initial fused feature information as the fused feature information; Based on the fused feature information, multiple second feature information including the fused feature information is selected from a preset feature information library; based on the ratio of the fused feature information to the second feature information, the second feature information is selected at a set ratio interval; second video data is obtained based on the second feature information; and a set number of video data to be played are selected from each second video data using a random sampling method; After obtaining the initial video data uploaded by the user, the video frame image data and audio data of the initial video data are obtained; a feature extraction model based on the attention mechanism is used to extract features from each video frame data to obtain image feature information corresponding to each video frame image data; an audio feature extraction model is used to extract features from the audio data to obtain audio feature information; a cluster analysis model is used to perform cluster analysis on the image feature information and the audio feature information to obtain the video type of the initial video data; the image feature information and the audio feature information are integrated to obtain video feature information; and the video feature information is added to a preset feature information library; When the browsed video type is a negative type, determine whether the duration of browsing the negative type video is greater than a second preset duration, and determine whether the number of negative type videos browsed is greater than a preset number; if all are negative, play the video data to be played in the order of update time; if one of them is yes, use the video screening model to screen the video data that meets the preset rules from the video information library and play it; the processing process of the video screening model includes: obtaining the interest type of the positive type in the pre-selected data; obtaining the video information library corresponding to the interest type; at least screening the video data in the video information library according to the video length, video upload time, and video activity in turn to obtain multiple initial screened video data; randomly extracting a second preset number of video data from each initial screened video data, and adding them to the to-be-played list in the order of upload time.

2. The multi-level data optimization method according to claim 1, characterized in that: The process of obtaining the fused feature information includes: determining the repetition of each feature data in the overall data of each first feature information, removing the feature data whose repetition is greater than the maximum value in the first preset repetition, so as to obtain the feature data to be arranged; adding the feature data to be arranged to the fused feature information in descending order of the repetition of the feature data to be arranged; when the data amount of the fused feature information reaches the set data amount, obtaining the final fused feature information.

3. The multi-level data optimization method according to claim 1, characterized in that: The processing process of the video screening model includes: obtaining a third preset number of active video data that is currently ranked at the top in terms of activity; obtaining positive type video data from the active video data to obtain second initial video data; screening out a second preset number of video data from the second initial video data in order of activity from high to low, and adding them to a to-be-played list in order of activity from high to low.

4. The multi-level data optimization method according to claim 1, characterized in that: The storage and analysis process of the feature information in the preset feature information library includes: storing the video feature information of the initial video data into a to-be-inserted list in a sub-feature information library corresponding to the video type according to the video type of the initial video data; and inserting the video feature information of the initial video data into the next position of the feature information with the maximum similarity according to the similarity between the video feature information of the initial video data and the existing video feature information in the sub-feature information library.

5. The multi-level data optimization method according to claim 1, characterized in that: The method of using an audio feature extraction model to extract features from audio data to obtain audio feature information includes: preprocessing the audio data; converting the preprocessed audio data into text data using a speech recognition model constructed using a recurrent neural network; and performing keyword analysis on the text data to extract key information from the text data to form audio feature information.

6. The multi-level data optimization method according to claim 1, characterized in that: Also includes: When there is no preset feature information containing fused feature information in the preset feature information library, the ratio of repeated feature data in the preset feature information and the fused feature information to the total data of the fused feature information is used as the similarity between the preset feature information and the fused feature information, and multiple second feature information are screened out in order of similarity from high to low.

7. A multi-level data optimization system based on big data, characterized in that: include: The data acquisition module acquires the user's pre-selected data and browsing data. The pre-selected data includes multiple interest categories selected by the user. The browsing data includes the video data browsed by the user, the duration of the video data, the video type, the duration of browsing similar videos, and the number of negative type videos. A data processing module is configured to filter out first browsing data having a browsing duration greater than a first preset duration from the browsing data, and obtain a plurality of first video data from the first browsing data; Obtain first feature information of first video data from a preset feature information library, fuse each first feature information to obtain fused feature information; including: judging whether each first feature information contains the same feature data, if so, taking the same feature data as the first fused feature data; obtaining a first preset number of third feature information close to the current viewing time in each first feature information; obtaining feature data with a repetition rate within a set repetition interval in the overall data of each third feature information as the second fused feature data; integrating the first fused feature data and the second fused feature data to obtain initial fused feature information; judging whether the data volume of the initial fused feature information is greater than the set data volume, if so, taking the feature data of the second fused feature data that is not greater than the set data volume as the fused feature information; if not, taking the initial fused feature information as the fused feature information; screening the fused feature information from the preset feature information library based on the fused feature information. multiple second feature information of the information; according to the ratio of the fused feature information to the second feature information, the second feature information is filtered at a set ratio interval; the second video data is obtained according to the second feature information; a set number of video data to be played are selected from each second video data by a random sampling method; and after obtaining the initial video data uploaded by the user, the video frame image data and audio data of the initial video data are obtained; a feature extraction model constructed based on the attention mechanism is used to extract features from each video frame data to obtain image feature information corresponding to each video frame image data, and an audio feature extraction model is used to extract features from the audio data to obtain audio feature information; a cluster analysis model is used to perform cluster analysis on the image feature information and the audio feature information to obtain the video type of the initial video data; each image feature information and audio feature information is integrated to obtain video feature information; and the video feature information is added to a preset feature information library; The video determination module, when the browsed video type is a negative type, determines whether the duration of browsing the negative type video is greater than a second preset duration, and determines whether the number of negative type videos browsed is greater than a preset number; if all are negative, play the video data to be played in the order of update time; if one of them is yes, use the video screening model to filter the video data that meets the preset rules from the video information library and play it; the processing process of the video screening model includes: obtaining the interest type of the positive type in the pre-selected data; obtaining the video information library corresponding to the interest type; at least filtering the video data in the video information library according to the video length, video upload time, and video activity in turn to obtain multiple initial filtered video data; randomly extracting a second preset number of video data from each initial filtered video data, and adding them to the to-be-played list in the order of upload time.

Images