Video recommendation method, device, equipment, storage medium and program product

Abstract

The embodiments of the present application disclose a video recommendation method, apparatus, device, storage medium and program product. The method includes: obtaining image features of key frame images in a video; performing feature matching processing on the image features and preset image features containing known categories, and obtaining a first category result of the video based on the obtained matching results; obtaining text information in the video, and performing text classification on the text information to obtain a second category result of the video; summarizing the first category results and the second category results to obtain a target category result, and the target category result is used to recommend the video. Compared with the prior art in which the recommended video is determined only based on the uploader's text description of the video, the video recommendation method disclosed in the present application determines the target category result based on the category recognition results of the key frame images and text information in the video, thereby achieving a target category result for accurate video recommendation based on video content recognition, thereby improving the accuracy of video recommendation.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video recommendation method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of video acquisition equipment and broadband networks, the rapid development of the Internet and video information have become the main carriers, and the scale of video has shown explosive growth. For example, as of June 30, 2019, the total number of online video users in my country reached 758 million, of which the scale of long video users was about 639 million and the scale of short video users was about 647 million. The average monthly active user number MAU (Monthly Active User) of the online video industry from January to November 2019 was close to 1 billion, ranking second among all mobile Internet sub-industries, with a penetration rate of more than 80%, while the average MAU of the short video industry in the same period of 2019 was close to 800 million, ranking fourth among the sub-industries, with a penetration rate of about 70%.

[0003] Furthermore, the explosive growth of video scale has also increased the difficulty of video retrieval for users. At present, the method of pushing relevant video content is usually to match the keywords entered by the user with the video description determined by the video uploader. On the one hand, the above method has the problem that due to the subjective error of the video uploader in the video description, the pushed relevant video content is far from the video that the user actually wants to obtain; on the other hand, there are too many related videos to be pushed, and the user cannot accurately locate the video he wants to obtain in a short time. Summary of the invention

[0004] To solve the above technical problems, the embodiments of the present application provide a video recommendation method, device, electronic device, computer storage medium and computer program product.

[0005] According to one aspect of an embodiment of the present application, a video recommendation method is provided, the video recommendation method comprising: obtaining image features of a key frame image in a video; performing feature matching processing on the image features and preset image features containing known categories, and obtaining a first category result of the video based on the obtained matching results, wherein the preset image features of the known categories include sample features of samples contained in a preset sample database, and the samples contained in the sample database include at least one of image samples and text samples; obtaining text information in the video, and performing text classification on the text information to obtain a second category result of the video; summarizing the first category result and the second category result to obtain a target category result, and the target category result is used to recommend the video.

[0006] According to one aspect of an embodiment of the present application, a video recommendation device is provided, the device comprising: an acquisition module, configured to acquire image features of each image frame in a video; an image classification module, configured to perform feature matching processing on the image features and preset image features containing known categories, and acquire a first category result of the video according to the obtained matching result, wherein the preset image features of the known category include sample features of samples contained in a preset sample database, and the samples contained in the sample database include at least one of image samples and text samples; a text classification module, configured to acquire text information in the video, and perform text classification on the text information to obtain a second category result of the video; and a summarization module, configured to summarize the first category result and the second category result to obtain a target category result, and the target category result is used to recommend the video.

[0007] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory storing computer-readable instructions; and a processor reading the computer-readable instructions stored in the memory to execute any one of the above-mentioned video recommendation methods.

[0008] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the video recommendation method as described above.

[0009] According to one aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, which implements the steps in the video recommendation method described above when executed by a processor.

[0010] In the technical solution provided in the embodiments of the present application, a first category result is obtained based on the matching result between the image features of the key frame image in the video and the preset image features containing known categories, and a target category result is determined based on the second category result obtained based on the text information in the video, and then a video recommendation is performed based on the target category result. It can be seen that compared with the present application which determines the recommended video only based on the text description of the video by the video uploader, the present application determines the target category result based on the first category result obtained based on the image features of the key frame image in the video and the second category result obtained based on the text information, and recommends the video based on the target category result, thereby realizing the target category result for more accurate video recommendation based on video content recognition, thereby improving the accuracy of video recommendation.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0013] Figure 1 is a schematic diagram of an exemplary video recommendation effect in the prior art;

[0014] Figure 2 is another exemplary schematic diagram of video recommendation effect in the prior art;

[0015] Figure 3 It is a schematic diagram of an implementation environment involved in this application;

[0016] Figure 4 is a flowchart of a video recommendation method shown in an exemplary embodiment of the present application;

[0017] Figure 5 yes Figure 4 A schematic diagram of an exemplary effect of a key frame image mentioned in the illustrated embodiment;

[0018] Figure 6 yes Figure 4 A schematic diagram of an exemplary effect of the audio-to-text content mentioned in the illustrated embodiment;

[0019] Figure 7 yes Figure 4 A schematic diagram of an exemplary effect after the image frame containing text information mentioned in the illustrated embodiment is converted into text;

[0020] Figure 8 yes Figure 4 A schematic diagram of an exemplary effect of the recognition result mentioned in the illustrated embodiment;

[0021] Fig. 9 yes Figure 4 A flowchart of step S410 in the illustrated embodiment in an exemplary embodiment;

[0022] Fig.10 yes Fig. 9 A flowchart of step S411 in the illustrated embodiment in an exemplary embodiment;

[0023] Fig.11 yes Fig.10 A flowchart of step S4111 in the illustrated embodiment in an exemplary embodiment;

[0024] Fig.12 yes Fig. 9A flowchart of step S4112 in the illustrated embodiment in an exemplary embodiment;

[0025] Fig.13 yes Figure 4 A flowchart of step S420 in the illustrated embodiment in an exemplary embodiment;

[0026] Fig.14 yes Figure 4 A flowchart of step S430 in the illustrated embodiment in an exemplary embodiment;

[0027] Fig.15 yes Fig.14 A flowchart of step S433 in the illustrated embodiment in an exemplary embodiment;

[0028] Fig.16 yes Fig.14 A flowchart of step S4333 in the illustrated embodiment in yet another exemplary embodiment;

[0029] Fig.17 yes Fig.14 A flowchart of step S433 in the illustrated embodiment in yet another exemplary embodiment;

[0030] Fig.18 is a schematic diagram of a process of obtaining a target category result of a video, shown in an exemplary application scenario;

[0031] Fig.19 yes Figure 4 A schematic diagram of the effect of performing video recommendation based on target category results in the embodiment shown;

[0032] Fig. 20 is a block diagram of a video recommendation device shown in an exemplary embodiment of the present application;

[0033] Fig.21 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0034] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0037] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0038] First of all, it should be noted that in the Internet era, the scale of videos has shown explosive growth. Video recommendation is a way to recommend videos that users want to obtain based on user needs, and the video recommendation method of this application refers to determining content recognition results by identifying video content, and then being able to make accurate recommendations based on content recognition results when users need to search for the video. In a broader sense, video recommendation methods based on video content recognition exist in all aspects of life and can involve any technical field that applies video recommendation based on video content recognition.

[0039] In the prior art, related video content is usually pushed based on the method of matching the keywords input by the user with the video description determined by the video uploader. However, in the above method: on the one hand, due to the subjective error of the video uploader in the video description, the pushed related video content is far from the video that the user actually wants to obtain; on the other hand, there are too many related videos to be pushed, and the user cannot accurately locate the video he wants to obtain in a short time.

[0040] For example, taking the video recommendation based on the keyword "scenery" input by the user as an example, the video recommendation method in the prior art is described. Figure 1 As shown, Figure 1 The uploader of each video and the uploader's description of the video are shown in the figure. When the user enters the keyword "scenery", the videos pushed based on the match between the keyword "scenery" and the relevant video descriptions include "Scenery along the way of a certain variety show", "VR beautiful scenery short film", "Scenery of the largest lake in a certain area" and a series of other videos. There are some recommended videos that are far from the scenery videos that the user actually wants to obtain.

[0041] For example, taking the video recommendation based on the keyword "basketball star A" input by the user as an example, the video recommendation method in the prior art is described. Figure 2 As shown, Figure 2 The uploader of each video and the uploader's description of the video are shown in the figure. When the user enters "basketball star A", the videos pushed based on the match between the keyword "basketball star A" and the relevant video descriptions include "basketball star A's game video", "basketball star A's interview" video, "basketball star A's life" video and a series of other videos. The recommended videos are far from the game video of basketball star A that the user actually wants to obtain.

[0042] It can be seen that, although the video recommendation can be realized by matching keywords with the video description of the video uploader, the recommended video is far from the video that the user actually wants to obtain, and there are too many related videos recommended, and the user cannot determine the video that he wants to obtain in a short time, resulting in low accuracy of the recommended video. Based on this, in order to make accurate video recommendations, the embodiments of the present application involve: obtaining the image features of the key frame image in the video, and matching the image features of the key frame image with the preset image features containing known categories, and then obtaining the first category result of the video according to the obtained matching results; obtaining the text information contained in the video, and performing text classification on the text information to obtain the second category result of the video; summarizing the first category result and the second category result to obtain the target category result, and the target category result is used to recommend the video, so as to realize video recommendation based on video content recognition. It can be seen that the present application is based on the category recognition processing of the video in multiple aspects such as images and texts, and determines the target category result, and then makes targeted video recommendations based on the target category result, which improves the accuracy of video recommendations and avoids the problem of inaccurate video recommendations caused by determining the recommended video only based on the text description of the video by the video uploader.

[0043] Figure 3 3 is a schematic diagram of an implementation environment involved in the present application. The implementation environment includes a smart terminal 310 and a server 320, and the smart terminal 310 and the server 320 are pre-established with a wired or wireless network connection.

[0044] like Figure 3As shown, the smart terminal 310 obtains the video uploaded by the video uploader and sends the video to the server 320. The server 320 obtains the image features of the key frame image in the video, and obtains the first category result of the video based on the matching result between the image features of the key frame image and the preset image features containing known categories, and then determines the text information contained in the video, and performs category recognition processing on the text information to obtain the second category result of the video, and then summarizes the first category result and the second category result to obtain the target category result, which is used to recommend the video.

[0045] in, Figure 3 The smart terminal 310 shown can be any terminal device that supports video recommendation, such as a smart phone, a car computer, a tablet computer, a laptop computer, or a wearable device, but is not limited thereto. Figure 3 The server 320 shown is a server, for example, it can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, which are not limited here. The smart terminal 310 can communicate with the server 320 through wireless networks such as 3G (third generation mobile information technology), 4G (fourth generation mobile information technology), and 5G (fifth generation mobile information technology), which are not limited here.

[0046] See also Figure 4 , Figure 4 is a flowchart of a video recommendation method shown in an exemplary embodiment of the present application. The video recommendation method can be applied to Figure 3 The implementation environment shown is specifically executed by the server 320 in the implementation environment. It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0047] The video recommendation method proposed in the embodiment of the present application is described in detail below using the server as the specific execution entity.

[0048] like Figure 4 As shown, in an exemplary embodiment, the video recommendation method proposed in this application includes at least steps S410 to S440, which are described in detail as follows:

[0049] Step S410, obtaining image features of key frame images in the video.

[0050] First, it should be noted that a key frame image refers to an image in a video that can be processed for recognition. It should be noted that, in order to improve recognition efficiency, the key frame image can be an image that has been screened or deduplicated in the video.

[0051] The image features of the key frame image refer to those obtained based on feature extraction of the key frame image.

[0052] The server processes the video, determines the key frame images in the video, and extracts features of the key frame images to obtain image features of the key frame images. Exemplarily, the server may use a neural network model to obtain the image features of the key frame images.

[0053] Step S420, performing feature matching processing on the image features and preset image features containing known categories, and obtaining a first category result of the video according to the obtained matching result.

[0054] The preset image features containing known categories refer to the image features of the preset images, and the preset image features have corresponding category results. Exemplarily, it can be the sample features of the samples in the preset sample database. It should be noted that the sample database includes not only the sample features of the image samples, but also the sample features of the text samples. It should also be noted that in order to maintain the samples in the sample database, the server can regularly update the image samples or text samples in the sample database. It should be further noted that the server can also save the image features of the extracted key frame images to the sample database, and can also save the text features of the text information to the sample database, so as to realize the continuous updating of the sample database.

[0055] The matching result may refer to the degree of similarity between the image features of the key frame image and the image features of the preset image. Exemplarily, if the image features of the key frame image and the image features of the preset image are relatively similar, for example, the similarity between the image features of the key frame image and the image features of the preset image is greater than a preset similarity threshold, the key frame image matches the preset image; if the image features of the key frame image and the image features of the preset image are not similar, for example, the similarity between the image features of the key frame image and the image features of the preset image is less than or equal to a preset similarity threshold, the key frame image does not match the preset image.

[0056] The first category result refers to the category result determined according to the image features of the key frame image. For example, if the key frame image includes historical sites, the first category result of the key frame image can be determined as tourism, travel, or national geography. For another example, if the key frame image includes star A, the first category result of the key frame image can be determined as star A or the name of the TV series star A plays.

[0057] The server matches the key frame images in the video with the preset images containing known categories, and then obtains the first category result of the video based on the matching results. For details, please refer to Figure 5 , Figure 5 An example of a key frame image is shown in . It should be noted that the key frame image can be obtained by determining a key frame image at a preset time interval based on the video sequence of the video, for example, extracting a frame of image as a key frame image at an interval of one second; or by segmenting the video and determining an image with better clarity in each segment as a key frame image, for example, determining an image sequence of the same content in a video as one segment, such as the content of the first half of the image sequence in the video is a person, and the content of the second half of the image sequence is a certain landscape, the video can be divided into two segments, and at least one frame of clearer image in each of the first half of the image sequence and the second half of the image sequence is determined as a key frame image.

[0058] Step S430, obtaining text information in the video, and performing text classification on the text information to obtain a second category result of the video.

[0059] The text information can be determined based on the audio information in the video. For example, the audio in the video is converted into text to obtain text information. For example, the server converts the audio information in the TV series A into text and obtains the following information: Figure 6 The text information can also be determined based on the image frames containing text information in the video. For example, the image frames containing text information in the video are subjected to text recognition processing to obtain the text information. For example, the server performs recognition processing on the image frames containing text information in the TV series A to obtain the following text information: Figure 7 The text information shown. It should be noted that the server can use video OCR technology to perform text recognition processing on the image frames containing text information in the video. It should also be noted that the text information in the embodiment of the present application can be determined based on the audio information in the video, and can also be determined based on the image frames containing text information in the video, or can be determined based on the audio information in the video and the image frames containing text information. The embodiment of the present application does not limit this.

[0060] The second category of results refers to the category results determined based on the specific content of the text information in the video, for example, based on Figure 6 The category results determined by the text information in can be novels, chats, letters or lines, etc. For example, based on Figure 7 The category results determined by the text information in can be movie reviews, book reviews, or news, etc.

[0061] The server obtains the second category result of the video by performing category recognition processing on the text information contained in the video. Exemplarily, on the one hand, the server obtains the second category result by performing category recognition processing on the audio-to-text information in the video; on the other hand, the server can also obtain the second category result by performing text-to-text processing on the image frames containing text information in the video and then performing category recognition processing. It should be noted that the server can perform the above two aspects of processing on the same video to determine the second category result.

[0062] Step S440: The first category results and the second category results are aggregated to obtain target category results, and the target category results are used to recommend videos.

[0063] The target category result is determined based on summarizing the first category results and the second category results. Exemplarily, on the one hand, the target category result can be the category result after deleting the duplicate category results in the first category results and the second category results. For example, if there are category results A, B, and C in the first category results and category results C, D, and E in the second category results, then the target category results are A, B, C, D, and E. On the other hand, the target category result can also be the category result after integrating the category results of similar categories in the first category results and the second category results. For example, if there are category results A, B, and C in the first category results and category results C', D, and E in the second category results, then the target category results are A, B, C (C'), D, and E.

[0064] The server aggregates the first category results and the second category results to obtain the target category results, so that the user can quickly search for the video to be watched based on the target category results when searching for videos. Figure 8 ,based on Figure 5 The key frame images of the example in Figure 6 The category results of Example 2 and Figure 7 In Example 2, the video OCR content is Figure 5 The first category results corresponding to the key frame images in the example, Figure 6 The second category results of Example 2 and Figure 7 The second category results corresponding to the video OCR content of Example 2 are summarized and processed to obtain Figure 8 The summary results in Figure 8 The target category results are obtained by summarizing the results in . The target category results include at least tourism, scenic spot recommendations, and national geography.

[0065] It should be noted that, considering the wide variety of videos uploaded by video uploaders, in order to improve the efficiency of video recommendation, after determining the video target category results, the server can cluster based on the target category results of each video to achieve more effective video recommendation. For example, the target category result of video 1 is A, the target category result of video 2 is B, the target category result of video 3 is A, the target category result of video 4 is B, and the target category result of video 5 is A. Then, videos 1, 3, and 5 with target category results A can be clustered, and videos 2 and 4 with target category results B can be clustered, so that when the target category result A is matched, the corresponding videos 1, 3, and 5 can be recommended.

[0066] It can be seen that the video recommendation method of the embodiment of the present application determines the target category result by obtaining the first category result obtained by the matching result between the image features of the key frame images in the video and the preset image features containing known categories and the second category result obtained according to the text information in the video, and then recommends the video according to the target category result. It can be seen that compared with the present application that determines the recommended video only based on the text description of the video by the video uploader, the present application determines the target category result based on the first category result obtained based on the image features of the key frame images in the video and the second category result obtained based on the text information, and recommends the video according to the target category result, so as to obtain the target category result for more accurate video recommendation based on the video content recognition, thereby improving the accuracy of video recommendation.

[0067] Fig. 9 FIG. 1 is a flowchart of a video recommendation method according to another exemplary embodiment. Fig. 9 In the illustrated embodiment, the key frame images of the video are specifically obtained by screening the image frames contained in the video. For other steps of obtaining the target category results for video recommendation, please refer to the contents described in the aforementioned embodiments and will not be described in detail here.

[0068] like Fig. 9 As shown, in an exemplary embodiment, the process of obtaining the key frame image in the video in step S410 also includes at least steps S411 to S412, which are described in detail as follows:

[0069] Step S411, filtering the image frames contained in the video to obtain key frame images of the video.

[0070] The server obtains each image frame in the video and selects the key frame image of the video from each image frame. It should be noted that the key frame image can be obtained by determining a key frame image at a preset time interval based on the video sequence of the video, for example, extracting a frame image as a key frame image at an interval of one second.

[0071] Step S412: extract image features from the key frame image to obtain image features of the key frame image.

[0072] The image features of the key frame images are obtained based on feature extraction of the key frame images.

[0073] The server extracts image features from the key frame image to obtain image features of the key frame image. It should be noted that the server can use a deep neural network model to extract multiple features of the key frame image, such as shape, structure or other aspects, and obtain feature information of the key frame image by combining multiple features.

[0074] It can be seen that the video recommendation method of the embodiment of the present application obtains the image features of the key frame image by extracting the image features of the determined key frame image, and then obtains the first category result of the video based on the image features of the key frame image. Compared with the classification processing based on all image frames in the video, the classification processing based on the image features of the determined key frame image in the present application avoids computational redundancy and improves the efficiency of video recommendation.

[0075] Fig.10 FIG. 1 is a flowchart of a video recommendation method according to another exemplary embodiment. Fig.10 In the illustrated embodiment, a candidate frame image sequence in the video is selected by acquiring the video, and a deduplication process is performed on the candidate frame image sequence to determine the key frame image.

[0076] like Fig.10 As shown, in an exemplary embodiment, the process of filtering the video in step S411 to obtain the key frame image of the video may include the following steps:

[0077] Step S4111, selecting a candidate frame image sequence from the image frames contained in the video.

[0078] The candidate frame image sequence refers to an image sequence in a video that can be used to identify and determine a key frame image. It should be noted that the images in the candidate frame image sequence can be images with higher definition in the video, or images acquired at preset intervals in the video. This embodiment of the application does not limit this.

[0079] The server selects a candidate frame image sequence from the image frames included in the video. For example, the server may obtain an image from the video at a preset interval and determine the candidate frame image sequence according to the acquisition order.

[0080] Step S4112, performing deduplication processing on the candidate frame image sequence, and using the image frames included in the candidate frame image sequence after the deduplication processing as key frame images.

[0081] Taking into account that there may be identical or similar images in the acquired candidate frame image sequence, in order to avoid computational redundancy caused by using identical or similar images for recognition processing, the embodiment of the present application deduplicates the identical or similar images in the candidate frame image sequence to obtain a candidate frame image sequence after deduplication, that is, an image sequence containing key frame images.

[0082] It can be seen that the video recommendation method of the embodiment of the present application selects a candidate frame image sequence from each image frame of the video; and performs deduplication processing on the candidate frame image sequence to obtain a candidate frame image sequence after deduplication processing, that is, an image sequence containing key frame images. It can be seen that the series of operations of obtaining key frame images based on the candidate frame image sequence avoids computational redundancy caused by using the same or similar images for recognition processing, thereby improving image recognition efficiency.

[0083] In another embodiment, Fig.11 As shown, the candidate frame image sequence can also be determined according to the following steps, as shown in detail:

[0084] Step S41111, extract frames from the video to obtain an image frame sequence.

[0085] Frame extraction refers to a processing method of extracting images from a video. For example, the server may extract images from the video at preset intervals, or may randomly extract images from the video. This embodiment of the present application does not limit this.

[0086] The server extracts frames from the video to obtain multiple frames of images, and determines an image frame sequence based on the extraction order of the multiple frames of images. For example, the server may extract images from the video at intervals of 0.5 seconds to obtain an image frame sequence.

[0087] Step S41112, determining the degree of color difference between adjacent image frames in the image frame sequence.

[0088] The degree of chromatic aberration refers to the difference in color between image frames.

[0089] The server determines the degree of color difference between adjacent image frames based on the image frame sequence, that is, determines the color difference between adjacent image frames.

[0090] Step S41113: if the degree of color difference is greater than a preset color difference threshold, any image frame in the adjacent image frames is deleted to obtain a candidate frame image sequence.

[0091] The preset color difference threshold refers to a value used to determine the degree of color difference between two images. For example, the value can be manually set.

[0092] The server determines whether the color difference degree between adjacent image frames is greater than a preset color difference threshold based on the color difference degree between the determined adjacent image frames. If so, any image frame in the adjacent image frames is deleted, and the remaining image frames after the deletion of the image frames are determined as candidate frame images, and the candidate frame image sequence is determined based on the acquisition order of the obtained candidate frame images.

[0093] It can be seen that the video recommendation method of the embodiment of the present application determines the image frame sequence by extracting frames from the video, and determines the candidate frame image sequence based on the degree of color difference between adjacent image frames in the image frame sequence. It can be seen that the series of operations of determining the candidate frame image sequence in the present application avoids computational redundancy and improves image recognition efficiency.

[0094] In another embodiment, Fig.12 As shown, the image features of the key frame image can also be determined according to the following steps, including steps S41121 to S41122, which are described in detail as follows:

[0095] Step S41121, extracting image features of each candidate frame image in the candidate frame image sequence, and calculating the first image feature similarity between any two candidate frame images in the candidate frame image sequence.

[0096] Image features refer to features extracted from candidate frame images.

[0097] The first image feature similarity refers to the feature similarity between two candidate frame images.

[0098] The server extracts features from each candidate frame image in the candidate frame image sequence to obtain image features of each candidate frame image, and calculates the first image feature similarity between any two candidate frame images in the candidate frame image sequence. Exemplarily, the server may use a deep neural network model to extract features from each candidate frame image in the candidate frame image sequence to obtain image features of each candidate frame image.

[0099] Step S41122: if the first image feature similarity is less than the first similarity threshold, any one of the two candidate frame images is deleted to obtain a key frame image.

[0100] The first similarity threshold is a value used to determine whether two candidate frame images are similar. It should be noted that the value can be set manually.

[0101] Based on the determined similarity of the first image features between any two candidate frame images, the server determines whether the similarity of the first image features between any two candidate frame images in the candidate frame image sequence is less than a first similarity threshold. If so, any frame image of the two candidate frame images is deleted, and the remaining candidate frame images are determined to be key frame images. If not, the two candidate frame images are determined to be key frame images.

[0102] It can be seen that the video recommendation method of this embodiment determines the image features of each candidate frame image in the candidate frame image sequence, and determines the first image feature similarity between any two candidate frame images in the candidate frame image sequence; if the first image feature similarity is less than the first similarity threshold, any frame image of the two candidate frame images is deleted to obtain a key frame image. It can be seen that the application deletes the same or similar images in the candidate frame image sequence by the similarity between the candidate frame images, avoiding computational redundancy and improving image recognition efficiency.

[0103] Fig.13 FIG. 1 is a flowchart of a video recommendation method according to another exemplary embodiment. Fig.13 In the illustrated embodiment, the first category result of the video is specifically obtained based on the feature similarity between the image features of the key frame image and the preset image features containing known categories. For other steps of obtaining the target category results for video recommendation, please refer to the contents described in the aforementioned embodiments and will not be repeated here.

[0104] like Fig.13 As shown, in an exemplary embodiment, step S420 performs feature matching processing on the image features and preset image features containing known categories, and the process of obtaining the first category result of the video according to the obtained matching results also includes at least steps S421 to S422, which are described in detail as follows:

[0105] Step S421, calculating the second image feature similarity between the image feature of the key frame image and the preset image feature containing the known category.

[0106] Preset image features refer to image features of preset images. Exemplarily, they may be sample features of samples in a preset sample database. It should be noted that the sample database includes not only sample features of image samples, but also sample features of text samples. It should also be noted that in order to maintain the samples in the sample database, the server may periodically update the image or text samples in the sample database. It should further be noted that the server may also save the image features of the extracted key frame images to the sample database, and may also save the text features of the text information to the sample database, so as to continuously update the sample database.

[0107] The second image feature similarity may be a value used to represent the degree of similarity of image features between the key frame image and the preset image. For example, it may be a similarity value between the key frame image and the preset image in terms of shape features, or it may be a similarity value between the key frame image and the preset image in terms of structural features. This embodiment of the application does not limit this.

[0108] The server calculates the second image feature similarity between the image feature of the key frame image and the image feature of the preset image. Exemplarily, the server calculates the shape feature similarity between the shape feature of the key frame image and the shape feature of the preset image, and the server may also calculate the structural feature similarity between the structural feature of the key frame image and the structural feature of the preset image.

[0109] Step S422: If the second image feature similarity is greater than the second similarity threshold, a first category result of the video is generated.

[0110] The second similarity threshold is a value used to determine whether the key frame image is similar to the preset image. It should be noted that the value can be set manually.

[0111] The server determines whether the second image feature similarity is greater than a second similarity threshold based on the determined second image feature similarity between the image feature of the key frame image and the preset image feature. If so, a first category result of the key frame image is generated; if not, it indicates that the key frame image is not similar to the preset image.

[0112] It should also be noted that the server can extract features of the key frame image based on a deep neural network to obtain feature information of the key frame image. It should further be noted that the server can extract features of the key frame image in terms of shape, structure or other aspects based on a deep neural network, and use the key frame image in terms of shape, deconstruction or other features to match feature samples in a preset sample database to obtain a first category result of the key frame image.

[0113] It can be seen that the video recommendation method of this embodiment extracts image features from key frame images to obtain image features of key frame images; calculates the second image feature similarity between the image features of the key frame images and the preset image features; if the second image feature similarity is greater than the second similarity threshold, the first category result of the key frame image is generated. It can be seen that the method of determining the first category result of the video using the image feature similarity between the key frame image and the preset image in this application is conducive to the accuracy of recognition.

[0114] Fig.14 FIG. 1 is a flowchart of a video recommendation method according to another exemplary embodiment. Fig.14 In the illustrated embodiment, the second category result of the video is specifically obtained by classifying the first sub-text information determined by the audio information in the video and the second sub-text information determined by the image frame containing text information in the video. For other steps of obtaining the target category result for video recommendation, please refer to the contents described in the aforementioned embodiments and will not be repeated here.

[0115] like Fig.14As shown, in an exemplary embodiment, the process of obtaining text information in the video and classifying the text information in step S430 to obtain the second category result of the video also includes at least steps S431 to S433, which are described in detail as follows:

[0116] Step S431, obtaining audio information in the video, and converting the audio information into text to obtain first sub-text information.

[0117] Audio information refers to the sound information contained in the video, for example, the sound information of the interviewer and the interviewee in the interview video A.

[0118] Text conversion refers to converting the audio information in the video into text information.

[0119] The first sub-text information refers to text information obtained by transcribing audio information. For example, the text information obtained by transcribing audio information in interviewer video A is "Come on".

[0120] The server extracts audio information from the video, and performs text recognition processing on the audio information to obtain first subtext information.

[0121] Step S432, obtaining an image frame containing text information in the video, and performing text conversion processing on the image frame containing text information to obtain second sub-text information.

[0122] An image frame containing text information refers to an image frame displaying text information, for example, lines displayed in a certain TV series segment.

[0123] The second sub-text information refers to text information obtained by converting the image frame into text information. For example, the lines displayed in a certain TV series segment are converted into text information.

[0124] Considering that there may be text information in the image frame in the video, in order to improve the accuracy of video recommendation, the server of the embodiment of the present application performs text conversion processing on the image frame containing text information to obtain the second sub-text information. It should be noted that the server can use video OCR technology to perform text recognition processing on the image containing text information in the video to obtain the second sub-text information.

[0125] Step S433: Perform text classification based on the first subtext information and the second subtext information to obtain a second category result of the video.

[0126] Text classification refers to the category recognition process based on text information.

[0127] The server obtains the second category result of the video by performing category recognition processing on the first subtext information and the second subtext information. Exemplarily, the server may obtain the second category result of the video by performing word segmentation processing on the text information and then obtaining the second category result of the video according to the matching degree between the keywords after the word segmentation processing and the preset keywords of the corresponding category in the sample database.

[0128] It can be seen that the video recommendation method of the embodiment of the present application performs text classification processing based on the first sub-text information determined by the transcription of the audio information in the video and the second sub-text information determined by the transcription of the image frame containing text information in the video, and then obtains the second category result of the video. It can be seen that the second category result determined jointly based on the audio information in the video and the text information in the image frame improves the accuracy of category recognition.

[0129] Due to the diversity of ways of identifying text information, the embodiments of the present application use a variety of identification methods to perform category identification processing on text information containing first sub-text information and second sub-text information. Exemplarily, the server can determine the second category result of the text information through the text feature similarity between the text information and the preset text information containing a known category; the server can also search through the key information in the text information to determine the second category result of the text information; or, the server can also perform word segmentation recognition processing on the text information to determine the second category result of the text information. It should be noted that the server can use any of the above methods to determine the second category result of the text information; it can also combine any two of the above three methods to determine the second category result of the text information; or, the server uses the above three methods to comprehensively determine the second category result of the text information. For details, please refer to the following solutions.

[0130] In another embodiment, Fig.15 As shown, the process of performing text classification based on the first subtext information and the second subtext information to obtain the second category result of the video also includes at least steps S4331 to S4332, which are described in detail as follows:

[0131] Step S4331 : Perform text feature extraction processing on the first subtext information and the second subtext information respectively to obtain text features of the first subtext information and text features of the second subtext information.

[0132] The text feature of the first sub-text information is obtained by extracting the text features of the first sub-text information.

[0133] The text feature of the second sub-text information is obtained by extracting the text features of the second sub-text information.

[0134] The server extracts features of the text information of the audio-to-text or image-to-text containing text information to obtain text features of the first subtext information and text features of the second subtext information. Exemplarily, the server may use a deep neural network model to extract features of the first subtext information and the second subtext information, respectively.

[0135] Step S4332, calculating the text feature similarity between the text feature and the preset text features containing known categories.

[0136] The preset text feature is feature information of a preset text. For example, it may be a sample feature of a sample in a preset sample database. It should be noted that the sample database includes not only sample features of image samples, but also sample features of text samples.

[0137] The server calculates the text feature similarity between the text feature and the preset text feature, that is, the server calculates the similarity between the text feature and the text feature of the sample in the preset sample database.

[0138] Step S4333: If the text feature similarity is greater than the third similarity threshold, a second category result of the video is generated.

[0139] The third similarity threshold is a value used to determine whether the text feature of the first sub-text information and the text feature of the second sub-text information are similar to the preset text feature. It should be noted that the value can be set manually.

[0140] The server determines whether the text feature similarity is greater than a third similarity threshold based on the determined text feature similarity between the text feature of the first sub-text information and the text feature of the second sub-text information and the preset text feature; if so, a second category result of the video is generated; if not, it indicates that the text feature of the first sub-text information and the text feature of the second sub-text information are not similar to the preset text feature.

[0141] It can be seen that the video recommendation method of this embodiment obtains the second category result of the video through the text feature similarity between the first sub-text information and the second sub-text information and the preset text information, that is, the second category result of the video is obtained according to the text feature similarity between the text features of the first sub-text information and the second sub-text information and the text features of the samples in the preset sample database, which is beneficial to the accuracy of the first sub-text information and the second sub-text information category recognition.

[0142] In another embodiment, Fig.16 As shown, the process of performing text classification based on the first subtext information and the second subtext information to obtain the second category result of the video also includes at least steps S4334 to S4335, which are described in detail as follows:

[0143] Step S4334: preprocess the first subtext information and the second subtext information to obtain preprocessed first subtext information and preprocessed second subtext information.

[0144] Preprocessing refers to performing word segmentation on the first subtext information and the second subtext information. For example, if the text information is ABCDE, the preprocessing is to segment the text information into AB, C, and DE.

[0145] The server performs word segmentation processing on the first subtext information and the second subtext information, so as to facilitate category recognition based on the preprocessed first subtext information and the preprocessed second subtext information, thereby improving text information recognition efficiency.

[0146] Step S4335: perform recognition processing on the preprocessed first subtext information and the preprocessed second subtext information to obtain a second category result of the video.

[0147] The server performs recognition processing on the preprocessed first subtext information and the preprocessed second subtext information, that is, recognizes and processes the first subtext information after word segmentation processing and the second subtext information after word segmentation processing, and obtains the second category result of the video. For example, continuing with the example in step S4333, the server uses AB, C, and DE after word segmentation processing for recognition processing, that is, it can use AB for category recognition processing, and can also use DE for recognition processing, which is conducive to obtaining the second category result.

[0148] It can be seen that the video recommendation method of this embodiment performs word segmentation on the first sub-text information and the second sub-text information, and then performs text recognition on the first sub-text information after word segmentation and the second sub-text information after word segmentation. Compared with the overall recognition of the first sub-text information and the second sub-text information, word segmentation recognition is beneficial to determining the second category result of the video, thereby improving recognition efficiency and accuracy.

[0149] In another embodiment, Fig.17 As shown, the process of performing text classification based on the first subtext information and the second subtext information to obtain the second category result of the video also includes at least steps S4336 to S4337, which are described in detail as follows:

[0150] Step S4336: perform key information identification on the first subtext information and the second subtext information to obtain key information respectively included in the first subtext information and the second subtext information.

[0151] The key information refers to the keyword information in the first subtext information and the second subtext information. For example, the text information is ABCDE, and the key information is BC.

[0152] The server performs keyword recognition on the first subtext information and the second subtext information respectively to obtain key information in the first subtext information and key information in the second subtext information, thereby facilitating retrieval processing using the key information in the first subtext information and the key information in the second subtext information.

[0153] Step S4337, use the key information to perform retrieval processing, and generate a second category result based on the obtained retrieval results.

[0154] The search result may refer to text information in which key information in the first subtext information and key information in the second subtext information match sample text information in a preset sample database.

[0155] The server determines matching sample text information in a preset database based on key information in the first subtext information and key information in the second subtext information, and determines a second category result of the video based on the category to which the matching sample text information in the sample database belongs.

[0156] It can be seen that the video recommendation method of this embodiment obtains the second category result of the video by performing key information recognition processing on the first sub-text information and the second sub-text information, and performing retrieval processing on the key information in the first sub-text information and the key information in the second sub-text information. That is, the server uses the key information in the first sub-text information and the key information in the second sub-text information to match the sample text information in the sample database to obtain sample information that can be used to determine the text information. Compared with using the entire text information for matching and retrieval in the sample database, key information retrieval is conducive to improving recognition efficiency and accuracy.

[0157] In order to clearly describe the above embodiments, the following is based on Fig.18 The above embodiment is described in detail in the specific flow chart. Fig.18 As shown, the server obtains the video and breaks down the video content into image, audio, text information and other data, and classifies them according to the match between the image, audio, text information and other data and the corresponding samples in the sample database, and then makes a possibility judgment on the relevant categories based on the category recognition results of the image, audio and text information, and tries to remove irrelevant video content that may appear when the user searches for videos, thereby reducing the user's search time, reducing time cost, and improving the user experience.

[0158] On the one hand, the server extracts frames from the images in the video to obtain an image frame sequence, and compares the degree of color difference between adjacent image frames in the image frame sequence to obtain a candidate frame image sequence, and then deduplicates the same or similar images in the candidate frame image sequence to obtain key frame images. Feature points are then extracted from the key frame images in terms of shape, deconstruction, and complex features, and the second image feature similarity between the image features of the key frame images and preset image features of known categories is calculated. If the second image feature similarity is greater than the second similarity threshold, the first category result is obtained.

[0159] On the other hand, the server performs audio-to-text processing on the audio information to obtain the first subtext information, and performs text-to-text processing on the image containing the text information to obtain the second subtext information, and determines the second category result of the video based on the text feature similarity between the text information containing the first subtext information and the second subtext information and the preset text features containing the known category. Alternatively, the text containing the first subtext information and the second subtext information is preprocessed to obtain the preprocessed first subtext information and the preprocessed second subtext information, and the preprocessed first subtext information and the preprocessed second subtext information are identified to obtain the second category result of the video. Alternatively, the text information containing the first subtext information and the second subtext information is identified for key information to obtain the key information respectively contained in the first subtext information and the second subtext information; the key information is used for retrieval processing, and the second category result is generated based on the obtained retrieval results.

[0160] The server aggregates the first category results and the second category results to obtain the target category results for video recommendation. Fig.19 , Fig.19 2 shows a schematic diagram of the effect of video recommendation based on the target category results. Among them, "National Geographic" and the like under the recommended content are the above-mentioned target category results.

[0161] It should be noted that the method for determining the second category results based on the text information is as follows: the server determines the second category results of the text information based on the text feature similarity between the text information and the preset text information; or, the server searches through the key information in the text information to determine the second category results of the text information; and, the server performs word segmentation recognition processing on the text information to determine the second category results of the text information.

[0162] Fig. 20 is a block diagram of a video recommendation device shown in an exemplary embodiment of the present application. The video recommendation device can be applied to Figure 3The video recommendation device may also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the video content recognition-based recommendation device is applicable.

[0163] like Fig. 20 As shown, the exemplary video recommendation device 2000 includes an acquisition module 2010, an image classification module 2020, a text classification module 2030 and a summary module 2040, specifically:

[0164] The acquisition module 2010 is configured to acquire image features of each image frame in the video.

[0165] The image classification module 2020 is configured to perform feature matching processing on the image features and the preset image features, and obtain the first category result of the video according to the obtained matching results.

[0166] The text classification module 2030 is configured to obtain text information in the video and perform text classification on the text information to obtain a second category result of the video.

[0167] The aggregation module 2040 is configured to aggregate the first category results and the second category results to obtain a target category result, and the target category result is used to recommend the video.

[0168] In this exemplary video recommendation device, a target category result is determined based on a first category result obtained according to the matching result between image features of key frame images in a video and preset image features containing known categories, and a second category result obtained according to text information in the video, and then a video recommendation is performed based on the target category result. It can be seen that compared with the present application which determines the recommended video only based on the text description of the video by the video uploader, the present application determines the target category result based on the category recognition results of both the key frame images and text information in the video, and recommends the video based on the target category result, thereby achieving target category results for more accurate video recommendation based on video content recognition, thereby improving the accuracy of video recommendation.

[0169] Based on the above exemplary embodiment, the acquisition module 2010 further includes a screening module and a feature extraction module. Specifically:

[0170] The screening module is configured to screen the image frames contained in the video to obtain the key frame images of the video.

[0171] The deduplication module is configured to extract image features from the key frame images to obtain image features of the key frame images.

[0172] In this exemplary video recommendation device, the image features of a certain key frame image are extracted to obtain the image features of the key frame image, and then the first category result of the video is obtained based on the image features of the key frame image. Compared with classifying all image frames in the video, the present application performs classification based on the image features of the certain key frame image, which avoids computational redundancy and improves the efficiency of video recommendation.

[0173] Based on the above exemplary embodiment, the screening module includes a selection module and a deduplication processing module. Specifically:

[0174] The selection module is configured to select a candidate frame image sequence from the image frames contained in the video.

[0175] The deduplication processing module is configured to perform deduplication processing on the candidate frame image sequence, and use the image frames contained in the candidate frame image sequence after the deduplication processing as key frame images.

[0176] In the exemplary video recommendation device, a candidate frame image sequence is selected from each image frame of the video; and the candidate frame image sequence is deduplicated to obtain a candidate frame image sequence after deduplication, that is, an image sequence containing key frame images. It can be seen that the series of operations of obtaining key frame images based on the candidate frame image sequence avoids computational redundancy caused by using the same or similar images for recognition processing, thereby improving image recognition efficiency.

[0177] Based on the above exemplary embodiment, the selection module includes a frame extraction module, a color difference determination module and a deletion module. Specifically:

[0178] The frame extraction module is configured to perform frame extraction processing on the video to obtain an image frame sequence.

[0179] The color difference determination module is configured to determine the degree of color difference between adjacent image frames in the image frame sequence.

[0180] The deletion module is configured to delete any image frame in adjacent image frames if the degree of color difference is greater than a preset color difference threshold, so as to obtain a candidate frame image sequence.

[0181] In this exemplary video recommendation device, the image frame sequence is determined by extracting frames from the video, and the candidate frame image sequence is determined based on the degree of color difference between adjacent image frames in the image frame sequence. It can be seen that the series of operations of determining the candidate frame image sequence in this application avoids computational redundancy and improves image recognition efficiency.

[0182] Based on the above exemplary embodiment, the deduplication processing module includes a calculation module and a judgment and deletion module. Specifically:

[0183] The calculation module is configured to extract image features of each candidate frame image in the candidate frame image sequence, and calculate the first image feature similarity between any two candidate frame images in the candidate frame image sequence.

[0184] The judgment and deletion module is configured to delete any one of the two candidate frame images if the first image feature similarity is less than a first similarity threshold to obtain a key frame image.

[0185] In this exemplary video recommendation device, by determining the image features of each candidate frame image in the candidate frame image sequence, and determining the first image feature similarity between any two candidate frame images in the candidate frame image sequence; if the first image feature similarity is less than the first similarity threshold, then deleting any one of the two candidate frame images to obtain a key frame image. It can be seen that the present application deletes the same or similar images in the candidate frame image sequence by the similarity between the candidate frame images, thereby avoiding computational redundancy and improving image recognition efficiency.

[0186] Based on the above exemplary embodiment, the image classification module 2020 includes a second image feature similarity calculation module and a second image feature similarity determination module. Specifically:

[0187] The second image feature similarity calculation module is configured to calculate the second image feature similarity between the image feature and the preset image features containing known categories.

[0188] The second image feature similarity judgment module is configured to generate a first category result of the key frame image if the second image feature similarity is greater than a second similarity threshold.

[0189] In this exemplary video recommendation device, image features of the key frame image are extracted to obtain the image features of the key frame image; the second image feature similarity between the image features of the key frame image and the preset image features is calculated; if the second image feature similarity is greater than the second similarity threshold, the first category result of the key frame image is generated. It can be seen that the method of determining the first category result of the video by using the image feature similarity between the key frame image and the preset image in this application is conducive to the accuracy of recognition.

[0190] Based on the above exemplary embodiment, the text classification module 2030 includes a first sub-text information acquisition module, a second sub-text information acquisition module and a second classification result determination module. Specifically:

[0191] The first sub-text information acquisition module is configured to acquire audio information in the video and perform text conversion on the audio information to obtain the first sub-text information.

[0192] The second sub-text information acquisition module is configured to acquire image frames containing text information in the video, and perform text conversion processing on the image frames containing text information to obtain second sub-text information.

[0193] The second classification result determination module is configured to perform text classification based on the first sub-text information and the second sub-text information to obtain a second category result of the video.

[0194] In this exemplary video recommendation device, text classification processing is performed based on first sub-text information determined by transcribing audio information in the video and second sub-text information determined by transcribing image frames containing text information in the video, thereby obtaining a second category result of the video, thereby improving the accuracy of the second category result determined jointly based on the audio information in the video and the text information in the image frames.

[0195] Based on the above exemplary embodiment, the second classification result determination module includes a text feature extraction module, a text feature similarity calculation module and a text feature similarity judgment module. Specifically:

[0196] The text feature extraction module is configured to perform text feature extraction processing on the first sub-text information and the second sub-text information respectively to obtain text features of the first sub-text information and text features of the second sub-text information.

[0197] The text feature similarity calculation module is configured to calculate the text feature similarity between the text feature and the preset text features containing known categories.

[0198] The text feature similarity judgment is configured to generate a second category result of the video if the text feature similarity is greater than a third similarity threshold.

[0199] In this exemplary video recommendation device, the second category result of the video is obtained by the text feature similarity between the first sub-text information and the second sub-text information and the preset text information, that is, the second category result of the video is obtained by the text feature similarity between the text features of the first sub-text information and the second sub-text information and the text features of the sample in the preset sample database, which is beneficial to the accuracy of the first sub-text information and the second sub-text information category identification.

[0200] Based on the above exemplary embodiment, the second classification result determination module includes a text preprocessing module and a recognition module after preprocessing. Specifically:

[0201] The text preprocessing module is configured to preprocess the first subtext information and the second subtext information to obtain the preprocessed first subtext information and the preprocessed second subtext information, wherein the preprocessing includes word segmentation processing.

[0202] The preprocessed recognition module is configured to perform recognition processing on the preprocessed first subtext information and the preprocessed second subtext information to obtain a second category result of the video.

[0203] In this exemplary video recommendation device, word segmentation is performed on the first sub-text information and the second sub-text information, and then text recognition is performed on the first sub-text information after word segmentation and the second sub-text information after word segmentation. Compared with the overall recognition of the first sub-text information and the second sub-text information, word segmentation recognition is beneficial to determining the second category result of the video, thereby improving recognition efficiency and accuracy.

[0204] Based on the above exemplary embodiment, the second classification result determination module includes a key information identification module and a retrieval module. Specifically:

[0205] The key information identification module is configured to perform key information identification on the first subtext information and the second subtext information to obtain the key information respectively contained in the first subtext information and the second subtext information.

[0206] The retrieval module is configured to perform retrieval processing using the key information and generate a second category result based on the obtained retrieval results.

[0207] In the exemplary video recommendation device, the second category result of the video is obtained by performing key information recognition processing on the first sub-text information and the second sub-text information, and performing retrieval processing on the key information in the first sub-text information and the key information in the second sub-text information. That is, the server uses the key information in the first sub-text information and the key information in the second sub-text information to match the sample text information in the sample database to obtain sample information that can be used to determine the text information. Compared with using the entire text information for matching and retrieval in the sample database, key information retrieval is conducive to improving recognition efficiency and accuracy.

[0208] It should be noted that the video recommendation device provided in the above embodiment and the video recommendation method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In actual application, the video recommendation device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0209] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the video recommendation method provided in the above-mentioned embodiments.

[0210] Fig.21 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Fig.21 The computer system 2100 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0211] like Fig.21 As shown, the computer system 2100 includes a central processing unit (CPU) 2101, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 2102 or the program loaded from the storage part 2108 to the random access memory (RAM) 2103, such as executing the method described in the above embodiment. Various programs and data required for system operation are also stored in the RAM 2103. The CPU 2101, the ROM 2102 and the RAM 2103 are connected to each other through the bus 2104. The input / output (I / O) interface 2105 is also connected to the bus 2104.

[0212] The following components are connected to the I / O interface 2105: an input section 2106 including a keyboard, a mouse, etc.; an output section 2107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 2108 including a hard disk, etc.; and a communication section 2109 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 2109 performs communication processing via a network such as the Internet. A drive 2110 is also connected to the I / O interface 2105 as needed. A removable medium 2111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 2110 as needed so that a computer program read therefrom is installed into the storage section 2108 as needed.

[0213] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 2109, and / or installed from a removable medium 2111. When the computer program is executed by the central processing unit (CPU) 2101, various functions defined in the system of the present application are executed.

[0214] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier, which carries a computer-readable computer program. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0215] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0216] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0217] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the video recommendation method as described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0218] Another aspect of the present application further provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the video recommendation method provided in each of the above embodiments.

[0219] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A video recommendation method, characterized in that: The method comprises: Acquire image features of a key frame image in a video, wherein the image features are obtained by performing multi-faceted feature extraction on the key frame image using a deep network model; Calculating a second image feature similarity between the image feature and a preset image feature of a known category; wherein the preset image feature of the known category includes a sample feature of a sample included in a preset sample database, and the sample included in the sample database includes at least one of an image sample and a text sample; If the second image feature similarity is greater than a second similarity threshold, generating a first category result of the key frame image; Acquire text information in the video, and perform text classification on the text information to obtain a second category result of the video; The first category result and the second category result are aggregated to obtain a target category result, where the target category result is used to recommend the video, and the target category result includes at least two categories corresponding to the video.

2. The method according to claim 1, characterized in that The step of obtaining image features of key frame images in the video includes: Screening the image frames contained in the video to obtain key frame images of the video; Image features are extracted from the key frame image to obtain image features of the key frame image.

3. The method according to claim 2, characterized in that The step of screening the video to obtain the key frame image of the video includes: Selecting a candidate frame image sequence from the image frames contained in the video; The candidate frame image sequence is deduplicated, and the image frames included in the candidate frame image sequence after the deduplication process are used as the key frame images.

4. The method according to claim 3, characterized in that The step of selecting a candidate frame image sequence from the image frames contained in the video comprises: Performing frame extraction processing on the video to obtain an image frame sequence; Determining a degree of color difference between adjacent image frames in the image frame sequence; If the color difference degree is greater than a preset color difference threshold, any image frame in the adjacent image frames is deleted to obtain the candidate frame image sequence.

5. The method according to claim 3, characterized in that: The step of performing deduplication processing on the candidate frame image sequence and using the image frames contained in the deduplicated candidate frame image sequence as the key frame image comprises: Extracting image features of each candidate frame image in the candidate frame image sequence, and calculating the first image feature similarity between any two candidate frame images in the candidate frame image sequence; If the first image feature similarity is less than a first similarity threshold, any one of the two candidate frame images is deleted to obtain the key frame image.

6. The method according to claim 1, characterized in that The step of obtaining text information in the video and performing text classification on the text information to obtain a second category result of the video includes: Acquire audio information in the video, and convert the audio information into text to obtain first subtext information; Acquire an image frame containing text information in the video, and perform text conversion processing on the image frame containing text information to obtain second sub-text information; Text classification is performed based on the first sub-text information and the second sub-text information to obtain a second category result of the video.

7. The method according to claim 6, characterized in that The step of performing text classification based on the first sub-text information and the second sub-text information to obtain a second category result of the video includes: Performing text feature extraction processing on the first subtext information and the second subtext information respectively to obtain text features of the first subtext information and text features of the second subtext information; Calculating text feature similarity between the text feature and preset text features containing known categories; If the text feature similarity is greater than a third similarity threshold, a second category result of the video is generated.

8. The method according to claim 6, characterized in that The step of performing text classification based on the first sub-text information and the second sub-text information to obtain a second category result of the video includes: Preprocessing the first subtext information and the second subtext information to obtain preprocessed first subtext information and preprocessed second subtext information, wherein the preprocessing includes word segmentation processing; The preprocessed first subtext information and the preprocessed second subtext information are identified to obtain a second category result of the video.

9. The method according to claim 6, characterized in that The step of performing text classification based on the first sub-text information and the second sub-text information to obtain a second category result of the video includes: Performing key information identification on the first subtext information and the second subtext information to obtain key information respectively included in the first subtext information and the second subtext information; The key information is used to perform a search process, and the second category results are generated according to the obtained search results.

10. A video recommendation device, characterized in that: The device comprises: An acquisition module is configured to acquire image features of a key frame image in a video, wherein the image features are obtained by performing multi-faceted feature extraction on the key frame image using a deep network model; An image classification module, configured to calculate a second image feature similarity between the image feature and a preset image feature of a known category; wherein the preset image feature of the known category comprises a sample feature of a sample contained in a preset sample database, and the sample contained in the sample database comprises at least one of an image sample and a text sample; If the second image feature similarity is greater than a second similarity threshold, generating a first category result of the key frame image; A text classification module is configured to obtain text information in the video and perform text classification on the text information to obtain a second category result of the video; The aggregation module is configured to aggregate the first category result and the second category result to obtain a target category result, wherein the target category result is used to recommend the video, and the target category result includes at least two categories corresponding to the video.

11. An electronic device, characterized in that: include: a memory storing computer-readable instructions; A processor reads the computer-readable instructions stored in the memory to execute the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images