Content Search Method, Apparatus, Terminal, Storage Medium and Program Product

Abstract

Embodiments of the present application disclose a content search method, apparatus, terminal, storage medium, and program product. Embodiments of the present application obtain multimedia data, such as video, and the feedback volume corresponding to a multimedia segment. The multimedia data includes multiple multimedia segments, and each multimedia segment includes multiple multimedia elements. The feedback volume characterizes the degree of user feedback on the multimedia segment. At least one key segment is determined in the multimedia data according to the feedback volume corresponding to each multimedia segment. Feature extraction is performed on the multimedia elements in the key segment to obtain the segment features of the key segment. The overall features of the multimedia data are determined according to the segment features of the key segment. Content search is performed based on the overall features of the multimedia data to obtain a search result corresponding to the multimedia data. Embodiments of the present application can improve the efficiency of content search.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular, to a content search method, apparatus, terminal, storage medium, and program product. Background Art

[0002] In recent years, videos, audios, etc. have grown explosively on the Internet. In order to reduce the search difficulty of audios, videos, etc., features (Embedding) are often used to represent the content of audios, videos, etc. A content search website can measure the semantic similarity between two audios or two videos through the distance between two features. The more similar two audios or two videos are, the smaller the feature distance between them, so that the content search website can perform content search through features.

[0003] However, current content search requires a large amount of computing time, resulting in low content search efficiency. Summary of the Invention

[0004] Embodiments of this application provide a content search method, apparatus, terminal, storage medium, and program product, which can improve the content search efficiency.

[0005] Embodiments of this application provide a content search method, including:

[0006] Obtain multimedia data and the feedback amount corresponding to a multimedia segment. The multimedia data includes multiple multimedia segments, and each multimedia segment includes multiple multimedia elements. The feedback amount represents the degree of user feedback on the multimedia segment;

[0007] Determine at least one key segment in the multimedia data according to the feedback amount corresponding to each multimedia segment;

[0008] Extract features of the multimedia elements in the key segment to obtain the segment features of the key segment;

[0009] Determine the overall features of the multimedia data according to the segment features of the key segment;

[0010] Perform content search based on the overall features of the multimedia data to obtain the search results corresponding to the multimedia data.

[0011] Embodiments of this application also provide a content search apparatus, including:

[0012] An obtaining unit, configured to obtain multimedia data and the feedback amount corresponding to a multimedia segment. The multimedia data includes multiple multimedia segments, and each multimedia segment includes multiple multimedia elements. The feedback amount represents the degree of user feedback on the multimedia segment;

[0013] A first determination unit, configured to determine at least one key segment in the multimedia data according to the feedback amount corresponding to each multimedia segment;

[0014] A feature extraction unit, configured to extract features of multimedia elements in the key segment to obtain segment features of the key segment;

[0015] A second determination unit, configured to determine the overall feature of the multimedia data according to the segment features of the key segment;

[0016] A content search unit, configured to perform content search based on the overall feature of the multimedia data to obtain a search result corresponding to the multimedia data.

[0017] In some embodiments, determining at least one key segment in the multimedia data according to the feedback amount corresponding to each multimedia segment includes:

[0018] Performing sorting processing on multiple multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted by the feedback amount;

[0019] Determining at least one key segment in the multimedia data according to the sorted multimedia segments.

[0020] In some embodiments, determining at least one key segment in the multimedia data according to the sorted multimedia segments includes:

[0021] When the number of multimedia segments in the multimedia data is greater than a preset quantity, determining the preset quantity of key segments in the multimedia data according to the sorted multimedia segments.

[0022] In some embodiments, further includes:

[0023] When the number of multimedia segments in the multimedia data is not greater than the preset quantity, using each multimedia segment in the multimedia data as a key segment.

[0024] In some embodiments, extracting features of multimedia elements in the key segment to obtain segment features of the key segment includes:

[0025] Dividing the preset quantity by the number of multimedia segments in the multimedia data to obtain a target number;

[0026] Determining the target number of target elements from the key segment;

[0027] Extracting features of the target elements in the key segment to obtain segment features of the key segment.

[0028] In some embodiments, determining the target number of target elements from the key segment includes:

[0029] When the number of targets is a decimal, round the number of targets to obtain the rounded number of targets;

[0030] Multiply the rounded number of targets by the number of multimedia segments to obtain the rounded total number;

[0031] Determine the number difference, where the number difference is the difference between the rounded total number and the preset quantity;

[0032] Among multiple multimedia segments, determine N - 1 first key segments and 1 second key segment, where N is the number of multimedia segments;

[0033] Determine the target number of target elements from the first key segments;

[0034] Determine the target number of target elements from the second key segment, where the new target number is the sum of the number difference and the rounded number of targets.

[0035] In some embodiments, determining N - 1 first key segments and 1 second key segment among multiple multimedia segments includes:

[0036] When multiple multimedia segments are sorted in descending order of the feedback amount, the first key segments are the first N - 1 segments among the sorted multimedia segments;

[0037] When multiple multimedia segments are sorted in ascending order of the feedback amount, the first key segments are the last N - 1 segments among the sorted multimedia segments.

[0038] In some embodiments, the multimedia element is composed of text data and images. Sorting multiple multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted by the feedback amount includes:

[0039] When the sum of the feedback amounts corresponding to all multimedia segments in the multimedia data is not less than the preset feedback amount, sort multiple multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted by the feedback amount;

[0040] When it is determined that all feedback amounts in the multimedia data are less than the preset feedback amount, determine the text amount ratio of the multimedia segment, where the text amount ratio of the multimedia segment is the ratio of the text amount of all text data in the multimedia data to the text amount of all text data in the multimedia segment;

[0041] Determine the color ratio, where the color ratio is the ratio of the image color change amount of all images in the multimedia data to the image color change amount of all images in the multimedia segment, and the image color change amount represents the degree of color change of the image;

[0042] Sort multiple multimedia segments according to the text volume ratio and color ratio to obtain the sorted multimedia segments.

[0043] In some embodiments, the multimedia elements are composed of images. Sort multiple multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted by the feedback amount, including:

[0044] When the sum of the feedback amounts corresponding to all multimedia segments in the multimedia data is not less than the preset feedback amount, sort multiple multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted by the feedback amount;

[0045] When it is determined that all feedback amounts in the multimedia data are less than the preset feedback amount, determine the color ratio. The color ratio is the ratio of the image color change amount of all multimedia elements in the multimedia data to the image color change amount of all multimedia elements in the multimedia segment. The image color change amount represents the degree of color change of the image;

[0046] Sort multiple multimedia segments according to the color ratio to obtain the multimedia segments arranged by the color ratio.

[0047] In some embodiments, the multimedia elements are composed of text data. Sort multiple multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted by the feedback amount, including:

[0048] When the sum of the feedback amounts corresponding to all multimedia segments in the multimedia data is not less than the preset feedback amount, sort multiple multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted by the feedback amount;

[0049] When all feedback amounts in the multimedia data are less than the preset feedback amount, determine the text volume ratio of the multimedia segment. The text volume ratio of the multimedia segment is the ratio of the text volume of all text data in the multimedia data to the text volume of all text data in the multimedia segment;

[0050] Sort multiple multimedia segments according to the text volume ratio to obtain the multimedia segments sorted by the text volume ratio.

[0051] In some embodiments, determining the color ratio includes:

[0052] Determine the pixel color difference between each pixel of two adjacent frames of multimedia elements in the multimedia segment;

[0053] Sum up the pixel color differences between each pixel of two adjacent frames of multimedia elements to obtain the color difference between two adjacent frames of multimedia elements;

[0054] Sum the color differences between all adjacent two frames of multimedia elements in the multimedia clip to obtain the color difference corresponding to the multimedia clip;

[0055] Sum the color differences corresponding to all multimedia clips in the multimedia data to obtain the color difference corresponding to the multimedia data;

[0056] Determine the color ratio according to the color difference corresponding to the multimedia data and the color difference corresponding to the multimedia clip.

[0057] In some embodiments, the multimedia element is composed of images. Before obtaining the multimedia data, it further includes:

[0058] Determine the target element from the multimedia elements of the multimedia data according to the pixel color value of each pixel in the multimedia element;

[0059] Segment the multimedia data according to the target element to obtain a plurality of multimedia clips.

[0060] In some embodiments, determining the target element from the multimedia elements of the multimedia data according to the pixel color value of each pixel in the multimedia element includes:

[0061] Determine the color mean value of each frame of multimedia element, and the color mean value is the average value of the pixel colors of all pixels in the multimedia element;

[0062] Use the multimedia element with the pixel color mean value greater than the preset pixel color as the target element.

[0063] In some embodiments, determining the target element from the multimedia elements of the multimedia data according to the pixel color value of each pixel in the multimedia element includes:

[0064] Determine the color change value of adjacent two frames of multimedia elements according to the pixel color value of each pixel of the adjacent two frames of multimedia elements;

[0065] When it is determined that the color change value is greater than the preset color change value, use any one frame of the adjacent two frames of multimedia elements as the target element.

[0066] An embodiment of the present application further provides an electronic device, including a memory storing multiple instructions; the processor loads the instructions from the memory to execute the steps in any one of the content search methods provided by the embodiments of the present application.

[0067] An embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores multiple instructions, and the instructions are suitable for being loaded by a processor to execute the steps in any one of the content search methods provided by the embodiments of the present application.

[0068] An embodiment of the present application also provides a computer program product, including computer programs / instructions, which when executed by a processor implement the steps in any of the content search methods provided by the embodiments of the present application.

[0069] An embodiment of the present application can obtain multimedia data and the feedback amount corresponding to the multimedia segment. The multimedia data includes multiple multimedia segments, and each multimedia segment includes multiple multimedia elements. The feedback amount represents the degree of user feedback on the multimedia segment. At least one key segment is determined from the multimedia data according to the feedback amount corresponding to each multimedia segment. Feature extraction is performed on the multimedia elements in the key segment to obtain the segment features of the key segment. The overall features of the multimedia data are determined according to the segment features of the key segment. Content search is performed based on the overall features of the multimedia data to obtain the search results corresponding to the multimedia data.

[0070] In the present application, at least one key segment can be determined from the multimedia data through the feedback amount corresponding to the multimedia segment, and then target elements can be determined from the multimedia elements of the key segment. Among them, the user's feedback degree on the key segment is higher than that of other multimedia segments in the multimedia data. In this way, the multimedia elements in the key segment are conducive to representing the content in the multimedia data. Therefore, the overall features of the multimedia data can be determined according to the segment features of the key segment, so that the corresponding search results can be searched according to the overall features of the multimedia data. In this way, the multimedia elements involved in generating the overall features of the multimedia data are reduced. As a result, after the generation efficiency of the overall features of the multimedia data is improved, the content search efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative efforts.

[0072] Figure 1a is a schematic diagram of the scenario of the content search method provided by the embodiment of the present application;

[0073] Figure 1b is a schematic flowchart of the content search method provided by the embodiment of the present application;

[0074] Figure 2 is a schematic flowchart of the content search method provided by the embodiment of the present application applied to a video;

[0075] Figure 3It is a schematic structural diagram of the content search device provided by an embodiment of the present application;

[0076] Figure 4 It is a schematic structural diagram of the terminal provided by an embodiment of the present application. Specific embodiments

[0077] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0078] An embodiment of the present application provides a content search method, device, terminal, storage medium, and program product.

[0079] Among them, the content search device can be specifically integrated in an electronic device, and the electronic device can be a device such as a terminal or a server. Among them, the terminal can be a device such as a mobile phone, a tablet computer, a smart Bluetooth device, a laptop computer, a desktop computer, a smart home appliance, or a vehicle terminal; the server can be a single server or a server cluster or cloud server composed of multiple servers.

[0080] In some embodiments, the content search device can also be integrated in multiple electronic devices. For example, the content search device can be integrated in multiple servers, and multiple servers are used to implement the content search method of the present application.

[0081] In some embodiments, the server can also be implemented in the form of a terminal.

[0082] For example, referring to Figure 1a , the server can obtain multimedia data and the feedback amount corresponding to the multimedia segment from the terminal through the network. The multimedia data includes multiple multimedia segments, and the multimedia segment includes multiple multimedia elements. The feedback amount represents the feedback degree of the user to the multimedia segment; according to the feedback amount corresponding to each multimedia segment, at least one key segment is determined in the multimedia data; feature extraction is performed on the multimedia elements in the key segment to obtain the segment feature of the key segment; according to the segment feature of the key segment, the overall feature of the multimedia data is determined; content search is performed based on the overall feature of the multimedia data to obtain the search result corresponding to the multimedia data, and the server sends the search result to the terminal through the network.

[0083] Among them, after obtaining the user's permission or consent, the server can obtain multimedia data and the corresponding feedback amount of the multimedia segment from the terminal through the network. Through the feedback amount corresponding to the multimedia segment, the multimedia segment with a relatively high feedback degree from the user, that is, the key segment, can be determined from multiple multimedia segments. The key segment can represent the important content of the multimedia data, and according to the multimedia elements in the key segment, the segment features of the key segment are determined. According to the segment features of the key segment, the overall features of the multimedia data are determined, reducing the multimedia elements required to calculate the overall features of the multimedia data. In this way, the overall features of the multimedia data can be obtained quickly, so that the corresponding search results can be quickly searched according to the overall features of the multimedia data, and the search results are sent to the terminal through the network. Thus, the search efficiency of the content is improved.

[0084] The following will be described in detail respectively. It should be noted that the serial numbers of the following embodiments do not limit the preferred order of the embodiments.

[0085] In this embodiment, a content search method is provided. As Figure 1b shown, this content search method can be executed by an electronic device, and the specific process can be as follows:

[0086] 110. Obtain multimedia data and the corresponding feedback amount of the multimedia segment. The multimedia data includes multiple multimedia segments, and the multimedia segment includes multiple multimedia elements. The feedback amount represents the feedback degree of the user to the multimedia segment.

[0087] Among them, the multimedia data is the multimedia data waiting for content search. For example, the multimedia data can be a video, an audio, an electronic novel, and so on.

[0088] The multimedia segment is a data segment in the multimedia data, and the multimedia data is composed of multiple multimedia segments. For example, the multimedia segment can be a video segment in a video, and the video is composed of multiple video segments. The multimedia segment can also be an audio segment in an audio, and the audio is composed of multiple audio segments. The multimedia segment can also be a novel segment in an electronic novel, and the novel is composed of multiple novel segments, and so on.

[0089] For example, each video segment in a video can be composed of images with the same number of frames. The video segments in the video can also be obtained by dividing the image frames after scene conversion, and can also be obtained by dividing the black frames in the video. The black frame is a frame whose average color value of pixels is lower than a preset threshold, and can also be obtained by jointly dividing the image frames after scene conversion and the black frames, and so on.

[0090] For example, each audio segment in an audio can be composed of audio frames with the same number of frames. The audio segments in the audio can also be divided according to the audio intensity, and can also be divided according to the pitch, and so on.

[0091] For example, a novel segment can be composed of novel chapters published on the same date, or can be composed of novel chapters with the same chapter name, and so on.

[0092] The feedback amount characterizes the degree of user feedback on the multimedia segment. For example, when the multimedia data is a video, the feedback amount can be the frequency of bullet screens sent by users during the playback of the video segment, or the amount of bullet screens sent by users during the video playback, or the reward value sent by users during the playback of the video segment. It can also be composed of the amount of bullet screens and the reward value, or can be composed of the frequency of sending bullet screens and the reward frequency, and so on. When the multimedia data is an audio, the feedback amount can also be the frequency of bullet screens sent by users during the playback of the audio segment, or the amount of bullet screens sent by users during the playback of the audio segment, or the reward value sent by users during the playback of the audio segment, and so on. When the multimedia data is an electronic novel, the feedback amount can be the number of user comments received by the novel segment, or the frequency of user comments received by the novel segment, or the reward value received by the novel segment, and so on.

[0093] When the feedback amount is composed of the amount of bullet screens and the reward value, the size of the feedback amount can be calculated according to the proportion of the amount of bullet screens in the feedback amount and the ratio of the reward value in the feedback amount.

[0094] When the feedback amount is composed of the frequency of sending bullet screens and the reward frequency, the size of the feedback amount can be calculated according to the proportion of the frequency of sending bullet screens in the feedback amount and the ratio of the reward frequency in the feedback amount.

[0095] The multimedia element can be a unit that constitutes the multimedia segment. For example, when the multimedia data is a video, the multimedia element is the image frame and audio frame that constitute the video segment. When the multimedia data is an audio, the multimedia element is the audio frame that constitutes the audio segment. When the multimedia data is an electronic novel, the multimedia element is the chapter that constitutes the novel segment, and so on.

[0096] In some embodiments, considering that evenly dividing the multimedia data according to the time sequence may result in the lack of association among the multimedia elements in the multimedia segment, leading to the possible absence of key data in the multimedia segment. To make the multimedia elements in the multimedia segment associated with each other, when the multimedia element is composed of images, before obtaining the multimedia data, it further includes:

[0097] Determine the target element from the multimedia elements of the multimedia data according to the pixel color value of each pixel in the multimedia element;

[0098] Segment the multimedia data according to the target element to obtain multiple multimedia segments.

[0099] The pixel color value is the color value of a pixel. For example, the pixel color can be the RGB value of the pixel or the HSV value of the pixel.

[0100] The target element is a multimedia element in the multimedia data where the scene changes. For example, the target element can be the black frame used for the above video segment division, or any one of two adjacent frames where the color values of the pixels at the same position in the two adjacent frames change significantly, or the black frame and any one of the above two adjacent frames, and so on.

[0101] For example, segment the video according to the black frame to obtain multiple video segments. It can also be segmenting the video according to any one of the above two adjacent frames to obtain multiple video segments. It can also be segmenting the video according to the black frame and any one of the above two adjacent frames to obtain multiple video segments.

[0102] In some embodiments, considering that when the scene in the video changes, there are images with relatively fixed colors among several consecutive frames where the scene changes. Thus, it is possible to determine that the scene in the video has changed through the images with relatively fixed colors. According to the pixel color values of each pixel in the multimedia element, the target element is determined from the multimedia elements of the multimedia data, including:

[0103] Determine the color mean value of each frame of the multimedia element. The color mean value is the average of the pixel colors of all pixels in the multimedia element;

[0104] Take the multimedia element with a pixel color mean value greater than the preset pixel color as the target element.

[0105] Among them, the color mean value is the average of the pixel colors of all pixels in the multimedia element. For example, the pixel color mean value can be obtained by dividing the pixel colors of all pixels in the multimedia element by the number of pixels in the multimedia element. Specifically, it can be the RGB values of all pixels in the multimedia element divided by the number of pixels in the multimedia element, the HSV values of all pixels in the multimedia element divided by the number of pixels in the multimedia element, and so on.

[0106] The preset pixel color is used to measure the pixel color mean value of the pixels in the multimedia element.

[0107] The target element is a multimedia element with a pixel color mean value greater than the preset pixel color, and this target element corresponds to the above black frame.

[0108] For example, divide the RGB values of all pixels in the multimedia element by the number of pixels in the multimedia element to obtain the pixel color mean value of each pixel in the multimedia element, and then compare the pixel color mean value with the preset pixel color, so as to determine the target element related to the scene change in the video, that is, the black frame related to the scene change in the video can be determined.

[0109] In some embodiments, considering that when the scene of the video changes, the colors of the pixels at the same position in two adjacent frames of images will change significantly. Thus, when the pixel color of each pixel changes significantly, it can be determined that the scene of the video has changed. According to the pixel color values of each pixel in the multimedia element, the target element is determined from the multimedia elements of the multimedia data, including:

[0110] Determine the color change value of two adjacent multimedia elements according to the pixel color values of each pixel of the two adjacent multimedia elements;

[0111] When it is determined that the color change value is greater than the preset color change value, any one of the two adjacent multimedia elements is used as the target element.

[0112] Among them, the color change value is used to characterize the change in the pixel color values of two adjacent frames of images. The color change value can be the difference between the pixel color values of the pixels at the same position in two adjacent frames of images, or the difference between the pixel color values of all pixels in the previous image and the pixel color values of all pixels in the next image, etc.

[0113] The preset color change value is used to measure the color change of two adjacent multimedia elements. For example, the preset color change value is used to measure the change amount of the pixel color values of the pixels at the same position in two adjacent frames, or to measure the change amount between the pixel color values of all pixels in the previous multimedia element and the pixel color values of all pixels in the next multimedia element, where the previous multimedia element and the next multimedia element are two adjacent multimedia elements.

[0114] After the color change value is greater than the preset color change value, the multimedia element with the earlier time sequence in the two adjacent multimedia elements is used as the target element, or the multimedia element with the later time sequence in the adjacent multimedia elements is used as the target element.

[0115] Method for determining the color change value of two adjacent multimedia elements:

[0116] 1. Divide the pixel color values of all pixels in the multimedia element by the number of pixels in the multimedia element to obtain the average pixel color of each pixel in the multimedia element, and then subtract the average pixel color of the latter multimedia element in the adjacent multimedia elements from the average pixel color of the former multimedia element in the adjacent multimedia elements to obtain the color change value of the two adjacent multimedia elements.

[0117] 2. Add up the pixel color values of all pixels in the multimedia element to obtain the total pixel color value of all pixels in the multimedia element. Then subtract the total pixel color value of the latter multimedia element in adjacent multimedia elements from the total pixel color value of the former multimedia element in adjacent multimedia elements to obtain the color change value between two adjacent frames of multimedia elements.

[0118] 120. Determine at least one key segment in the multimedia data according to the feedback amount corresponding to each multimedia segment.

[0119] Among them, for the key segment compared with other multimedia segments, the feedback degree of its users is relatively high. For example, the key segment can be a multimedia segment in the multimedia data that receives a relatively high user response. Specifically, the key segment can be located by at least one of the received bullet screen amount, bullet screen frequency, and reward value.

[0120] For example, the feedback amount refers to the bullet screen amount. The multiple multimedia segments include multimedia segment A, multimedia segment B, multimedia segment C, and multimedia segment D. Among them, the bullet screen amount of multimedia segment A is 1000, the bullet screen amount of multimedia segment B is 800, the bullet screen amount of multimedia segment C is 200, and the bullet screen amount of multimedia segment D is 500. Then the key segment is multimedia segment A, or it can be multimedia segment A and multimedia segment B, or it can be multimedia segment A, multimedia segment B, and multimedia segment C, and so on.

[0121] In some embodiments, in order to facilitate locating the key segment from multiple multimedia segments, determining at least one key segment in the multimedia data according to the feedback amount corresponding to each multimedia segment includes:

[0122] Sort the multiple multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted by the feedback amount.

[0123] Determine at least one key segment in the multimedia data according to the sorted multimedia segments.

[0124] For example, the multimedia data is a video, and the multimedia segment is a video segment. When the user watches the video segment, they will send bullet screens, comments, or rewards, etc. When the feedback amount is the bullet screen amount corresponding to the video segment, sort the multiple video segments according to the bullet screen amount received by each video segment, so that the video segments sorted by the bullet screen amount can be obtained. The sorted video segments can be sorted from high to low or from low to high according to the feedback amount.

[0125] For example, if the multimedia data is audio and the multimedia segment is an audio segment, when the user listens to the audio segment, they may send comments, bullet screens, or rewards. When the feedback volume is the reward value corresponding to the audio segment, the multiple audio segments are sorted according to the reward values received by each audio segment, so that the audio segments sorted by the reward values can be obtained.

[0126] For example, if the multimedia data is an e-novel and the multimedia segment is a novel segment, when the user reads the e-novel, they may send comments and rewards. When the feedback volume consists of the comment volume and the reward value of the novel segment, the multiple novel segments are sorted according to the feedback volume corresponding to each novel segment, so that the novel segments sorted by the feedback volume can be obtained.

[0127] In some embodiments, considering reducing the amount of computation required to calculate the features of the multimedia data, at least one key segment is determined in the multimedia data according to the sorted multimedia segments, including:

[0128] When the number of multimedia segments in the multimedia data is greater than a preset quantity, the preset quantity of key segments are determined in the multimedia data according to the sorted multimedia segments.

[0129] For example, if the number of multimedia segments is 15 and the preset quantity is 10, and 15 is greater than 10, then 10 key segments are determined from the sorted multimedia segments.

[0130] In some embodiments, considering reducing the amount of computation required to calculate the features of the multimedia data, it further includes:

[0131] When the number of multimedia segments in the multimedia data is not greater than the preset quantity, each multimedia segment in the multimedia data is used as a key segment.

[0132] For example, if the number of multimedia segments is 10 and the preset quantity is 10, and the number of multimedia segments is equal to the preset quantity, then 10 key segments are determined from the 10 multimedia segments. For example, if the number of multimedia segments is 9 and the preset quantity is 10, and the number of multimedia segments is equal to the preset quantity, then 9 key segments are determined from the 9 multimedia segments.

[0133] In some embodiments, considering that there may be a situation where the feedback volume of the multimedia data is small, which may easily lead to an objective sorting of the multimedia segments. To avoid sorting the multimedia segments only according to the feedback volume, the multimedia segments are sorted according to the content of the multimedia segments, as described below. The multimedia element consists of text data and images. The multiple multimedia segments are sorted according to the feedback volume corresponding to each multimedia segment to obtain the multimedia segments sorted by the feedback volume, including:

[0134] When the sum of the feedback amounts corresponding to all multimedia segments in the multimedia data is not less than a preset feedback amount, sort the multiple multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted by the feedback amount;

[0135] When it is determined that all the feedback amounts in the multimedia data are less than the preset feedback amount, determine the text amount ratio of the multimedia segment. The text amount ratio of the multimedia segment is the ratio of the text amount of all the text data in the multimedia data to the text amount of all the text data in the multimedia segment;

[0136] Determine the color ratio. The color ratio is the ratio of the image color change amount of all the images in the multimedia data to the image color change amount of all the images in the multimedia segment. The image color change amount represents the degree of color change of the image;

[0137] Sort the multiple multimedia segments according to the text amount ratio and the color ratio to obtain the sorted multimedia segments.

[0138] Among them, the preset feedback amount is used to measure all the feedback amounts in the multimedia data.

[0139] The text data is the text data to be used when sorting the multimedia segments. For example, the text data can be the number of words, paragraph data, etc. Specifically, the text data can be the line data in the video, the lyric data in the audio, the text data in the e-novel, etc.

[0140] The text amount is the statement data in the text data. For example, the text amount can be the number of lines in the video segment, the number of lyrics in the audio segment, the number of paragraphs in the novel segment, etc.

[0141] The text amount ratio is the ratio of the text amount of the multimedia data to the text amount of the multimedia segment. For example, the text amount ratio can be the ratio of the number of lines in the video to the number of lines in the video segment, the ratio of the number of lyrics in the audio to the number of lyrics in the audio segment, the ratio of the number of paragraphs in the e-novel to the number of paragraphs in the novel segment, etc.

[0142] For example, if the text amount of each multimedia segment in the multimedia data is (w1, w2,..., w n ), then the text amount ratio If the text amount of the multimedia data Then make all

[0143] The change amount of the image color is the change amount of the pixel color values of all pixels in two adjacent frames of multimedia elements. For example, if the pixel color value of the same pixel in the previous multimedia element is A and the pixel color value in the next multimedia element is B, then the change amount of the pixel color value of this pixel is A - B, and the sum of the pixel color values of all pixels in two adjacent frames of multimedia elements is equal to the sum of the change amounts of the pixel color values of each pixel.

[0144] For example, the text volume ratio of the multimedia segment is The color ratio is Then the proportion occupied by the multimedia segment relative to the multimedia data can be According to the Sort multiple multimedia segments to obtain the multimedia segments sorted according to the color ratio.

[0145] In some embodiments, considering that the images in each multimedia segment are different, in order to evaluate the images in each multimedia segment using the same benchmark so that the multimedia segments can be sorted through this evaluation and the color ratio can be determined, it includes:

[0146] Determine the pixel color difference between each pixel of two adjacent frames of multimedia elements in the multimedia segment;

[0147] Sum the pixel color differences between each pixel of two adjacent frames of multimedia elements to obtain the color difference between two adjacent frames of multimedia elements;

[0148] Sum the color differences between all two adjacent frames of multimedia elements in the multimedia segment to obtain the color difference corresponding to the multimedia segment;

[0149] Sum the color differences corresponding to all multimedia segments in the multimedia data to obtain the color difference corresponding to the multimedia data;

[0150] Determine the color ratio according to the color difference corresponding to the multimedia data and the color difference corresponding to the multimedia segment.

[0151] Among them, the pixel color difference is the pixel color difference of the same-position pixels of two adjacent frames of multimedia elements. For example, if the pixel color at (x, y) of the previous multimedia element of two adjacent frames of multimedia elements is A and the pixel color at (x, y) of the next multimedia element of two adjacent frames of multimedia elements is B, then the pixel color difference at (x, y) of two adjacent frames of multimedia elements is A - B.

[0152] The color difference between two adjacent frames of multimedia elements includes the pixel color differences between all the same-position pixels in two adjacent frames of multimedia elements.

[0153] For example, in the multimedia data, the multimedia segment is (v1, v2, …, v n ), for a certain multimedia segment v i , assuming that v i is composed of t i frames, then v i = (I1, I2, …, I ti ), where is a multimedia element (element) with height H, width W, and 3 channels.

[0154] The color difference d k between two adjacent frames of multimedia elements:

[0155]

[0156] The color difference corresponding to the multimedia segment includes the color differences between all adjacent two frames of multimedia elements in the multimedia segment.

[0157] The color difference a i corresponding to the multimedia segment:

[0158] Then the color difference a i corresponding to the multimedia segment in the multimedia data is (a1, a2, …, a n ).

[0159] The color difference corresponding to the multimedia data includes the color differences corresponding to all multimedia segments in the multimedia data.

[0160] The color difference corresponding to the multimedia data:

[0161] Color ratio If the color difference corresponding to the multimedia data then make all

[0162] The image color change amount of all images in the multimedia data can be the sum of the image color change amounts of all adjacent two frames of images in the multimedia data.

[0163] The image color change amount of all images in the multimedia segment can be the sum of the image color change amounts of all adjacent two frames of images in the multimedia segment.

[0164] The color ratio is the ratio of the image color change amount of all images in the multimedia data to the image color change amount of all images in the multimedia segment.

[0165] In some embodiments, considering that there is a small amount of feedback in multimedia data, which easily leads to an insufficiently objective sorting of multimedia segments. To avoid sorting multimedia segments only based on the amount of feedback, the multimedia segments will be sorted according to the content of the multimedia segments, as described below. The multimedia elements are composed of images. According to the amount of feedback corresponding to each multimedia segment, multiple multimedia segments are sorted to obtain the multimedia segments sorted according to the amount of feedback, including:

[0166] When the sum of the amounts of feedback corresponding to all multimedia segments in the multimedia data is not less than the preset amount of feedback, multiple multimedia segments are sorted according to the amount of feedback corresponding to each multimedia segment to obtain the multimedia segments sorted according to the amount of feedback;

[0167] When it is determined that all the amounts of feedback in the multimedia data are less than the preset amount of feedback, the color ratio is determined. The color ratio is the ratio of the image color change amount of all multimedia elements in the multimedia data to the image color change amount of all multimedia elements in the multimedia segment. The image color change amount represents the degree of color change of the image;

[0168] Multiple multimedia segments are sorted according to the color ratio to obtain the multimedia segments arranged according to the color ratio.

[0169] In some embodiments, considering that there is a small amount of feedback in multimedia data, which easily leads to an insufficiently objective sorting of multimedia segments. To avoid sorting multimedia segments only based on the amount of feedback, the multimedia segments will be sorted according to the content of the multimedia segments, as described below. The multimedia elements are composed of text data. According to the amount of feedback corresponding to each multimedia segment, multiple multimedia segments are sorted to obtain the multimedia segments sorted according to the amount of feedback, including:

[0170] When the sum of the amounts of feedback corresponding to all multimedia segments in the multimedia data is not less than the preset amount of feedback, multiple multimedia segments are sorted according to the amount of feedback corresponding to each multimedia segment to obtain the multimedia segments sorted according to the amount of feedback;

[0171] When all the amounts of feedback in the multimedia data are less than the preset amount of feedback, the text amount ratio of the multimedia segment is determined. The text amount ratio of the multimedia segment is the ratio of the text amount of all text data in the multimedia data to the text amount of all text data in the multimedia segment;

[0172] Multiple multimedia segments are sorted according to the text amount ratio to obtain the multimedia segments sorted according to the text amount.

[0173] In some embodiments, considering that there is usually a relatively large amount of feedback at the beginning of multimedia data, and it is difficult for the multimedia segments corresponding to this amount of feedback to characterize the content of the multimedia data. To reduce the impact of the amount of feedback on the determination of key segments, when the sum of the amounts of feedback corresponding to all multimedia segments in the multimedia data is not less than a preset amount of feedback, the multiple multimedia segments are sorted according to the amount of feedback corresponding to each multimedia segment, and the multimedia segments sorted according to the amount of feedback are obtained, including:

[0174] Determine a partial amount of feedback among all the amounts of feedback in the multimedia data, where the partial amount of feedback is the amount of feedback corresponding to partial multimedia data;

[0175] When it is determined that the partial amount of feedback is not less than the preset amount of feedback, the multiple multimedia segments are sorted according to the amount of feedback corresponding to each multimedia segment, and the multimedia segments sorted according to the amount of feedback are obtained;

[0176] When it is determined that all the amounts of feedback in the multimedia data are less than the preset amount of feedback, determine the text volume ratio of the multimedia segment, where the text volume ratio of the multimedia segment is the ratio of the text volume of all the text data in the multimedia data to the text volume of all the text data in the multimedia segment, including:

[0177] When the partial amount of feedback is less than the preset amount of feedback, determine the text volume ratio of the multimedia segment, where the text volume ratio of the multimedia segment is the ratio of the text volume of all the text data in the multimedia data to the text volume of all the text data in the multimedia segment.

[0178] Among them, the partial multimedia data is part of the multimedia data. For example, when the multimedia data is a video or audio, the partial multimedia data can be the multimedia data corresponding to a preset time period later, specifically, the multimedia data corresponding to 5 seconds later. Or when the multimedia data is an e-novel, the partial multimedia data can be the multimedia data corresponding to a preset chapter later.

[0179] The partial amount of feedback is the degree of feedback of the user on the partial multimedia data. For example, the partial amount of feedback can be the number of bullet screens, bullet screen frequency, rewards, etc. sent by the user when watching or listening to the partial multimedia data.

[0180] The preset amount of feedback is used to measure the partial amount of feedback among all the amounts of feedback in the multimedia data.

[0181] In some embodiments, considering that the amount of feedback corresponding to each multimedia segment is not the same, in order to evaluate the amount of feedback corresponding to each multimedia segment using the same benchmark so that the multimedia segments can be sorted through this evaluation, the multiple multimedia segments are sorted according to the amount of feedback corresponding to each multimedia segment, and the multimedia segments sorted according to the amount of feedback are obtained, including:

[0182] Determine the weight corresponding to the multimedia segment according to the feedback volume ratio, where the feedback volume ratio is the ratio of the length of the multimedia segment to the feedback volume corresponding to the multimedia segment;

[0183] Sort the multiple multimedia segments according to the weight corresponding to each multimedia segment to obtain the multimedia segments sorted by weight.

[0184] Wherein, the length of the multimedia segment is the content length of the multimedia segment. For example, when the multimedia data is video or audio, the length of the multimedia segment can be the time length of the video segment or the audio segment. When the multimedia data is an electronic novel, the length of the multimedia segment can be the chapter, the number of words in the novel segment, and so on.

[0185] The feedback volume ratio is the ratio of the length of the multimedia segment to the feedback volume corresponding to the multimedia segment. For example, the multimedia data includes N multimedia segments, and the feedback volumes corresponding to the multimedia segments in the multimedia data are (C1, C2, …, C n ), and the lengths of the multimedia segments are (t1, t2, …, t n ), then the feedback volume ratio is S i ,

[0186] The weight corresponding to the multimedia segment is the proportion of the multimedia segment in the multimedia data. For example, the weight corresponding to the multimedia can be the feedback volume ratio, and so on.

[0187] 130. Extract features of the multimedia elements in the key segment to obtain the segment features of the key segment.

[0188] Wherein, the multimedia elements in the key segment are at least one multimedia element in the key segment, and the at least one multimedia element can be at any position in the key segment. For example, when the multimedia data is video, the target element is at least one frame of image in the key segment, and the at least one frame of image can be in the middle position of the key segment. For example, when the multimedia data is audio, the target element is at least one audio frame in the key segment, and the at least one audio frame can be in the middle position of the key segment. For example, when the multimedia data is an electronic novel and the key segment consists of novel chapters published on the same date, the target element is at least one chapter of the novel in the key segment.

[0189] The segment features of the key segment are the feature data obtained after feature extraction of the multimedia elements in the key segment. For example, the segment features of the key segment can be the feature vector corresponding to the image, or the feature vector corresponding to the audio frame, or the feature vector corresponding to the novel chapter, and so on.

[0190] In some embodiments, considering reducing the amount of computation required to calculate the features of multimedia data, when the number of multimedia segments in the multimedia data is not greater than a preset quantity, each multimedia segment in the multimedia data is regarded as a key segment, and feature extraction is performed on the multimedia elements in the key segment to obtain the segment features of the key segment, including:

[0191] Divide the preset quantity by the number of multimedia segments in the multimedia data to obtain the target number;

[0192] Determine the target number of target elements from the key segments;

[0193] Perform feature extraction on the target elements in the key segment to obtain the segment features of the key segment.

[0194] Wherein, the target number is the number of target elements in the multimedia segment. For example, if the preset quantity is a and the number of multimedia segments is b, then the target number = a / b.

[0195] For example, when the preset quantity is 10 and the number of multimedia segments is 5, the target number is equal to 2. Therefore, 2 target elements need to be determined from each key segment.

[0196] In some embodiments, to avoid the influence of a decimal target number on determining target elements from multimedia segments, according to the sorted multimedia segments, determining the target number of target elements from the key segments includes:

[0197] When the target number is a decimal, perform rounding on the target number to obtain the rounded target number;

[0198] Multiply the rounded target number by the number of multimedia segments to obtain the rounded total number;

[0199] Determine the number difference, where the number difference is the difference between the rounded total number and the preset quantity;

[0200] Among multiple multimedia segments, determine N - 1 first key segments and 1 second key segment, where N is the number of multimedia segments;

[0201] Determine the rounded target number of target elements from the first key segments;

[0202] Determine the new target number of target elements from the second key segment, where the new target number is the sum of the number difference and the rounded target number.

[0203] Wherein, the rounded target number is the number corresponding to the decimal target number after rounding. For example, when the preset quantity is 10, the number of multimedia segments is 4, and the target number is 2.5, it can be rounded down, then the rounded target number can be 2, or rounded up, then the rounded target number can be 3.

[0204] The total number of rounding is the sum of the rounding target numbers of all key segments. For example, when the rounding target number is 2 and the number of multimedia segments is 4, the total number of rounding is 8. When the rounding target number is 3 and the number of key segments is 4, the total number of rounding is 12.

[0205] The difference in the number is the difference between the preset data quantity and the total number of rounding. For example, when the total number of rounding is 8 and the preset data quantity is 10, the difference in the number is 2. When the total number of rounding is 12 and the preset data quantity is 10, the difference in the number is -2.

[0206] The new target number is the sum of the difference in the number and the rounding target number. For example, when the difference in the number is 2 and the rounding target number is 2, the new target number is 4. When the difference in the number is -2 and the rounding target number is 3, the new target number is 1.

[0207] For example, when the new target number is 4, 4 target elements need to be determined from the second key segment. When the new target number is 1, 1 target element needs to be determined from the second key segment.

[0208] In some embodiments, considering that the arrangement order is likely to affect the determination of target elements, among the sorted multimedia segments, N - 1 first key segments and 1 second key segment are determined, including:

[0209] When multiple multimedia segments are sorted in descending order of the feedback amount, the first key segments are the first N - 1 segments in the sorted multimedia segments;

[0210] When multiple multimedia segments are sorted in ascending order of the feedback amount, the first key segments are the last N - 1 segments in the sorted multimedia segments.

[0211] For example, when the sorted multimedia segments are sorted in descending order of the feedback amount and there are 4 sorted multimedia segments, the first key segments are the first 3 segments in the sorted multimedia segments, and the second key segment is the 4th segment in the sorted multimedia segments. When the sorted multimedia segments are sorted in ascending order of the feedback amount, the first key segments are the last 3 segments in the sorted multimedia segments, and the second key segment is the 1st segment in the sorted multimedia segments.

[0212] 140. Determine the overall feature of the multimedia data according to the segment features of the key segments.

[0213] Among them, the overall feature of the multimedia data can represent the content in the multimedia data. For example, the overall feature of the multimedia data can be video Embedding, audio Embedding, e - novel Embedding, and so on.

[0214] There are at least two methods for determining the overall features of multimedia data as follows:

[0215] 1. Based on Singular Value Decomposition (SVD), determine the overall features of multimedia data according to the segment features of key segments.

[0216] 2. Based on a neural network, determine the overall features of multimedia data according to the segment features of key segments. The specific method is as follows:

[0217] Fuse the segment features of all key segments to obtain the features of multimedia data.

[0218] Method for feature fusion:

[0219] Supervise the learning of the features of multimedia data based on the content of multimedia data, so that the overall features of multimedia data can be extracted after the segment features of key segments pass through the middle layer (usually a fully connected layer) of the network.

[0220] 150. Conduct content search based on the overall features of multimedia data to obtain the search results corresponding to the multimedia data.

[0221] Among them, the search results are multimedia data similar to the content of the multimedia data. For example, if the multimedia data is an incomplete video, the search content can be the complete video corresponding to the multimedia data, or other videos similar to the video content, etc.

[0222] Method for content search:

[0223] Determine the search content similar to the overall features of multimedia data according to the similarity.

[0224] In some embodiments, extract video Embedding based on the feedback degree of the user on multimedia segments, and extract important frame pictures in the video by using the feedback amount of the user on multimedia segments. Specifically, on the one hand, first divide the complete video into several video segments, then calculate the number of bullet screens in each video segment, sort the video segments according to the number of video bullet screens, and finally extract frames from the key segments with a large number of video bullet screens to obtain the segment features of the key segments, and extract video Embedding features according to the segment features of the key segments; on the other hand, if the number of bullet screens in the video does not reach the threshold, sort the video segments according to the text volume ratio and color ratio, and finally determine at least one key segment from the sorted video segments, extract frames from the key segments to obtain the segment features of the key segments, and extract video Embedding features according to the segment features of the key segments.

[0225] As can be seen from the above, on the one hand, the calculation process of the present invention is simple and has good universality, without any restrictions on the final video model, and it can be easily modified and combined with existing video models; on the other hand, the present invention uses the feedback volume of users to extract frames from video segments. Specifically, based on the number of bullet screens in the video segment, typical scenes and key pictures that users are concerned about in the video are determined, so that the extracted video frames are more representative, and thus the effect of subsequent video Embedding can be improved.

[0226] As can be seen from the above, the embodiments of the present application can obtain multimedia data and the feedback volume corresponding to the multimedia segment. The multimedia data includes multiple multimedia segments, and each multimedia segment includes multiple multimedia elements. The feedback volume represents the degree of feedback of users on the multimedia segment. At least one key segment is determined from the multimedia data according to the feedback volume corresponding to each multimedia segment; feature extraction is performed on the multimedia elements in the key segment to obtain the segment features of the key segment; the overall features of the multimedia data are determined according to the segment features of the key segment; and content search is performed based on the overall features of the multimedia data to obtain the search results corresponding to the multimedia data.

[0227] In the embodiments of the present invention, through the feedback volume corresponding to the multimedia segment, the multimedia segments with a higher degree of feedback from users, that is, the key segments, can be determined from multiple multimedia segments. The key segments can represent the important content of the multimedia data, and the segment features of the key segments are determined according to the multimedia elements in the key segments. According to the segment features of the key segments, the overall features of the multimedia data are determined, reducing the multimedia elements required to calculate the overall features of the multimedia data. In this way, the overall features of the multimedia data can be quickly obtained, so that the corresponding search results can be quickly searched according to the overall features of the multimedia data. Thus, the search efficiency of the content is improved.

[0228] The method described in the above embodiments will be further described in detail below.

[0229] In this embodiment, taking the multimedia data as a video as an example, the method of the embodiments of the present application will be described in detail.

[0230] As Figure 2 shown, a specific process of a content search method can be executed by an electronic device, and the specific process can be as follows:

[0231] 210. Obtain multimedia data.

[0232] 220. Determine a target element from multiple frames of multimedia elements according to the pixel color value of each pixel in the multimedia element.

[0233] In some embodiments, determining a target element from multiple frames of multimedia elements according to the pixel color values of each pixel in the multimedia elements includes:

[0234] Determine the color mean value of each frame of multimedia element, where the color mean value is the average of the pixel colors of all pixels in the multimedia element;

[0235] Take the multimedia element with a pixel color mean value greater than the preset pixel color as the target element.

[0236] In some embodiments, determining a target element from multiple frames of multimedia elements according to the pixel color values of each pixel in the multimedia elements includes:

[0237] Determine the color change value of two adjacent frames of multimedia elements according to the pixel color values of each pixel in the two adjacent frames of multimedia elements;

[0238] Determine that the color change value is greater than the preset color change value;

[0239] Take any one of the two adjacent frames of multimedia elements as the target element.

[0240] In some embodiments, determining a target element from multiple frames of multimedia elements according to the pixel color values of each pixel in the multimedia elements includes:

[0241] Determine the pixel color mean value of each pixel in each frame of multimedia element, where the pixel color mean value is the average of the pixel colors of all pixels in the multimedia element;

[0242] Determine the color change value of two adjacent frames of multimedia elements according to the pixel color values of each pixel in the two adjacent frames of multimedia elements;

[0243] Determine that the color change value is greater than the preset color change value;

[0244] Take the multimedia element with a pixel color mean value greater than the preset pixel color, and any one of the two adjacent frames of multimedia elements as the target element.

[0245] In some embodiments, the pixel color value of a pixel generally uses an RGB value. Considering that it is convenient for color comparison, before determining the color change value of two adjacent frames of multimedia elements according to the pixel color values of each pixel in the two adjacent frames of multimedia elements, it includes:

[0246] Perform a color space conversion on the RGB value to obtain an HSV value.

[0247] 230. Segment the multimedia data according to the target element to obtain multiple multimedia segments.

[0248] 240. When all the feedback amounts in the multimedia data are not less than the preset feedback amount, sorting processing is performed on multiple multimedia segments according to the feedback amount corresponding to each multimedia segment, and the multimedia segments sorted by the feedback amount are obtained.

[0249] In some embodiments, when all the feedback amounts in the multimedia data are not less than the preset feedback amount, sorting processing is performed on multiple multimedia segments according to the feedback amount corresponding to each multimedia segment, and the multimedia segments sorted by the feedback amount are obtained, including:

[0250] Determine that among all the feedback amounts in the multimedia data, some feedback amounts are the feedback amounts corresponding to some of the multimedia data;

[0251] When it is determined that some feedback amounts are not less than the preset feedback amount, sorting processing is performed on multiple multimedia segments according to the feedback amount corresponding to each multimedia segment, and the multimedia segments sorted by the feedback amount are obtained.

[0252] In some embodiments, sorting processing is performed on multiple multimedia segments according to the feedback amount corresponding to each multimedia segment, and the multimedia segments sorted by the feedback amount are obtained, including:

[0253] Determine the weight corresponding to the multimedia segment according to the feedback amount ratio, where the feedback amount ratio is the ratio of the length of the multimedia segment to the feedback amount corresponding to the multimedia segment;

[0254] Sorting processing is performed on multiple multimedia segments according to the weight corresponding to each multimedia segment, and the multimedia segments sorted by the weight are obtained.

[0255] 250. When all the feedback amounts in the multimedia data are less than the preset feedback amount, sorting processing is performed on multiple multimedia segments according to the text amount ratio and the color ratio, and the multimedia segments sorted by the color ratio are obtained.

[0256] In some embodiments, when all the feedback amounts in the multimedia data are less than the preset feedback amount, sorting processing is performed on multiple multimedia segments according to the text amount ratio and the color ratio, and the multimedia segments sorted by the color ratio are obtained, including:

[0257] When some feedback amounts are less than the preset feedback amount, sorting processing is performed on multiple multimedia segments according to the text amount ratio and the color ratio, and the multimedia segments sorted by the color ratio are obtained.

[0258] In some embodiments, the method for determining the text amount ratio is:

[0259] Determine the text amount ratio according to the ratio of the text amount of the multimedia data to the text amount of the multimedia segment.

[0260] In some embodiments, determining a color ratio includes:

[0261] Determining the difference in pixel colors between each pixel of adjacent two frames of multimedia elements in a multimedia segment;

[0262] Summing up the differences in pixel colors between each pixel of adjacent two frames of multimedia elements to obtain the color difference between adjacent two frames of multimedia elements;

[0263] Summing up the color differences between all adjacent two frames of multimedia elements in the multimedia segment to obtain the color difference corresponding to the multimedia segment;

[0264] Summing up the color differences corresponding to all multimedia segments in the multimedia data to obtain the color difference corresponding to the multimedia data;

[0265] Determining the color ratio according to the color difference corresponding to the multimedia data and the color difference corresponding to the multimedia segment.

[0266] 260. Determining at least one key segment in the multimedia data according to the sorted multimedia segments.

[0267] In some embodiments, determining at least one key segment in the multimedia data according to the sorted multimedia segments includes:

[0268] When the number of multimedia segments in the multimedia data is not less than a preset quantity, determining the preset quantity of key segments in the multimedia data according to the sorted multimedia segments;

[0269] When the number of multimedia segments in the multimedia data is less than the preset quantity, taking each multimedia segment in the multimedia data as a key segment.

[0270] In some embodiments, extracting features of multimedia elements in the key segment to obtain the segment features of the key segment includes:

[0271] Dividing the preset quantity by the number of multimedia segments in the multimedia data to obtain a target number;

[0272] Determining the target number of target elements from the key segments according to the sorted multimedia segments;

[0273] Extracting features of the target elements in the key segment to obtain the segment features of the key segment.

[0274] In some embodiments, determining the target number of target elements from the key segments according to the sorted multimedia segments includes:

[0275] When the target number is a decimal, performing rounding on the target number to obtain a rounded target number;

[0276] Multiply the rounded target number by the number of multimedia segments to obtain the total rounded number.

[0277] Determine the number difference, which is the difference between the total rounded number and the preset quantity.

[0278] Among the sorted multimedia segments, determine N - 1 first key segments and 1 second key segment, where N is the number of multimedia segments.

[0279] Determine the target number of target elements from the first key segments.

[0280] Determine the target number of target elements from the second key segment, where the target number is the sum of the number difference and the rounded target number.

[0281] 270. Extract features from the multimedia elements in the key segments to obtain the segment features of the key segments.

[0282] 280. Determine the overall features of the multimedia data according to the segment features of the key segments.

[0283] 290. Perform content search based on the overall features of the multimedia data to obtain the search results corresponding to the multimedia data.

[0284] As can be seen from the above, on the one hand, the calculation process of the present invention is simple and has good universality, which is convenient for reforming and combining the process of obtaining target elements with the process of obtaining features of multimedia data. On the other hand, the present invention uses the feedback degree of users to extract frames. Specifically, based on the feedback amount, the important content concerned by users in the multimedia data is judged, so that the extracted target elements are more representative. Therefore, the generation effect of the features of the multimedia data can be improved.

[0285] To better implement the above method, the embodiment of the present application also provides a content search device. For example, as Figure 3 shown, the content search device may include an acquisition unit 310, a first determination unit 320, a feature extraction unit 330, a second determination unit 340, and a content search unit 350, as follows:

[0286] (1). Acquisition unit 310.

[0287] The acquisition unit 310 can be used to acquire multimedia data and the feedback amount corresponding to the multimedia segments. The multimedia data includes multiple multimedia segments, and the multimedia segments include multiple multimedia elements. The feedback amount represents the feedback degree of the user to the multimedia segments.

[0288] In some embodiments, the multimedia elements are composed of images. Before acquiring the multimedia data, it further includes:

[0289] Determine a target element from the multimedia elements of the multimedia data according to the pixel color value of each pixel in the multimedia element;

[0290] Segment the multimedia data according to the target element to obtain a plurality of multimedia segments.

[0291] In some embodiments, determining a target element from the multimedia elements of the multimedia data according to the pixel color value of each pixel in the multimedia element includes:

[0292] Determine the color average value of each frame of multimedia element, where the color average value is the average value of the pixel colors of all pixels in the multimedia element;

[0293] Use the multimedia element with a pixel color average value greater than the preset pixel color as the target element.

[0294] In some embodiments, determining a target element from the multimedia elements of the multimedia data according to the pixel color value of each pixel in the multimedia element includes:

[0295] Determine the color change value of two adjacent frames of multimedia elements according to the pixel color value of each pixel of the two adjacent frames of multimedia elements;

[0296] When it is determined that the color change value is greater than the preset color change value, use any one of the two adjacent frames of multimedia elements as the target element.

[0297] (2). The first determination unit 320.

[0298] The first determination unit 320 can be used to determine at least one key segment in the multimedia data according to the feedback amount corresponding to each multimedia segment.

[0299] In some embodiments, determining at least one key segment in the multimedia data according to the feedback amount corresponding to each multimedia segment includes:

[0300] Perform a sorting process on the plurality of multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted according to the feedback amount;

[0301] Determine at least one key segment in the multimedia data according to the sorted multimedia segments.

[0302] In some embodiments, determining at least one key segment in the multimedia data according to the sorted multimedia segments includes:

[0303] When the number of multimedia segments in the multimedia data is greater than the preset quantity, then determine the preset quantity of key segments in the multimedia data according to the sorted multimedia segments.

[0304] In some embodiments, it further includes:

[0305] When the number of multimedia segments in the multimedia data is not greater than a preset quantity, each multimedia segment in the multimedia data is used as a key segment.

[0306] In some embodiments, the multimedia element is composed of text data and images. According to the feedback amount corresponding to each multimedia segment, multiple multimedia segments are sorted to obtain the multimedia segments sorted according to the feedback amount, including:

[0307] When the sum of the feedback amounts corresponding to all multimedia segments in the multimedia data is not less than a preset feedback amount, multiple multimedia segments are sorted according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted according to the feedback amount;

[0308] When it is determined that all the feedback amounts in the multimedia data are less than the preset feedback amount, the text volume ratio of the multimedia segment is determined. The text volume ratio of the multimedia segment is the ratio of the text volume of all the text data in the multimedia data to the text volume of all the text data in the multimedia segment;

[0309] The color ratio is determined. The color ratio is the ratio of the image color change amount of all the images in the multimedia data to the image color change amount of all the images in the multimedia segment. The image color change amount represents the degree of color change of the image;

[0310] Multiple multimedia segments are sorted according to the text volume ratio and the color ratio to obtain the sorted multimedia segments.

[0311] In some embodiments, the multimedia element is composed of images. According to the feedback amount corresponding to each multimedia segment, multiple multimedia segments are sorted to obtain the multimedia segments sorted according to the feedback amount, including:

[0312] When the sum of the feedback amounts corresponding to all multimedia segments in the multimedia data is not less than a preset feedback amount, multiple multimedia segments are sorted according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted according to the feedback amount;

[0313] When it is determined that all the feedback amounts in the multimedia data are less than the preset feedback amount, the color ratio is determined. The color ratio is the ratio of the image color change amount of all the multimedia elements in the multimedia data to the image color change amount of all the multimedia elements in the multimedia segment. The image color change amount represents the degree of color change of the image;

[0314] Multiple multimedia segments are sorted according to the color ratio to obtain the multimedia segments arranged according to the color ratio.

[0315] In some embodiments, the multimedia elements are composed of text data. According to the amount of feedback corresponding to each multimedia segment, multiple multimedia segments are sorted to obtain the multimedia segments sorted by the amount of feedback, including:

[0316] When the sum of the amounts of feedback corresponding to all multimedia segments in the multimedia data is not less than the preset amount of feedback, multiple multimedia segments are sorted according to the amount of feedback corresponding to each multimedia segment to obtain the multimedia segments sorted by the amount of feedback;

[0317] When all the amounts of feedback in the multimedia data are less than the preset amount of feedback, determine the text volume ratio of the multimedia segment. The text volume ratio of the multimedia segment is the ratio of the text volume of all the text data in the multimedia data to the text volume of all the text data in the multimedia segment;

[0318] Multiple multimedia segments are sorted according to the text volume ratio to obtain the multimedia segments sorted by the text volume ratio.

[0319] In some embodiments, determining the color ratio includes:

[0320] Determine the pixel color difference between each pixel of two adjacent frames of multimedia elements in the multimedia segment;

[0321] Sum up the pixel color differences between each pixel of two adjacent frames of multimedia elements to obtain the color difference between two adjacent frames of multimedia elements;

[0322] Sum up the color differences between all two adjacent frames of multimedia elements in the multimedia segment to obtain the color difference corresponding to the multimedia segment;

[0323] Sum up the color differences corresponding to all multimedia segments in the multimedia data to obtain the color difference corresponding to the multimedia data;

[0324] Determine the color ratio according to the color difference corresponding to the multimedia data and the color difference corresponding to the multimedia segment.

[0325] In some embodiments, extracting features from the multimedia elements in the key segment to obtain the segment features of the key segment includes:

[0326] Divide the preset quantity by the number of multimedia segments in the multimedia data to obtain the target quantity;

[0327] Determine the target number of target elements from the key segment;

[0328] Extract features from the target elements in the key segment to obtain the segment features of the key segment.

[0329] In some embodiments, determining a target number of target elements from key segments includes:

[0330] When the target number is a decimal, perform rounding on the target number to obtain a rounded target number;

[0331] Multiply the rounded target number by the number of multimedia segments to obtain a rounded total number;

[0332] Determine a number difference, where the number difference is the difference between the rounded total number and a preset quantity;

[0333] Among multiple multimedia segments, determine N - 1 first key segments and 1 second key segment, where N is the number of the multimedia segments;

[0334] Determine a target number of target elements from the first key segments;

[0335] Determine a number of new target elements from the second key segment, where the number of new target elements is the sum of the number difference and the rounded target number.

[0336] In some embodiments, among multiple multimedia segments, determining N - 1 first key segments and 1 second key segment includes:

[0337] When the sorted multimedia segments are sorted in descending order of feedback volume, the first key segments are the first N - 1 segments of the sorted multimedia segments;

[0338] When the sorted multimedia segments are sorted in ascending order of feedback volume, the first key segments are the last N - 1 segments of the sorted multimedia segments.

[0339] (III). Feature extraction unit 330.

[0340] The feature extraction unit 330 can be used to extract features of multimedia elements in the key segments to obtain segment features of the key segments.

[0341] (IV). Second determination unit 340.

[0342] The second determination unit 340 can be used to determine the overall features of the multimedia data based on the segment features of the key segments.

[0343] (V). Content search unit 350.

[0344] The content search unit 350 can be used to perform content search based on the overall features of the multimedia data to obtain a search result corresponding to the multimedia data.

[0345] In specific implementation, each of the above units can be implemented as an independent entity, or can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of each of the above units, reference can be made to the method embodiments described above, which will not be elaborated here.

[0346] As can be seen from the above, the content search device in this embodiment obtains multimedia data and the feedback amount corresponding to the multimedia segment by the obtaining unit. The multimedia data includes multiple multimedia segments, and each multimedia segment includes multiple multimedia elements. The feedback amount represents the feedback degree of the user to the multimedia segment. The first determination unit determines at least one key segment in the multimedia data according to the feedback amount corresponding to each multimedia segment. The feature extraction unit extracts features of the multimedia elements in the key segment to obtain the segment features of the key segment. The second determination unit determines the overall features of the multimedia data according to the segment features of the key segment. The content search unit performs content search based on the overall features of the multimedia data to obtain the search results corresponding to the multimedia data.

[0347] Therefore, the embodiments of the present application can reduce the multimedia elements involved in generating the overall features of the multimedia data. Therefore, after the generation efficiency of the overall features of the multimedia data is improved, the content search efficiency can be improved.

[0348] The embodiments of the present application further provide an electronic device, which can be a device such as a terminal or a server.

[0349] In this embodiment, the electronic device in this embodiment will be described in detail by taking the example that the electronic device is a terminal. For example, as Figure 4 shown, it shows a schematic structural diagram of the terminal involved in the embodiments of the present application. Specifically:

[0350] The terminal may include a processor 410 with one or more processing cores, a memory 420 with one or more computer-readable storage media, a power supply 430, an input module 440, and a communication module 450 and other components. Those skilled in the art can understand that Figure 4 the terminal structure shown in

[0351] The processor 410 is the control center of the terminal, connecting various parts of the entire terminal through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 420, and by invoking the data stored in the memory 420, it executes various functions of the terminal and processes data. In some embodiments, the processor 410 may include one or more processing cores; in some embodiments, the processor 410 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 410 either.

[0352] The memory 420 can be used to store software programs and modules. The processor 410 executes various functional applications and data processing by running the software programs and modules stored in the memory 420. The memory 420 mainly includes a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, image playback function, etc.); the data storage area can store the data created according to the use of the terminal. In addition, the memory 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 420 may also include a memory controller to provide the processor 410 with access to the memory 420.

[0353] The terminal also includes a power supply 430 that powers each component. In some embodiments, the power supply 430 can be logically connected to the processor 410 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 430 may also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0354] The terminal may also include an input module 440, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0355] The terminal may also include a communication module 450. In some embodiments, the communication module 450 may include a wireless module. The terminal can perform short-distance wireless transmission through the wireless module of the communication module 450, thereby providing users with wireless broadband Internet access. For example, the communication module 450 can be used to help users send and receive emails, browse web pages, and access streaming media, etc.

[0356] Although not shown, the terminal may further include a display unit and the like, which will not be elaborated here. Specifically, in this embodiment, the processor 410 in the terminal will load the executable files corresponding to the processes of one or more application programs into the memory 420 according to the following instructions, and the processor 410 will run the application programs stored in the memory 420 to implement various functions as follows:

[0357] Obtain multimedia data and the feedback amount corresponding to the multimedia segment. The multimedia data includes multiple multimedia segments, and each multimedia segment includes multiple multimedia elements. The feedback amount represents the degree of user feedback on the multimedia segment;

[0358] Determine at least one key segment in the multimedia data according to the feedback amount corresponding to each multimedia segment;

[0359] Extract features of the multimedia elements in the key segment to obtain the segment features of the key segment;

[0360] Determine the overall features of the multimedia data according to the segment features of the key segment;

[0361] Perform content search based on the overall features of the multimedia data to obtain the search results corresponding to the multimedia data.

[0362] For the specific implementation of each of the above operations, reference may be made to the previous embodiments, which will not be elaborated here.

[0363] As can be seen from the above, through the feedback amount corresponding to the multimedia segment, the multimedia segments with a higher degree of user feedback, that is, the key segments, can be determined from multiple multimedia segments. The key segments can represent the important content of the multimedia data, and according to the multimedia elements in the key segments, the segment features of the key segments are determined. According to the segment features of the key segments, the overall features of the multimedia data are determined, reducing the multimedia elements required to calculate the overall features of the multimedia data. In this way, the overall features of the multimedia data can be obtained quickly, so that the corresponding search results can be quickly searched according to the overall features of the multimedia data. Thus, the content search efficiency is improved.

[0364] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions or by controlling related hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0365] Therefore, an embodiment of the present application provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions can be loaded by a processor to execute the steps in any content search method provided by the embodiment of the present application. For example, the instructions can execute the following steps:

[0366] Obtain multimedia data and the feedback amount corresponding to the multimedia segment. The multimedia data includes multiple multimedia segments, and each multimedia segment includes multiple multimedia elements. The feedback amount represents the degree of user feedback on the multimedia segment;

[0367] Determine at least one key segment in the multimedia data according to the feedback amount corresponding to each multimedia segment;

[0368] Extract features of the multimedia elements in the key segment to obtain the segment features of the key segment;

[0369] Determine the overall features of the multimedia data according to the segment features of the key segment;

[0370] Perform content search based on the overall features of the multimedia data to obtain the search results corresponding to the multimedia data.

[0371] Wherein, the storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

[0372] According to one aspect of the present application, a computer program product is provided, including computer programs / instructions, and the computer programs / instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer programs / instructions from the computer-readable storage medium, and the processor executes the computer programs / instructions, so that the electronic device executes the methods provided in various alternative implementations of the content search aspect in the above embodiments.

[0373] Since the instructions stored in the storage medium can execute the steps in any of the content search methods provided in the embodiments of the present application, therefore, the beneficial effects that can be achieved by any of the content search methods provided in the embodiments of the present application can be realized. For details, please refer to the previous embodiments and will not be repeated here.

[0374] The above has introduced in detail a content search method, device, electronic device, storage medium and program product provided by the embodiments of the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A content search method, characterized in that, Including: Obtain multimedia data and the feedback amount corresponding to a multimedia segment. The multimedia data includes a plurality of the multimedia segments, and each multimedia segment includes a plurality of multimedia elements. The feedback amount represents the degree of feedback of a user on the multimedia segment. When the sum of the feedback amounts corresponding to all the multimedia segments in the multimedia data is not less than a preset feedback amount, perform a sorting process on the plurality of multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted according to the feedback amount. When the multimedia element is composed of text data and an image, and it is determined that all the feedback amounts in the multimedia data are less than the preset feedback amount, determine the text amount ratio of the multimedia segment. The text amount ratio of the multimedia segment is the ratio of the text amount of all the text data in the multimedia data to the text amount of all the text data in the multimedia segment. Determine the color ratio. The color ratio is the ratio of the image color change amount of all the images in the multimedia data to the image color change amount of all the images in the multimedia segment. The image color change amount represents the degree of color change of the image. Perform a sorting process on the plurality of multimedia segments according to the text amount ratio and the color ratio to obtain the sorted multimedia segments. Or, when the multimedia element is composed of an image, and it is determined that all the feedback amounts in the multimedia data are less than the preset feedback amount, determine the color ratio. The color ratio is the ratio of the image color change amount of all the multimedia elements in the multimedia data to the image color change amount of all the multimedia elements in the multimedia segment. The image color change amount represents the degree of color change of the image. Perform a sorting process on the plurality of multimedia segments according to the color ratio to obtain the multimedia segments arranged according to the color ratio. Or, when the multimedia element is composed of text data, and the sum of the feedback amounts corresponding to all the multimedia segments in the multimedia data is not less than the preset feedback amount, perform a sorting process on the plurality of multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted according to the feedback amount. When all the feedback amounts in the multimedia data are less than the preset feedback amount, determine the text amount ratio of the multimedia segment. The text amount ratio of the multimedia segment is the ratio of the text amount of all the text data in the multimedia data to the text amount of all the text data in the multimedia segment. Perform a sorting process on the plurality of multimedia segments according to the text amount ratio to obtain the multimedia segments sorted according to the text amount ratio. Determine at least one key segment in the multimedia data according to the sorted multimedia segments. Extract features of the multimedia elements in the key segment to obtain the segment features of the key segment. Determine the overall features of the multimedia data according to the segment features of the key segment. Perform content search based on the overall features of the multimedia data to obtain the search results corresponding to the multimedia data.

2. The content search method according to claim 1, wherein Determining at least one key segment in the multimedia data according to the sorted multimedia segments includes: When the number of multimedia segments in the multimedia data is greater than a preset quantity, determining the preset quantity of key segments in the multimedia data according to the sorted multimedia segments.

3. The content search method according to claim 1, characterized in that, It further includes: When the number of multimedia segments in the multimedia data is not greater than the preset quantity, taking each multimedia segment in the multimedia data as a key segment.

4. The content search method according to claim 3, wherein Extracting features of multimedia elements in the key segments to obtain segment features of the key segments includes: Dividing the preset quantity by the number of multimedia segments in the multimedia data to obtain a target number; Determining the target number of target elements from the key segments; Extracting features of the target elements in the key segments to obtain segment features of the key segments.

5. The content search method according to claim 4, wherein Determining the target number of target elements from the key segments includes: When the target number is a decimal, performing a rounding operation on the target number to obtain a rounded target number; Multiplying the rounded target number by the number of multimedia segments to obtain a rounded total number; Determining a number difference, where the number difference is the difference between the rounded total number and the preset quantity; Determining N - 1 first key segments and 1 second key segment among multiple multimedia segments, where N is the number of multimedia segments; Determining the target number of target elements from the first key segments; Determining the number of new target elements from the second key segment, where the number of new target elements is the sum of the number difference and the rounded target number.

6. The content search method according to claim 5, wherein Determining N - 1 first key segments and 1 second key segment among multiple multimedia segments includes: When multiple multimedia segments are sorted from largest to smallest according to the feedback amount, the first key segments are the first N - 1 segments in the sorted multimedia segments; When multiple multimedia segments are sorted from smallest to largest according to the feedback amount, the first key segments are the last N - 1 segments in the sorted multimedia segments.

7. The content search method according to claim 1, characterized in that Determining the color ratio includes: Determining the pixel color difference between each pixel of adjacent two frames of multimedia elements in the multimedia segment; Summing the pixel color differences between each pixel of adjacent two frames of multimedia elements to obtain the color difference between adjacent two frames of multimedia elements; Summing the color differences between all adjacent two frames of multimedia elements in the multimedia segment to obtain the color difference corresponding to the multimedia segment; Summing the color differences corresponding to all multimedia segments in the multimedia data to obtain the color difference corresponding to the multimedia data; Determining the color ratio according to the color difference corresponding to the multimedia data and the color difference corresponding to the multimedia segment.

8. The content search method according to claim 1, characterized in that The multimedia element is composed of images. Before obtaining the multimedia data, it further includes: Determining target elements from the multimedia elements of the multimedia data according to the pixel color value of each pixel in the multimedia element; Segment the multimedia data according to the target element to obtain a plurality of multimedia segments.

9. The content search method according to claim 8, wherein The determining the target element from the multimedia element of the multimedia data according to the pixel color value of each pixel in the multimedia element includes: Determine the color mean value of each frame of the multimedia element, where the color mean value is the average value of the pixel colors of all pixels in the multimedia element; Use the multimedia element with the pixel color mean value greater than the preset pixel color as the target element.

10. The content search method according to claim 8, wherein The determining the target element from the multimedia element of the multimedia data according to the pixel color value of each pixel in the multimedia element includes: Determine the color change value of two adjacent frames of the multimedia element according to the pixel color value of each pixel in the two adjacent frames of the multimedia element; When it is determined that the color change value is greater than the preset color change value, use any one of the two adjacent frames of the multimedia element as the target element.

11. A content search device, characterized in that, Includes: An acquisition unit, configured to acquire multimedia data and a feedback amount corresponding to a multimedia segment, where the multimedia data includes a plurality of the multimedia segments, the multimedia segment includes a plurality of multimedia elements, and the feedback amount represents the feedback degree of a user on the multimedia segment; A first determination unit, configured to When the sum of the feedback amounts corresponding to all the multimedia segments in the multimedia data is not less than a preset feedback amount, perform a sorting process on the plurality of multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted according to the feedback amount; When the multimedia element is composed of text data and an image, and it is determined that all the feedback amounts in the multimedia data are less than the preset feedback amount, determine the text amount ratio of the multimedia segment, where the text amount ratio of the multimedia segment is the ratio of the text amount of all the text data in the multimedia data to the text amount of all the text data in the multimedia segment; Determine a color ratio, where the color ratio is the ratio of the image color change amount of all the images in the multimedia data to the image color change amount of all the images in the multimedia segment, and the image color change amount represents the color change degree of the image; Perform a sorting process on the plurality of multimedia segments according to the text amount ratio and the color ratio to obtain the sorted multimedia segments; Or, when the multimedia element is composed of an image, and it is determined that all the feedback amounts in the multimedia data are less than the preset feedback amount, determine a color ratio, where the color ratio is the ratio of the image color change amount of all the multimedia elements in the multimedia data to the image color change amount of all the multimedia elements in the multimedia segment, and the image color change amount represents the color change degree of the image; Perform a sorting process on the plurality of multimedia segments according to the color ratio to obtain the multimedia segments arranged according to the color ratio; Alternatively, when the multimedia element is composed of text data and the sum of the feedback amounts corresponding to all multimedia segments in the multimedia data is not less than a preset feedback amount, sorting processing is performed on the multiple multimedia segments according to the feedback amount corresponding to each multimedia segment to obtain the multimedia segments sorted according to the feedback amount; When all the feedback amounts in the multimedia data are less than the preset feedback amount, determining the text amount ratio of the multimedia segment, where the text amount ratio of the multimedia segment is the ratio of the text amount of all the text data in the multimedia data to the text amount of all the text data in the multimedia segment; Sorting processing is performed on the multiple multimedia segments according to the text amount ratio to obtain the multimedia segments sorted according to the text amount ratio; At least one key segment is determined in the multimedia data according to the sorted multimedia segments; A feature extraction unit, configured to extract features of the multimedia element in the key segment to obtain the segment features of the key segment; A second determination unit, configured to determine the overall features of the multimedia data according to the segment features of the key segment; A content search unit, configured to perform content search based on the overall features of the multimedia data to obtain the search result corresponding to the multimedia data.

12. An electronic device, characterized in that, Comprising a processor and a memory, where the memory stores multiple instructions; the processor loads the instructions from the memory to execute the steps in the content search method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores multiple instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the content search method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, Comprising a computer program / instructions, where when the computer program / instructions are executed by a processor, the steps in the content search method according to any one of claims 1 to 10 are implemented.

Images