Video retrieval method, device, equipment, medium and program product

Abstract

The present application provides a video retrieval method, device, equipment, medium and program product to solve the problem of low efficiency of related video retrieval. The method includes: performing emotion recognition on the user to determine the target video segment that the user is interested in in the video to be retrieved that the user is watching; scanning and recording the target video segment to obtain a first video file, and streamlining the first video file to obtain a second video file; performing feature analysis on the second video file to obtain video feature information; matching the video feature information with the preset information in the video library, and obtaining the film and television information corresponding to the target video segment based on the matching result. The present application can improve the efficiency of video retrieval.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video retrieval method, device, equipment, medium and program product. Background Art

[0002] At present, in the input method of video retrieval, whether it is based on text description information or video clips, users are required to actively input retrieval information or upload video files, and then parse and match them. When a user needs to search for the complete plot of an unfamiliar video clip, using text information to search requires generating relevant memories of the target video and forming language text, which increases memory load and conversion pressure. If using video clip search, users need to save video clips in various ways, and then actively upload relevant video libraries for parsing, which prolongs the operation path, is time-consuming and labor-intensive, and results in low efficiency of video retrieval. Summary of the invention

[0003] The embodiments of the present application provide a video retrieval method, apparatus, device, medium and program product to solve the problem of low efficiency of related video retrieval.

[0004] To solve the above technical problems, this application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a video retrieval method, comprising:

[0006] Perform emotion recognition on the user to determine the target video clips that the user is interested in among the videos to be retrieved that the user is watching;

[0007] Scanning and recording the target video clip to obtain a first video file, and streamlining the first video file to obtain a second video file;

[0008] Performing feature analysis on the second video file to obtain video feature information;

[0009] The video feature information is matched with preset information in a video library, and film and television information corresponding to the target video segment is obtained based on the matching result.

[0010] Optionally, the performing emotion recognition on the user to determine the target video segment that the user is interested in among the videos to be retrieved and watched by the user includes:

[0011] When a user watches a video to be retrieved, recording a facial video of the user;

[0012] Using a facial emotion recognition model to perform emotion recognition on the facial video to obtain emotion information of the user watching the video to be retrieved;

[0013] When the calm element value indicated by the emotion information of the first segment is lower than the first value, the first segment is determined to be a target video segment of interest to the user, the first segment is a video segment in the video to be retrieved, and the first value is an average of multiple emotion element values ​​indicated by the emotion information of the first segment, and the multiple emotion element values ​​include the calm element value.

[0014] Optionally, the streamlining the first video file to obtain the second video file includes:

[0015] Determine the stillness corresponding to each frame in the first video file by using a stillness model;

[0016] Determine a micro-static segment according to the stillness corresponding to each frame in the first video file, wherein the micro-static segment is a video segment in the first video file whose stillness is lower than a first threshold;

[0017] According to the stillness corresponding to each frame in the micro-still segment, the micro-still segment in the first video file is subjected to frame extraction processing to obtain a second video file.

[0018] Optionally, the stillness model includes a lens picture module and a speech recognition module, and the step of using the stillness model to determine the stillness corresponding to each frame in the first video file includes:

[0019] Determine a color histogram corresponding to each frame in the first video file by using the lens picture module;

[0020] Determine the timbre characteristics corresponding to each frame of the first video file by using the speech recognition module;

[0021] According to the color histogram and timbre characteristics of the i-1th frame and the i-th frame in the first video file, respectively determine the color histogram similarity and timbre difference between the i-1th frame and the i-th frame;

[0022] The stillness of the i-th frame is determined according to the color histogram similarity and timbre difference between the i-1-th frame and the i-th frame, wherein i is an integer greater than 1.

[0023] Optionally, performing frame extraction processing on the micro-static segment in the first video file according to the stillness corresponding to each frame in the micro-static segment to obtain the second video file includes:

[0024] Determine the number of frame extractions according to the stillness corresponding to each frame in the micro-still segment and the second threshold, wherein the number of frame extractions is the product of the sum of the number of frames whose stillness exceeds the second threshold and a second value, and the second value is used to adjust the frame extraction amplitude;

[0025] According to the frame extraction number, frame extraction is performed on the micro-static segment in the first video file to obtain a second video file.

[0026] Optionally, scanning and recording the target video segment to obtain a first video file includes:

[0027] The target video clip is scanned and recorded by screen recording or video recording to obtain a first video file.

[0028] Optionally, the method further comprises:

[0029] Sorting the video information according to the play order of the video associated with the target video clip to obtain a video list corresponding to the target video clip;

[0030] The video information includes at least one of the following:

[0031] The total number of movies and TV programs associated with the target video clip, the names of the movies and TV programs, and the time location of the video content of the target video clip in the corresponding movies and TV programs.

[0032] In a second aspect, an embodiment of the present application provides a video retrieval device, including:

[0033] A determination module is used to perform emotion recognition on the user and determine a target video segment that the user is interested in among the videos to be retrieved that the user is watching;

[0034] A first processing module, configured to scan and record the target video clip to obtain a first video file, and to simplify the first video file to obtain a second video file;

[0035] An analysis module, used for performing feature analysis on the second video file to obtain video feature information;

[0036] The second processing module is used to match the video feature information with the preset information in the video library, and obtain the film and television information corresponding to the target video segment based on the matching result.

[0037] Optionally, the determining module includes:

[0038] A first recording unit, configured to record a facial video of the user when the user is watching the video to be retrieved;

[0039] A recognition unit, used to perform emotion recognition on the facial video using a facial emotion recognition model to obtain emotion information of the user when watching the video to be retrieved;

[0040] A first determination unit is configured to determine that the first segment is a target video segment of interest to a user when a calm element value indicated by the emotional information of the first segment is lower than a first value, the first segment being a video segment in the video to be retrieved, and the first value being an average of multiple emotional element values ​​indicated by the emotional information of the first segment, the multiple emotional element values ​​including the calm element value.

[0041] Optionally, the first processing module includes:

[0042] A second determining unit, configured to determine the stillness corresponding to each frame in the first video file by using a stillness model;

[0043] a third determining unit, configured to determine a micro-static segment according to the stillness corresponding to each frame in the first video file, wherein the micro-static segment is a video segment in the first video file whose stillness is lower than a first threshold;

[0044] The processing unit is used to perform frame extraction processing on the micro-static segment in the first video file according to the stillness corresponding to each frame of the micro-static segment to obtain a second video file.

[0045] Optionally, the stillness model includes a lens picture module and a speech recognition module, and the second determination unit is specifically used to:

[0046] Determine a color histogram corresponding to each frame in the first video file by using the lens picture module;

[0047] Determine the timbre characteristics corresponding to each frame of the first video file by using the speech recognition module;

[0048] According to the color histogram and timbre characteristics of the i-1th frame and the i-th frame in the first video file, respectively determine the color histogram similarity and timbre difference between the i-1th frame and the i-th frame;

[0049] The stillness of the i-th frame is determined according to the color histogram similarity and timbre difference between the i-1-th frame and the i-th frame, wherein i is an integer greater than 1.

[0050] Optionally, the processing unit is specifically configured to:

[0051] Determine the number of frame extractions according to the stillness corresponding to each frame in the micro-still segment and the second threshold, wherein the number of frame extractions is the product of the sum of the number of frames whose stillness exceeds the second threshold and a second value, and the second value is used to adjust the frame extraction amplitude;

[0052] According to the frame extraction number, frame extraction is performed on the micro-static segment in the first video file to obtain a second video file.

[0053] Optionally, the first processing module includes:

[0054] The second recording unit is used to scan and record the target video segment in a screen recording manner or a video recording manner to obtain a first video file.

[0055] Optionally, the device further comprises:

[0056] A sorting module, used to sort the video information according to the play order of the video associated with the target video segment, to obtain a video list corresponding to the target video segment;

[0057] The video information includes at least one of the following:

[0058] The total number of movies and TV programs associated with the target video clip, the names of the movies and TV programs, and the time location of the video content of the target video clip in the corresponding movies and TV programs.

[0059] In a third aspect, an embodiment of the present application provides an electronic device, including a transceiver and a processor, wherein the processor is configured to:

[0060] Perform emotion recognition on the user to determine the target video clips that the user is interested in among the videos to be retrieved that the user is watching;

[0061] Scanning and recording the target video clip to obtain a first video file, and streamlining the first video file to obtain a second video file;

[0062] Performing feature analysis on the second video file to obtain video feature information;

[0063] The video feature information is matched with preset information in a video library, and film and television information corresponding to the target video segment is obtained based on the matching result.

[0064] Optionally, the processor is specifically configured to:

[0065] When a user watches a video to be retrieved, recording a facial video of the user;

[0066] Using a facial emotion recognition model to perform emotion recognition on the facial video to obtain emotion information of the user watching the video to be retrieved;

[0067] When the calm element value indicated by the emotion information of the first segment is lower than the first value, the first segment is determined to be a target video segment of interest to the user, the first segment is a video segment in the video to be retrieved, and the first value is an average of multiple emotion element values ​​indicated by the emotion information of the first segment, and the multiple emotion element values ​​include the calm element value.

[0068] Optionally, the processor is specifically configured to:

[0069] Determine the stillness corresponding to each frame in the first video file by using a stillness model;

[0070] Determine a micro-static segment according to the stillness corresponding to each frame in the first video file, wherein the micro-static segment is a video segment in the first video file whose stillness is lower than a first threshold;

[0071] According to the stillness corresponding to each frame in the micro-still segment, the micro-still segment in the first video file is subjected to frame extraction processing to obtain a second video file.

[0072] Optionally, the stillness model includes a lens picture module and a speech recognition module, and the processor is specifically used to:

[0073] Determine a color histogram corresponding to each frame in the first video file by using the lens picture module;

[0074] Determine the timbre characteristics corresponding to each frame of the first video file by using the speech recognition module;

[0075] According to the color histogram and timbre characteristics of the i-1th frame and the i-th frame in the first video file, respectively determine the color histogram similarity and timbre difference between the i-1th frame and the i-th frame;

[0076] The stillness of the i-th frame is determined according to the color histogram similarity and timbre difference between the i-1-th frame and the i-th frame, wherein i is an integer greater than 1.

[0077] Optionally, the processor is specifically configured to:

[0078] Determine the number of frame extractions according to the stillness corresponding to each frame in the micro-still segment and the second threshold, wherein the number of frame extractions is the product of the sum of the number of frames whose stillness exceeds the second threshold and a second value, and the second value is used to adjust the frame extraction amplitude;

[0079] According to the frame extraction number, frame extraction is performed on the micro-static segment in the first video file to obtain a second video file.

[0080] Optionally, the processor is specifically configured to:

[0081] The target video clip is scanned and recorded by screen recording or video recording to obtain a first video file.

[0082] Optionally, the processor is further configured to:

[0083] Sorting the video information according to the play order of the video associated with the target video clip to obtain a video list corresponding to the target video clip;

[0084] The video information includes at least one of the following:

[0085] The total number of movies and TV programs associated with the target video clip, the names of the movies and TV programs, and the time location of the video content of the target video clip in the corresponding movies and TV programs.

[0086] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the video retrieval method described in the first aspect above.

[0087] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the video retrieval method described in the first aspect above are implemented.

[0088] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the video retrieval method as described in the first aspect above.

[0089] In an embodiment of the present application, the above-mentioned video retrieval method can determine the target video segment that the user is interested in in the video to be retrieved watched by the user by performing emotion recognition on the user, scan and record the target video segment to obtain a first video file, streamline the first video file to obtain a second video file, perform feature analysis on the second video file to obtain video feature information, and match the video feature information with preset information in the video library, and obtain the film and television information corresponding to the target video segment based on the matching result, so that the user does not need to manually input or upload the video file, and the video segment that the user is interested in can be automatically identified without the user manually selecting, thereby saving the user's time and reducing the user's memory load, thereby improving the efficiency of video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0091] Figure 1 This is one of the flow charts of a video retrieval method provided in an embodiment of the present application;

[0092] Figure 2 This is one of the interactive schematic diagrams of a video retrieval provided in an embodiment of the present application;

[0093] Figure 3 This is the second interactive schematic diagram of a video retrieval provided in an embodiment of the present application;

[0094] Figure 4 This is a third interactive schematic diagram of a video retrieval provided in an embodiment of the present application;

[0095] Figure 5 This is the second flowchart of a video retrieval method provided in an embodiment of the present application;

[0096] Figure 6 is a structural schematic diagram of a video retrieval device provided in an embodiment of the present application;

[0097] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0099] For ease of understanding, some contents involved in the embodiments of the present application are described below:

[0100] Driven by multiple factors such as technological progress, the rise of digital media platforms and the popularization of mobile devices, video has become one of the most important media forms and communication methods in people's daily production and life. Currently, various applications, including short video and long video websites, provide search functions to facilitate users to find target videos in massive videos.

[0101] In terms of video retrieval input, related technologies provide the following solutions:

[0102] 1. Based on user input text: This type of method generally uses a neural network to extract video features from the entire video, and another neural network to extract text features from the query text. Based on the text features, all video features in the video library are sorted by relevance, and the most relevant videos to the query text are found, and the video collection is returned to the user in order. This method has now been widely used.

[0103] 2. Based on shot features: The related technology provides a method to find complete videos through video clips. When a user uploads a video, the feature extraction model is used to extract features of the video shots. The cosine similarity is used to calculate the cosine similarity of the shot feature vectors of the video clip to be retrieved and the shot feature vectors of the videos in the library one by one, and the similarities are sorted. The complete video with the most similar shot is taken as the search result.

[0104] In terms of video segmentation and recognition, existing technologies provide the following solutions:

[0105] 1. Shot segmentation method based on similarity: The related technology provides a shot boundary detection method, which uses a new neural network architecture AutoShot, which is an improved model based on the TransnetV2 model. The main principle is to use the model to process the input video frame by frame, compare the output features of adjacent frames for similarity, and if it is lower than a certain threshold, it is regarded as a shot boundary, thereby dividing the entire video into different shots from multiple shot boundaries.

[0106] 2. Based on viewing time and emotion recognition: The relevant technology provides a video push method and system based on face recognition technology, which proposes to identify and analyze the video watched by the last user based on the viewing time factor and the emotion factor. When it is greater than a certain threshold, the keywords of the video are extracted, and the extracted keywords are used to retrieve videos and push similar videos to users.

[0107] At present, in the input method of video retrieval, whether it is based on text description information or video clips, users are required to actively input retrieval information or upload video files, and then parse and match them. In addition, in the video segmentation and recognition methods, related technologies mostly focus on video images and perform shot segmentation based on similarity, but for video retrieval, this method can only refine the video granularity and cannot optimize the accuracy.

[0108] In the embodiments of the present application, a video retrieval method, apparatus, device, medium and program product are proposed to solve the problem of low efficiency of related video retrieval.

[0109] See also Figure 1 , Figure 1is one of the flow charts of a video retrieval method provided in an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0110] Step 101: Perform emotion recognition on the user to determine the target video clips that the user is interested in among the videos to be retrieved and watched by the user.

[0111] Specifically, the video retrieval method can be applied to the first terminal, and the emotion recognition of the user can be to recognize the emotion of the user when watching the video to be retrieved, and can be specifically to use the first application of the first terminal to perform emotion recognition. The video to be retrieved can be a video watched by the user when using the second application of the second terminal, and the first terminal and the second terminal can be the same device or different devices. The target video segment can be a segment of the video to be retrieved that the user is interested in, and can be determined by the emotional changes of the user.

[0112] Step 102: Scan and record the target video clip to obtain a first video file, and simplify the first video file to obtain a second video file.

[0113] Specifically, the above-mentioned streamlining process may be slicing or frame extraction of the first video file to reduce the size of the video file uploaded to the server for feature analysis.

[0114] Step 103: Perform feature analysis on the second video file to obtain video feature information.

[0115] Specifically, the above-mentioned feature analysis of the second video file can be to upload the second video file to the server, and through the trained model, use a neural network architecture mixed convolutional 3D network (Mixed Convolutional 3D Network, MC3) to capture the dynamic features of the video, and extract key feature information in the target video, including but not limited to character information, scene information, barrage text information, etc.

[0116] Step 104: Match the video feature information with the preset information in the video library, and obtain the film and television information corresponding to the target video clip based on the matching result.

[0117] Specifically, the above-mentioned matching process of the video feature information with the preset information in the video library may be to use the video feature information as an index to perform feature matching with the existing information in the video library to obtain a matching result, and obtain the film and television information corresponding to the target video segment based on the matching result.

[0118] In an embodiment of the present application, the above-mentioned video retrieval method can determine the target video segment that the user is interested in in the video to be retrieved watched by the user by performing emotion recognition on the user, scan and record the target video segment to obtain a first video file, streamline the first video file to obtain a second video file, perform feature analysis on the second video file to obtain video feature information, and match the video feature information with preset information in the video library, and obtain the film and television information corresponding to the target video segment based on the matching result, so that the user does not need to manually input or upload the video file, and the video segment that the user is interested in can be automatically identified without the user manually selecting, thereby saving the user's time and reducing the user's memory load, thereby improving the efficiency of video retrieval.

[0119] Optionally, the performing emotion recognition on the user to determine the target video segment that the user is interested in among the videos to be retrieved and watched by the user includes:

[0120] When a user watches a video to be retrieved, recording a facial video of the user;

[0121] Using a facial emotion recognition model to perform emotion recognition on the facial video to obtain emotion information of the user watching the video to be retrieved;

[0122] When the calm element value indicated by the emotion information of the first segment is lower than the first value, the first segment is determined to be a target video segment of interest to the user, the first segment is a video segment in the video to be retrieved, and the first value is an average of multiple emotion element values ​​indicated by the emotion information of the first segment, and the multiple emotion element values ​​include the calm element value.

[0123] Specifically, the facial video of the user can be recorded by turning on the front camera to record the face of the user while the user is watching the video to be retrieved, so as to obtain the facial video of the user. The facial emotion recognition model can be a facial emotion recognition model based on a deep convolutional neural network. For example, the facial emotion recognition model is obtained by training a multi-layer convolutional neural network model through facial emotion recognition (FER), and the facial feature contour of the user is recognized by detecting the face position in the image, and then analyzing the emotion corresponding to the facial expression, such as happiness, surprise, calmness, sadness, etc.

[0124] The above-mentioned emotional information may be the emotion corresponding to the user's facial expression determined by a facial emotion recognition model, including the emotion type and the element value corresponding to the emotion type. The above-mentioned calm element value may be the element value corresponding to the calm emotion determined by the facial emotion recognition model. The above-mentioned multiple emotion element values ​​may be all emotions corresponding to facial expressions, such as element values ​​corresponding to happiness, surprise, calmness, sadness, etc.

[0125] For example, assume that the input facial image is represented as X. After passing through the convolution layer, pooling layer and fully connected layer, the output vector Y is obtained, where each element of Y represents the probability distribution of the corresponding emotion category. The process is:

[0126] Y=f(X;W,b)

[0127] Among them, f is the mapping function of the entire neural network, W and b are the weight and bias parameters in the network respectively.

[0128] During the training process, the network parameters are adjusted by minimizing the loss function L so that the facial emotion recognition model can predict the emotion category corresponding to the input image as accurately as possible. The training objective of the facial emotion recognition model is expressed as:

[0129] Min w,b L(Y pred ,Y true )

[0130] Among them, Y pred is the probability distribution of emotion categories predicted by the model, Y true is the actual emotion label. Commonly used loss functions include cross entropy loss function.

[0131] The user's interest is calculated based on the trained facial emotion recognition model. When the calm element value in the Y vector is lower than the average value of the overall element value, the video corresponding to the interval is segmented and regarded as the target video segment that the user is interested in.

[0132] In this embodiment, the above-mentioned video retrieval method can record the facial video of the user when the user is watching the video to be retrieved, use the facial emotion recognition model to perform emotion recognition on the facial video, obtain the emotional information of the user watching the video to be retrieved, and determine the target video segment that the user is interested in based on the emotional information, thereby accurately determining the target video segment that the user is interested in, thereby improving the accuracy of video retrieval.

[0133] Optionally, the streamlining the first video file to obtain the second video file includes:

[0134] Determine the stillness corresponding to each frame in the first video file by using a stillness model;

[0135] Determine a micro-static segment according to the stillness corresponding to each frame in the first video file, wherein the micro-static segment is a video segment in the first video file whose stillness is lower than a first threshold;

[0136] According to the stillness corresponding to each frame in the micro-still segment, the micro-still segment in the first video file is subjected to frame extraction processing to obtain a second video file.

[0137] Specifically, the stillness model can be used to determine the stillness corresponding to each frame in the video, and the stillness can be understood as the similarity between the current frame and the previous frame. It can be understood that the micro-still video clip is specifically defined as a relatively still picture, such as the scene remains unchanged, the character's movement amplitude and position offset are extremely small, or the main picture is still with only commentary, subtitles or emoticons, which are sometimes present in the video gaps. Micro-still video clips are clips with low information density for the video retrieval process. If they are not processed, the retrieval efficiency will be reduced, and in severe cases, they will even interfere with the retrieval results.

[0138] In this embodiment, the above-mentioned video retrieval method can determine the stillness corresponding to each frame in the first video file by utilizing the stillness model, determine the micro-still segment according to the stillness corresponding to each frame in the first video file, and perform frame extraction processing on the micro-still segment in the first video file according to the stillness corresponding to each frame in the micro-still segment to obtain a second video file, so that the target video segment that needs to be uploaded to the back-end server and feature extracted later can be simplified, thereby improving the granularity and accuracy of uploading the target video segment.

[0139] Optionally, the stillness model includes a lens picture module and a speech recognition module, and the step of using the stillness model to determine the stillness corresponding to each frame in the first video file includes:

[0140] Determine a color histogram corresponding to each frame in the first video file by using the lens picture module;

[0141] Determine the timbre characteristics corresponding to each frame of the first video file by using the speech recognition module;

[0142] According to the color histogram and timbre characteristics of the i-1th frame and the i-th frame in the first video file, respectively determine the color histogram similarity and timbre difference between the i-1th frame and the i-th frame;

[0143] The stillness of the i-th frame is determined according to the color histogram similarity and timbre difference between the i-1-th frame and the i-th frame, wherein i is an integer greater than 1.

[0144] Specifically, the above-mentioned lens picture module can be used to calculate the color histogram of each frame of the video, divide the RGB color space of the image into three channels (R, G, B), each channel has a value range of 0 to 255, divide it into a certain number of intervals (bins), count the number or proportion of pixels in each channel bin, and merge the histograms of each color channel into a multidimensional RGB histogram as the feature value of the image frame.

[0145] The above-mentioned speech recognition module can be used to calculate all the timbres involved in the video, such as original soundtracks, background music, narration voices, sound effects, etc.

[0146] Exemplarily, assuming that the stillness model is S, the video segment V contains N frames, C(Vi) represents the color histogram of the i-th frame, and A(Vi) represents the timbre of the i-th frame.

[0147] The stationary model S(V) is defined as:

[0148]

[0149] Among them, Sim(C(V i-1 ),C(V i )) represents the color histogram similarity between the i-1th frame and the i-th frame, which can be calculated by a distance measurement method (such as Euclidean distance, cosine similarity, etc.). The higher the similarity value, the smaller the picture change and the higher the static degree;

[0150] Diff(A(Vi-1),A(Vi)) represents the timbre difference between the i-1th frame and the i-th frame, which can be calculated by measuring the distance between the timbre features. The smaller the timbre difference, the smaller the picture change and the higher the static degree.

[0151] In this embodiment, the above-mentioned video retrieval method can analyze the two features of lens image and voice recognition through the stillness model to determine the stillness of the target video clip. When the stillness value is lower than the preset threshold, it can be regarded as a micro-still video clip. Compared with the existing video shot segmentation rules in the related technology, such as fixed time interval, number of segmented segments, and video segmentation according to different scenes and angles, it can accurately identify micro-still video clips, thereby further improving the granularity and accuracy of target video uploading.

[0152] Optionally, performing frame extraction processing on the micro-static segment in the first video file according to the stillness corresponding to each frame in the micro-static segment to obtain the second video file includes:

[0153] Determine the number of frame extractions according to the stillness corresponding to each frame in the micro-still segment and the second threshold, wherein the number of frame extractions is the product of the sum of the number of frames whose stillness exceeds the second threshold and a second value, and the second value is used to adjust the frame extraction amplitude;

[0154] According to the frame extraction number, frame extraction is performed on the micro-static segment in the first video file to obtain a second video file.

[0155] Specifically, the second value mentioned above can be used to adjust the frame extraction amplitude, and the specific value can be continuously adjusted and changed according to the actual effect.

[0156] Exemplarily, assuming that the video segment V contains N frames, a stillness threshold, that is, a second threshold, is set, and when the stillness of a frame exceeds the threshold, video frame reduction is performed.

[0157] Define the number of frames after video frame reduction as M, and calculate it using the following formula

[0158]

[0159] Among them, k is a coefficient used to adjust the amplitude of frame reduction, k>0; δ(x) is a function with a value of 0 or 1, indicating that it takes a value of 1 when x>0 and a value of 0 otherwise; S(Vi) represents the stillness of the i-th frame; θ is the stillness threshold; the number of frames reduced is the product of k and the sum of the number of frames whose stillness exceeds the threshold. The amplitude of frame reduction can be controlled by adjusting the value of k.

[0160] In this embodiment, the above-mentioned video retrieval method can determine the number of frame extraction based on the stillness corresponding to each frame in the micro-still segment and a second threshold, the number of frame extraction is the product of the sum of the frames whose stillness exceeds the second threshold and the second value, the second value is used to adjust the frame extraction amplitude, and based on the number of frame extraction, the micro-still segment in the first video file is subjected to frame extraction processing, so that the number of frame extractions can be calculated according to the stillness value of the picture, and then the video segment can be simplified, thereby further improving the accuracy of video retrieval.

[0161] Optionally, scanning and recording the target video segment to obtain a first video file includes:

[0162] The target video clip is scanned and recorded by screen recording or video recording to obtain a first video file.

[0163] Specifically, the screen recording method may be to directly record the content displayed on the computer screen through the first application of the first terminal and generate a first video file, and the video recording method may be to use the rear camera of the first terminal to shoot the video played on the second application of the second terminal to generate a first video file.

[0164] For example, Figure 2 This is one of the interactive schematic diagrams of a video retrieval provided in the embodiment of the present application, such as Figure 2 As shown, add a video scanning and retrieval button in Application 1. Click the video scanning and retrieval button of Application 1 to display the scanning interface. Click the Start button in the default interface to use the rear camera to scan and record videos on other devices. Figure 3 This is a second interactive schematic diagram of a video retrieval provided in an embodiment of the present application, such as Figure 3 As shown, after clicking the switch button in the interface, you can switch to other applications in the same device, such as application 2, and add a control of application 1 above the layer of application 2 to display the search progress in real time, feedback the search results, and return to the functional operation of application 1. The existence of this control does not affect the normal operation of application 2 and the normal playback of the video.

[0165] In this embodiment, the above-mentioned video retrieval method can use screen recording or video recording to scan and record the target video clip to obtain a first video file, so that cross-application scanning can be used for video retrieval without the need for users to manually input or upload video files, thereby improving the convenience and efficiency of users' video retrieval.

[0166] Optionally, the method further comprises:

[0167] Sorting the video information according to the play order of the video associated with the target video clip to obtain a video list corresponding to the target video clip;

[0168] The video information includes at least one of the following:

[0169] The total number of movies and TV programs associated with the target video clip, the names of the movies and TV programs, and the time location of the video content of the target video clip in the corresponding movies and TV programs.

[0170] Specifically, when the target video clip involves multiple movies and TV shows, such as a mixed-cut video, the number of movies and TV shows identified is synchronously fed back during the scanning process. After all identification and matching are completed, the user can return to the first application to view the retrieved video set.

[0171] For example, Figure 4 This is a third interactive schematic diagram of a video retrieval provided in an embodiment of the present application, such as Figure 4 As shown, the interface of the first application can present the corresponding film and television list in the order of video clip identification, and display the location of the target video clip in the film and television, that is, the time period of which episode it appears. For example, target clip 1 appears in the 20th to 22nd minute of the first episode of film and television 1, and the user can click to watch the full video immediately.

[0172] In this embodiment, the above-mentioned video retrieval method can obtain a film and television list corresponding to the target video clip by sorting the film and television information according to the playback order of the film and television associated with the target video clip, and the film and television information includes at least one of the following: the total number of films and television associated with the target video clip, the name of the film and television, and the time positioning of the video content of the target video clip in the corresponding film and television, so that the user can clearly understand and use the retrieved film and television information, thereby further improving the user experience and user satisfaction.

[0173] For example, Figure 5 This is a second flowchart of a video retrieval method provided in an embodiment of the present application. Figure 5 As shown, the specific process includes the following:

[0174] Click the cross-app scanning function in the search interface of App 1;

[0175] Switch to other applications in the same device, such as Application 2, and scan and record the videos played in Application 2. You can also scan and record videos played in other devices, and start and stop scanning at any time;

[0176] The video will be segmented automatically during scanning, and the video clips that the user is interested in will be determined based on the emotion recognition model;

[0177] Then, the video is streamlined according to the stillness segmentation method, and after processing, the video slices are automatically uploaded to the server of application 1;

[0178] Parse the video clips, extract feature information and match it with the video library;

[0179] Finally, according to the order of the videos involved in the target video clip, the corresponding complete film and television list is displayed.

[0180] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a video retrieval device provided in an embodiment of the present application, such as Figure 6 As shown, the video retrieval device 600 includes:

[0181] A determination module 601 is used to perform emotion recognition on the user and determine a target video segment that the user is interested in from the video to be retrieved that the user is watching;

[0182] A first processing module 602 is used to scan and record the target video clip to obtain a first video file, and to simplify the first video file to obtain a second video file;

[0183] The analysis module 603 is used to perform feature analysis on the second video file to obtain video feature information;

[0184] The second processing module 604 is used to match the video feature information with the preset information in the video library, and obtain the film and television information corresponding to the target video segment based on the matching result.

[0185] Optionally, the determining module includes:

[0186] A first recording unit, configured to record a facial video of the user when the user is watching the video to be retrieved;

[0187] A recognition unit, used to perform emotion recognition on the facial video using a facial emotion recognition model to obtain emotion information of the user when watching the video to be retrieved;

[0188] A first determination unit is configured to determine that the first segment is a target video segment of interest to a user when a calm element value indicated by the emotional information of the first segment is lower than a first value, the first segment being a video segment in the video to be retrieved, and the first value being an average of multiple emotional element values ​​indicated by the emotional information of the first segment, the multiple emotional element values ​​including the calm element value.

[0189] Optionally, the first processing module includes:

[0190] A second determining unit, configured to determine the stillness corresponding to each frame in the first video file by using a stillness model;

[0191] a third determining unit, configured to determine a micro-static segment according to the stillness corresponding to each frame in the first video file, wherein the micro-static segment is a video segment in the first video file whose stillness is lower than a first threshold;

[0192] The processing unit is used to perform frame extraction processing on the micro-static segment in the first video file according to the stillness corresponding to each frame of the micro-static segment to obtain a second video file.

[0193] Optionally, the stillness model includes a lens picture module and a speech recognition module, and the second determination unit is specifically used to:

[0194] Determine a color histogram corresponding to each frame in the first video file by using the lens picture module;

[0195] Determine the timbre characteristics corresponding to each frame of the first video file by using the speech recognition module;

[0196] According to the color histogram and timbre characteristics of the i-1th frame and the i-th frame in the first video file, respectively determine the color histogram similarity and timbre difference between the i-1th frame and the i-th frame;

[0197] The stillness of the i-th frame is determined according to the color histogram similarity and timbre difference between the i-1-th frame and the i-th frame, wherein i is an integer greater than 1.

[0198] Optionally, the processing unit is specifically configured to:

[0199] Determine the number of frame extractions according to the stillness corresponding to each frame in the micro-still segment and the second threshold, wherein the number of frame extractions is the product of the sum of the number of frames whose stillness exceeds the second threshold and a second value, and the second value is used to adjust the frame extraction amplitude;

[0200] According to the frame extraction number, frame extraction is performed on the micro-static segment in the first video file to obtain a second video file.

[0201] Optionally, the first processing module includes:

[0202] The second recording unit is used to scan and record the target video segment in a screen recording manner or a video recording manner to obtain a first video file.

[0203] Optionally, the device further comprises:

[0204] A sorting module, used to sort the video information according to the play order of the video associated with the target video segment, to obtain a video list corresponding to the target video segment;

[0205] The video information includes at least one of the following:

[0206] The total number of movies and TV programs associated with the target video clip, the names of the movies and TV programs, and the time location of the video content of the target video clip in the corresponding movies and TV programs.

[0207] It should be noted that the video retrieval device provided in the embodiment of the present application is a device capable of executing the above-mentioned video retrieval method, and all implementation methods in the above-mentioned video retrieval method embodiment are applicable to the device and can achieve the same or similar beneficial effects. To avoid repeated description, this embodiment will not be described in detail.

[0208] For details, see Figure 7 As shown, an embodiment of the present application further provides an electronic device, including a bus 701, a transceiver 702, an antenna 703, a bus interface 704, a processor 705 and a memory 706.

[0209] The processor 705 is used for:

[0210] Perform emotion recognition on the user to determine the target video clips that the user is interested in among the videos to be retrieved that the user is watching;

[0211] Scanning and recording the target video clip to obtain a first video file, and streamlining the first video file to obtain a second video file;

[0212] Performing feature analysis on the second video file to obtain video feature information;

[0213] The video feature information is matched with preset information in a video library, and film and television information corresponding to the target video segment is obtained based on the matching result.

[0214] exist Figure 7 In the embodiment, the bus architecture (represented by bus 701) may include any number of interconnected buses and bridges, and bus 701 links various circuits including one or more processors represented by processor 705 and memory represented by memory 706. Bus 701 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. Bus interface 704 provides an interface between bus 701 and transceiver 702. Transceiver 702 may be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. Data processed by processor 705 is transmitted on a wireless medium via antenna 703, and further, antenna 703 also receives data and transmits the data to processor 705.

[0215] The processor 705 is responsible for managing the bus 701 and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management and other control functions. The memory 706 can be used to store data used by the processor 705 when performing operations.

[0216] Optionally, the processor 705 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD).

[0217] Optionally, the processor 705 is specifically configured to:

[0218] When a user watches a video to be retrieved, recording a facial video of the user;

[0219] Using a facial emotion recognition model to perform emotion recognition on the facial video to obtain emotion information of the user watching the video to be retrieved;

[0220] When the calm element value indicated by the emotion information of the first segment is lower than the first value, the first segment is determined to be a target video segment of interest to the user, the first segment is a video segment in the video to be retrieved, and the first value is an average of multiple emotion element values ​​indicated by the emotion information of the first segment, and the multiple emotion element values ​​include the calm element value.

[0221] Optionally, the processor 705 is specifically configured to:

[0222] Determine the stillness corresponding to each frame in the first video file by using a stillness model;

[0223] Determine a micro-static segment according to the stillness corresponding to each frame in the first video file, wherein the micro-static segment is a video segment in the first video file whose stillness is lower than a first threshold;

[0224] According to the stillness corresponding to each frame in the micro-still segment, the micro-still segment in the first video file is subjected to frame extraction processing to obtain a second video file.

[0225] Optionally, the stillness model includes a lens picture module and a speech recognition module, and the processor 705 is specifically configured to:

[0226] Determine a color histogram corresponding to each frame in the first video file by using the lens picture module;

[0227] Determine the timbre characteristics corresponding to each frame of the first video file by using the speech recognition module;

[0228] According to the color histogram and timbre characteristics of the i-1th frame and the i-th frame in the first video file, respectively determine the color histogram similarity and timbre difference between the i-1th frame and the i-th frame;

[0229] The stillness of the i-th frame is determined according to the color histogram similarity and timbre difference between the i-1-th frame and the i-th frame, wherein i is an integer greater than 1.

[0230] Optionally, the processor 705 is specifically configured to:

[0231] Determine the number of frame extractions according to the stillness corresponding to each frame in the micro-still segment and the second threshold, wherein the number of frame extractions is the product of the sum of the number of frames whose stillness exceeds the second threshold and a second value, and the second value is used to adjust the frame extraction amplitude;

[0232] According to the frame extraction number, frame extraction is performed on the micro-static segment in the first video file to obtain a second video file.

[0233] Optionally, the processor 705 is specifically configured to:

[0234] The target video clip is scanned and recorded by screen recording or video recording to obtain a first video file.

[0235] Optionally, the processor 705 is further configured to:

[0236] Sorting the video information according to the play order of the video associated with the target video clip to obtain a video list corresponding to the target video clip;

[0237] The video information includes at least one of the following:

[0238] The total number of movies and TV programs associated with the target video clip, the names of the movies and TV programs, and the time location of the video content of the target video clip in the corresponding movies and TV programs.

[0239] It should be noted that the electronic device provided in the embodiment of the present application is a device capable of executing the above-mentioned video retrieval method, and all implementation methods in the above-mentioned video retrieval method embodiment are applicable to the electronic device and can achieve the same or similar beneficial effects. To avoid repeated description, this embodiment will not be described in detail.

[0240] An embodiment of the present application also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, the various processes of the above-mentioned video retrieval method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0241] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned video retrieval method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0242] The embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned video retrieval method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0243] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0244] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0245] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A video retrieval method, characterized in that: include: Perform emotion recognition on the user to determine the target video clips that the user is interested in among the videos to be retrieved that the user is watching; Scanning and recording the target video clip to obtain a first video file, and streamlining the first video file to obtain a second video file; Performing feature analysis on the second video file to obtain video feature information; The video feature information is matched with preset information in a video library, and film and television information corresponding to the target video segment is obtained based on the matching result.

2. The method according to claim 1, characterized in that: The step of performing emotion recognition on the user and determining a target video segment that the user is interested in among the videos to be retrieved and watched by the user includes: When a user watches a video to be retrieved, recording a facial video of the user; Using a facial emotion recognition model to perform emotion recognition on the facial video to obtain emotion information of the user watching the video to be retrieved; When the calm element value indicated by the emotion information of the first segment is lower than the first value, the first segment is determined to be a target video segment of interest to the user, the first segment is a video segment in the video to be retrieved, and the first value is an average of multiple emotion element values ​​indicated by the emotion information of the first segment, and the multiple emotion element values ​​include the calm element value.

3. The method according to claim 1, characterized in that The streamlining of the first video file to obtain the second video file includes: Determine the stillness corresponding to each frame in the first video file by using a stillness model; Determine a micro-static segment according to the stillness corresponding to each frame in the first video file, wherein the micro-static segment is a video segment in the first video file whose stillness is lower than a first threshold; According to the stillness corresponding to each frame in the micro-still segment, the micro-still segment in the first video file is subjected to frame extraction processing to obtain a second video file.

4. The method according to claim 3, characterized in that The stillness model includes a lens picture module and a speech recognition module, and the stillness corresponding to each frame picture in the first video file is determined by using the stillness model, including: Determine a color histogram corresponding to each frame in the first video file by using the lens picture module; Determine the timbre characteristics corresponding to each frame of the first video file by using the speech recognition module; According to the color histogram and timbre characteristics of the i-1th frame and the i-th frame in the first video file, respectively determine the color histogram similarity and timbre difference between the i-1th frame and the i-th frame; The stillness of the i-th frame is determined according to the color histogram similarity and timbre difference between the i-1-th frame and the i-th frame, wherein i is an integer greater than 1.

5. The method according to claim 3, characterized in that: The step of performing frame extraction processing on the micro-static segment in the first video file according to the stillness corresponding to each frame in the micro-static segment to obtain the second video file includes: Determine the number of frame extractions according to the stillness corresponding to each frame in the micro-still segment and the second threshold, wherein the number of frame extractions is the product of the sum of the number of frames whose stillness exceeds the second threshold and a second value, and the second value is used to adjust the frame extraction amplitude; According to the frame extraction number, frame extraction is performed on the micro-static segment in the first video file to obtain a second video file.

6. The method according to claim 1, characterized in that The step of scanning and recording the target video segment to obtain a first video file includes: The target video clip is scanned and recorded by screen recording or video recording to obtain a first video file.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Sorting the video information according to the play order of the video associated with the target video clip to obtain a video list corresponding to the target video clip; The video information includes at least one of the following: The total number of movies and TV programs associated with the target video clip, the names of the movies and TV programs, and the time location of the video content of the target video clip in the corresponding movies and TV programs.

8. A video retrieval device, characterized in that: include: A determination module is used to perform emotion recognition on the user and determine a target video segment that the user is interested in among the videos to be retrieved that the user is watching; A first processing module, configured to scan and record the target video clip to obtain a first video file, and to simplify the first video file to obtain a second video file; An analysis module, used for performing feature analysis on the second video file to obtain video feature information; The second processing module is used to match the video feature information with the preset information in the video library, and obtain the film and television information corresponding to the target video segment based on the matching result.

9. An electronic device, characterized in that: The invention comprises a transceiver and a processor, wherein the processor is configured to: Perform emotion recognition on the user to determine the target video clips that the user is interested in among the videos to be retrieved that the user is watching; Scanning and recording the target video clip to obtain a first video file, and streamlining the first video file to obtain a second video file; Performing feature analysis on the second video file to obtain video feature information; The video feature information is matched with preset information in a video library, and film and television information corresponding to the target video segment is obtained based on the matching result.

10. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the video retrieval method according to any one of claims 1 to 7 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video retrieval method according to any one of claims 1 to 7 are implemented.

12. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the video retrieval method according to any one of claims 1 to 7.

Images