Video search method, device, computer equipment and storage medium

Abstract

The present application relates to a video search method, device, computer equipment and storage medium. The method includes: obtaining video search information, performing video search based on the video search information to obtain a search video; obtaining a candidate display video frame set from the search video, the candidate display video frame set including multiple candidate display video frames; obtaining the information relevance between the candidate display video frame and the video search information as the candidate information relevance; selecting a target display video frame related to the video search information from the candidate display video frame set based on the candidate information relevance; sending a video search result, the video search result including the target display video frame. The use of this method can improve the effectiveness of video search results.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video search method, device, computer equipment and storage medium. Background Art

[0002] With the development of computer technology and multimedia technology, people have an increasing demand for multimedia information. Video, as a form of multimedia information, has gradually become an important way for people to obtain information in their daily lives. For example, people can obtain the latest news or popular information through short videos.

[0003] Currently, people can search for videos in video playback software, and the video playback software displays video search results of multiple videos that have been searched. People can select a desired video from the displayed multiple videos to play according to the displayed video search results. However, it is often necessary to select the searched videos multiple times before finding the desired video, that is, the effectiveness of the displayed video search results is low. Summary of the invention

[0004] Based on this, it is necessary to provide a video search method, device, computer equipment and storage medium that can improve the effectiveness of video search results in response to the above technical problems.

[0005] A video search method, the method comprising: obtaining video search information, performing video search based on the video search information, and obtaining a search video; obtaining a candidate display video frame set from the search video, the candidate display video frame set including multiple candidate display video frames; obtaining information relevance between the candidate display video frames and the video search information as candidate information relevance; selecting a target display video frame related to the video search information from the candidate display video frame set based on the candidate information relevance; and sending a video search result, the video search result including the target display video frame.

[0006] A video search device, the device comprising: a search video obtaining module, used to obtain video search information, perform video search based on the video search information, and obtain a search video; a candidate display video frame set obtaining module, used to obtain a candidate display video frame set from the search video, the candidate display video frame set including multiple candidate display video frames; a candidate information relevance obtaining module, used to obtain information relevance between the candidate display video frame and the video search information as the candidate information relevance; a target display video frame obtaining module, used to select a target display video frame related to the video search information from the candidate display video frame set based on the candidate information relevance; a video search result sending module, used to send the video search result, the video search result including the target display video frame.

[0007] In some embodiments, the target display video frame obtaining module includes: an original display video frame obtaining unit, used to obtain the original display video frame corresponding to the search video; an original information relevance obtaining unit, used to obtain the information relevance between the original display video frame and the video search information as the original information relevance; a first target display video frame obtaining unit, used to determine the relative difference value of the candidate information relevance relative to the original information relevance, select from the candidate display video frame set the candidate display video frame whose relative difference value is greater than the difference threshold, and use at least one of the candidate display video frames whose relative difference value is greater than the difference threshold as the target display video frame related to the video search information.

[0008] In some embodiments, the original information relevance acquisition unit is also used to acquire the feature correlation between the original display video frame and the video search information as the original feature correlation; acquire the video interactivity corresponding to the original display video frame, the video interactivity being the video interactivity of the search video when the original display video frame is displayed as the video search result of the search video; obtain the original information relevance between the original display video frame and the video search information based on the video interactivity and the original feature relevance; the original information relevance is positively correlated with the video interactivity and the original feature relevance.

[0009] In some embodiments, the original information relevance acquisition unit is also used to obtain the video play possibility corresponding to the search video when the original display video frame is displayed as the video search result of the search video; obtain the video play completion corresponding to the search video when the original display video frame is displayed as the video search result of the search video; obtain the video interactivity corresponding to the original display video frame based on the video play possibility and the video play completion; the video interactivity is positively correlated with the video play possibility and the video play completion.

[0010] In some embodiments, the candidate information relevance obtaining module includes: a frame feature correlation obtaining unit, used to obtain the feature correlation between the candidate display video frame and the video search information as the frame feature correlation; a segment feature correlation obtaining unit, used to obtain the feature correlation between the video segment and the video search information as the segment feature correlation, wherein the candidate display video frame is obtained from the video segment, and the video segment is obtained by segmenting the search video; the candidate information relevance obtaining unit is used to obtain the information correlation between the candidate display video frame and the video search information based on the frame feature correlation and the segment feature correlation as the candidate information correlation, and the candidate information correlation is positively correlated with the frame feature correlation and the segment feature correlation.

[0011] In some embodiments, the module for obtaining the candidate display video frame set includes: a video segment set obtaining unit, which is used to obtain a video segment set obtained by segmenting the search video frame, wherein the video segment set includes multiple video segments; a key frame detection result obtaining unit, which is used to extract features of each video frame in the video frame sequence corresponding to the video segment to obtain a video frame feature sequence, and obtain key frame detection results corresponding to each video frame in the video frame sequence based on the video frame feature sequence; and a candidate display video frame obtaining unit, which is used to extract key frames corresponding to the video segment from the video frame sequence based on the key frame detection results corresponding to each video frame in the video frame sequence, as candidate display video frames in the candidate display video frame set.

[0012] In some embodiments, there are multiple search videos, and the target display video frame obtaining module includes: a display frame set selection composition unit, which is used to select candidate display video frames related to the video search information from the candidate display video frame set based on the candidate information relevance to form a selected display video frame set corresponding to the search video; a second target display video frame obtaining unit, which is used to select target display video frames corresponding to each search video from the selected display video frame sets corresponding to each of the search videos; wherein the video frame difference between the target display video frames corresponding to each of the search videos is greater than a difference threshold.

[0013] In some embodiments, the second target display video frame obtaining unit is also used to determine the search video of the target display video frame to be selected as the current video; obtain the target display video frames corresponding to each comparison video to form a comparison video frame set, and the comparison video is the search video of the determined target display video frame; from the selected display video frame set corresponding to the current video, select a video frame whose video frame difference with the target display video frame in the comparison video frame set is greater than a difference threshold, and use the video frame greater than the difference threshold as the target display video frame corresponding to the current video.

[0014] In some embodiments, the second target display video frame obtaining unit is further used to obtain the current display video frame from the selected display video frame set corresponding to the current video in descending order of candidate information relevance; obtain the current video frame difference between the current display video frame and the target display video frame in the comparison video frame set; when the current video frame difference corresponding to each target display video frame in the comparison video frame set is greater than a difference threshold, the current display video frame is used as the target display video frame corresponding to the current video, otherwise, the step of obtaining the current display video frame from the selected display video frame set corresponding to the current video in descending order of candidate information relevance is returned.

[0015] In some embodiments, the second target display video frame obtaining unit is also used to include: determining the search result ranking corresponding to each of the search videos; and determining the search video of the target display video frame to be selected from the multiple search videos obtained by the search according to the search result ranking, as the current video.

[0016] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program: obtaining video search information, performing a video search based on the video search information, and obtaining a search video; obtaining a candidate display video frame set from the search video, wherein the candidate display video frame set includes multiple candidate display video frames; obtaining information relevance between the candidate display video frames and the video search information as candidate information relevance; selecting a target display video frame related to the video search information from the candidate display video frame set based on the candidate information relevance; and sending a video search result, wherein the video search result includes the target display video frame.

[0017] A computer-readable storage medium stores a computer program, which implements the following steps when executed by a processor: obtaining video search information, performing a video search based on the video search information, and obtaining a search video; obtaining a candidate display video frame set from the search video, wherein the candidate display video frame set includes multiple candidate display video frames; obtaining information relevance between the candidate display video frames and the video search information as candidate information relevance; selecting a target display video frame related to the video search information from the candidate display video frame set based on the candidate information relevance; and sending a video search result, wherein the video search result includes the target display video frame.

[0018] The above-mentioned video search method, device, computer equipment and storage medium obtain video search information, perform video search based on the video search information to obtain the search video, obtain a candidate display video frame set from the search video, the candidate display video frame set includes multiple candidate display video frames, obtain the information relevance between the candidate display video frame and the video search information as the candidate information relevance, select the target display video frame related to the video search information from the candidate display video frame set based on the candidate information relevance, send the video search result, the video search result includes the target display video frame, so that the video frame with a greater relevance to the video search information in the searched video is returned to the terminal, thereby improving the relevance of the video search result and the video search information, thereby improving the effectiveness of the video search result.

[0019] A video search method, the method comprising: displaying a search information input area; receiving video search information through the search information input area; in response to a search operation on the search information input area, triggering a video search based on the video search information; displaying video search results corresponding to the searched video obtained by the search, the video search results including a target display video frame in the search video related to the video search information, the target display video frame being displayed as a video display frame in the video search results.

[0020] A video search device, the device comprising: a search information input area display module, used to display the search information input area; a video search information receiving module, used to receive video search information through the search information input area; a video search trigger module, used to trigger a video search based on the video search information in response to a search operation on the search information input area; a video search result display module, used to display video search results corresponding to the searched video, the video search results including a target display video frame in the search video related to the video search information, the target display video frame being displayed as a video display frame in the video search results.

[0021] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program: displaying a search information input area; receiving video search information through the search information input area; triggering a video search based on the video search information in response to a search operation on the search information input area; and displaying video search results corresponding to the searched video obtained by the search, wherein the video search results include a target display video frame in the search video that is related to the video search information, and the target display video frame is displayed as a video display frame in the video search results.

[0022] A computer-readable storage medium stores a computer program, which implements the following steps when executed by a processor: displaying a search information input area; receiving video search information through the search information input area; triggering a video search based on the video search information in response to a search operation on the search information input area; and displaying video search results corresponding to the searched video obtained by the search, wherein the video search results include a target display video frame in the search video that is related to the video search information, and the target display video frame is displayed as a video display frame in the video search results.

[0023] In some embodiments, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0024] The above-mentioned video search method, apparatus, computer equipment and storage medium display a search information input area, receive video search information through the search information input area, trigger a video search based on the video search information in response to a search operation on the search input area, and display video search results corresponding to the searched video obtained by the search. The video search results include target display video frames related to the video search information in the search video. The target display video frames are displayed as video display frames in the video search results, thereby improving the relevance of the video search results to the video search information and improving the effectiveness of the video search results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A diagram of an application environment of a video search method in some embodiments;

[0026] Figure 2 is a flowchart of a video search method in some embodiments;

[0027] Figure 3 is a structural diagram of a video frame correlation detection model in some embodiments;

[0028] Figure 4 is a structural diagram of a segment relevance detection model in some embodiments;

[0029] Figure 5 is a schematic diagram of a video search interface in some embodiments;

[0030] Figure 6 is a schematic diagram of a video search interface in some embodiments;

[0031] Figure 7 is a flowchart of a video search method in some embodiments;

[0032] Figure 8 is a schematic diagram of a video search method in some embodiments;

[0033] Fig. 9 is a structural block diagram of a video search device in some embodiments;

[0034] Fig.10 is a structural block diagram of a video search device in some embodiments;

[0035] Fig.11is an internal structure diagram of a computer device in some embodiments;

[0036] Fig.12 1 is a diagram of the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0038] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0039] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0040] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0041] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0042] The key technologies of speech technology include automatic speech recognition technology, speech synthesis technology and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the development direction of human-computer interaction in the future, among which speech has become one of the most promising human-computer interaction methods in the future.

[0043] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0044] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0045] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0046] The solution provided in the embodiments of the present application involves artificial intelligence speech technology, image processing, machine learning and other technologies, which are specifically described through the following embodiments:

[0047] The video search method provided by this application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. Video playback software can be installed in the terminal 102, and the server 104 can be a server corresponding to the video playback software. The video playback software can be, for example, a video software that plays short videos. The terminal 102 can display the user interface corresponding to the video playback software, receive the search information input or selected by the user through the user interface, and when the video search instruction is received through the user interface, send a video search request carrying the search information to the server 104. The server 104 can respond to the video search request, obtain the video corresponding to the search information, and return the video corresponding to the search information to the terminal 102. The terminal 102 can display the video returned by the server 104 on the user interface corresponding to the video playback software. The server 104 can also be a server corresponding to a video website. The terminal 102 can access the video website, receive the search information through the web page corresponding to the video website, and when the video search instruction is received through the web page, send a video search request to the server 104, and display the video requested by the video search request on the web page. The video website can also be called a video site. The video site can support a search function, and the user can search for the video content that he intends to watch through the search function of the video site.

[0048] Specifically, the terminal 102 may display a search information input area in the interface corresponding to the video playback software or the video website, receive the video search information through the search information input area, and in response to the search operation on the search input area, trigger a video search based on the video search information, and send a video search request carrying the video search information to the server. The server 104 may respond to the video search request, obtain the video search information from the video search request, perform a video search based on the video search information, obtain the search video, obtain a candidate display video frame set from the search video, the candidate display video frame set includes multiple candidate display video frames, obtain the information relevance between the candidate display video frame and the video search information as the candidate information relevance, select the target display video frame related to the video search information from the candidate display video frame set based on the candidate information relevance, and send the video search result to the terminal 102, the video search result includes the target display video frame. The terminal 102 may display the video search result, that is, may display the target display video frame corresponding to each search video, for example, may display the target display video frame as a video cover image.

[0049] Among them, the terminal 102 can be, but is not limited to, a laptop computer, a smart phone, a tablet computer, a desktop computer, a smart TV, a smart speaker, a smart watch, a car computer, and a portable wearable device, etc., but is not limited to this. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0050] It can be understood that the above application scenario is only an example and does not constitute a limitation on the video search provided by the embodiment of the present application. The method provided by the embodiment of the present application can also be applied in other application scenarios. For example, the video search provided by the present application can be performed by terminal 102, and terminal 102 can upload the obtained video search results to server 104. Server 104 can store the video search results and forward the video search results to other terminal devices.

[0051] In some embodiments, Figure 2 As shown, a video search method is provided, which can be executed by a terminal or a server, or by both the terminal and the server. Specifically, the method is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0052] S202, obtaining video search information, performing video search based on the video search information, and obtaining the search video.

[0053] The video search information is information used to search for videos. The searched videos are videos searched using the video search information. The searched videos may be one or more, and more means at least two. The video search information may also be referred to as "user current query".

[0054] Specifically, the terminal may display a video search interface, receive video search information for selection or input through the video search interface, and when the terminal receives a video search operation, send a video search request carrying the video search information to the server. The server may respond to the video search request, extract the video search information from the video search request, and search for a video matching the video search information from a candidate video set as the search video. The candidate video set may be pre-stored in the server, or may be obtained by the server from other devices. The candidate video set includes multiple candidate videos.

[0055] In some embodiments, each candidate video in the candidate video set may correspond to a video tag, and the server may compare the video search information with the video tag, and use the video with the consistent comparison as the search video. The video tag may include at least one of the subject of the video, the scene to which the video belongs, or an object in the video, and the object may be a person or an animal.

[0056] In some embodiments, the server can obtain the original display video frames corresponding to each search video, and return the original display video frames corresponding to each search video to the terminal, and the terminal can display the original display video frames corresponding to each search video. Among them, the display video frame is a video frame related to the video, and the display video frame can be used to reflect the content of the video, and can be used to introduce the search video, so it can be used to be displayed in the scene of the introduction video, such as as a cover picture of the video. The display video frame can be, for example, an image related to the title, theme, scene or key figure of the search video. The display video frame can be a video image extracted from the search video, for example, it can be a video image extracted from the search video based on the video search information, for example, a video frame whose correlation with the video search information is greater than a correlation threshold can be selected from the search video as a display video frame. The correlation threshold can also be preset or set as needed. The original display video frame refers to the display video frame used at the current time of the search video. The display video frame corresponding to the search video can be continuously updated. For example, the display video frame of the search video can be updated over time, and the display video frames used at different times can be the same or different, or the display video frame corresponding to the search video can be determined based on the video search information, that is, the display video frame can be updated along with the video search information.

[0057] In some embodiments, the terminal may display the displayed video frame as the cover image of the searched video. The cover image of the video is used to trigger the playback of the corresponding video. For example, when the terminal obtains a click operation on the cover image, the terminal may respond to the click operation, obtain the video corresponding to the video frame from the server and play the video corresponding to the cover image. The cover image of the video is a display element of the video, and the content of the video can be intuitively understood through the video cover image.

[0058] S204: Acquire a candidate display video frame set from the search video, where the candidate display video frame set includes a plurality of candidate display video frames.

[0059] The candidate display video frame set includes multiple candidate display video frames, and the candidate display video frame set refers to a set of video frames to be selected to match the video search information. For example, at least one video frame in the candidate display video frame set can be used as a video frame matching the video search information, and all video frames in the candidate display video frame set can be used as video frames matching the video search information. Of course, the correlation between the candidate display video frame and the video search information can also be calculated, and the video frame matching the video search information can be selected from the candidate display video frame set according to the calculated correlation. The candidate display video frame can be a video image extracted from the search video, for example, the candidate display video frame can include a key frame of the video. The candidate display video frame set can include the original display video frame corresponding to the search video, and of course, it can also exclude the original display video frame.

[0060] A key frame refers to a critical video frame in a search video, which is used to reflect the key information of the search video. For example, it can be a video frame related to at least one of the title, theme, scene or key characters of the search video. The search video can include a key frame identifier corresponding to the key frame, and the key frame identifier is used to indicate that the video frame is a key frame. Of course, the search video may not include a key frame identifier. The key frame in the search video can be obtained by detecting the search video, for example, the key frame in the search video can be detected by using a key frame detection network. The key frame detection network is used to detect key frames in a video.

[0061] Specifically, the server may extract video frames from the search video according to the video frame interval, and use the extracted video frames as candidate display video frames in the candidate display video frame set. The video frame interval may be preset or set as required, for example, 10 frames.

[0062] In some embodiments, the server can segment the search video, organize the segmented video segments into a video segment set, extract one or more video frames from each video segment, and use the video frames extracted from each video segment as candidate display video frames. For example, key frames can be extracted from the video segments, and the key frames extracted from each video segment are used as candidate display video frames.

[0063] In some embodiments, the server may segment the target video according to any one of the target frame interval or the target time interval to obtain individual video segments. The target frame interval and the target time interval may be preset or set as needed. The target frame interval refers to the number of video frames included in the video segment, and the target time interval refers to the duration occupied by the video segment. The target frame interval may be, for example, 10 frames, and the target time interval may be, for example, 1 second. The target time interval may also be referred to as the time interval length. For example, the search video may be segmented based on the time interval length t to obtain multiple video segments of duration t to form a video segment set.

[0064] In some embodiments, the server may obtain a trained key frame detection network, use the trained key frame detection network to perform key frame detection on the video clip, obtain key frames in the video clip, and form a key frame sequence. The key frame detection network is used to determine key frames from the video clip. The server may input the video frame sequence corresponding to the video clip into a video key frame sequence annotation model, and output the key frame sequence using the video key frame sequence annotation model. The video frame sequence is a sequence obtained by arranging the video frames in the video clip from front to back according to the playback time. Among them, the key frame sequence includes each key frame determined from the video clip, and each key frame in the key frame sequence can be arranged according to the order in the video frame sequence. The key frame detection network can be, for example, a video key frame sequence annotation model. The video key frame sequence annotation model is used to annotate each video frame in the video frame sequence corresponding to the video clip, thereby obtaining the annotation information corresponding to each video frame, determining whether the video frame is a key frame according to the annotation information, composing each key frame into a key frame sequence, and outputting the key frame sequence corresponding to the video clip. Among them, the annotation information includes positive annotation information for characterizing that the video frame is a key frame, and may also include negative annotation information for characterizing that the video frame is a non-key frame. The positive annotation information and the negative annotation information can be preset or set as needed. The positive annotation information can be 1, for example, and the negative annotation information can be 0, for example, that is, 0 indicates that the video frame is a non-key frame, and 1 indicates that the video frame is a key frame.

[0065] In some embodiments, the server can obtain a key frame detection network to be trained, obtain a training video clip, and obtain standard annotation information corresponding to each video frame in the training video clip. The standard annotation information is the correct annotation information corresponding to the video frame, that is, when the video frame is a key frame, the standard annotation information is positive annotation information, and when the video frame is not a key frame, the standard annotation information is negative annotation information. The video frame sequence corresponding to the training video clip is input into the key frame detection network. The key frame detection network annotates each video frame in the training video frame sequence to obtain the predicted annotation information corresponding to each video frame. Based on the annotation information difference between the standard annotation information and the predicted annotation information, the annotation network loss value is obtained. The annotation information difference refers to the difference between the standard annotation information and the predicted annotation information. The annotation network loss value is positively correlated with the annotation information difference. The server can adjust the network parameters of the key frame detection network in the direction of reducing the annotation network loss value until the network convergence condition is met, and the key frame detection network that meets the network convergence condition is used as the trained key frame detection network.

[0066] Among them, positive correlation means that when other conditions remain unchanged, the two variables change in the same direction, and when one variable changes from large to small, the other variable also changes from large to small. It can be understood that the positive correlation here means that the direction of change is consistent, but it does not require that when one variable changes a little, the other variable must also change. For example, when variable a is 10 to 20, variable b can be set to 100, and when variable a is 20 to 30, variable b can be set to 120. In this way, the direction of change of a and b is that when a increases, b also increases. But when a is in the range of 10 to 20, b can be unchanged.

[0067] S206: Obtain information relevance between the candidate display video frame and the video search information as the candidate information relevance.

[0068] The information relevance refers to the relevance between the video frame and the video search information, and the candidate information relevance refers to the relevance between the candidate display video frame and the video search information. The greater the candidate information relevance, the more the candidate display video frame matches the video search information.

[0069] Specifically, the server can extract image features from the candidate display video frame, and use the extracted image features as the candidate video frame features corresponding to the candidate display video frame. It can extract text features from the video search information, and use the extracted text features as the search information features. The correlation between the candidate video frame features and the search information features is calculated, and the calculated correlation is used as the frame feature correlation corresponding to the candidate display video frame. The candidate information correlation is obtained based on the frame feature correlation. The candidate information correlation is positively correlated with the frame feature correlation. For example, the frame feature correlation can be used as the candidate information correlation, or the frame feature correlation can be adjusted, and the result obtained after the adjustment is used as the candidate information correlation. Among them, the cosine similarity calculation formula can be used to calculate the correlation. For example, the cosine similarity between the candidate video frame features and the search information features can be calculated, and the cosine similarity can be used as the frame feature correlation corresponding to the candidate display video frame. Of course, the correlation calculation method can also be used in other ways, which are not limited here.

[0070] In some embodiments, the server may use a trained video frame relevance detection model to determine the frame feature relevance between the candidate display video frame and the video search information. Figure 3 As shown, a structural diagram of a video frame relevance detection model is shown. The video frame relevance detection model may include a search information feature extraction network, a video frame feature extraction network, and a frame feature relevance detection unit. The search information feature extraction network is used to extract text features from video search information to obtain search information features corresponding to the video search information. The video frame feature extraction network is used to extract image features from video frames to obtain video frame features. For example, image features are extracted from candidate display video frames to obtain candidate video frame features corresponding to the candidate display video frames. The frame feature relevance detection unit is used to calculate the relevance between video frame features and search information features to obtain frame feature relevance.

[0071] In some embodiments, the server may obtain a video frame relevance detection model to be trained, obtain training search information and training samples, the training samples may include at least one of positive samples or negative samples, the positive samples are video frames that are related to the training search information or have a relevance greater than a relevance threshold, and the negative samples are video frames that are unrelated to the training search information or have a relevance less than the relevance threshold. The server may train the video frame relevance detection model using the training samples and the training search information to obtain a trained video frame relevance detection model.

[0072] In some embodiments, the candidate display video frame is obtained from the video segment obtained by segmenting the search video. The server can determine the video segment corresponding to the candidate display video frame from the segmented video segments, and calculate the correlation between the video segment corresponding to the candidate display video frame and the video search information. For example, the video segment can be feature extracted to obtain video segment features, and the video segment features can be correlated with the search information features to obtain segment feature correlation. The candidate information correlation corresponding to the candidate display video frame is determined based on the segment feature correlation and the frame feature correlation. The segment feature correlation can be calculated using the cosine similarity calculation formula, or the segment feature correlation can be calculated using the attention mechanism. For example, the segment feature correlation can be calculated using formula (1), where Q represents the video segment feature, K represents the search information feature, and V represents the search information feature. Attention(Q,K,V) represents the segment feature correlation. d k is the dimension of the feature.

[0073]

[0074] In some embodiments, the video clip feature may include at least one of a text content feature, an image content feature, or an audio content feature. The text content feature is a feature corresponding to the text content in the video clip, the image content feature is a feature corresponding to the image content in the video clip, and the audio content feature is a feature corresponding to the audio content in the video clip. The text content may include various text data in the video clip, for example, it may include at least one of image text, audio text, or bullet text. Image text refers to text data extracted from an image. For example, the server may use a trained image text detection model to detect the text data contained in the image of the video clip to obtain image text. The image text detection model may be, for example, a trained OCR (Optical Character Recognition) model. The OCR model may be used to recognize text in an image, such as an ID number, name, address, and bank card number on an ID card. The image text may also be referred to as OCR text. Audio text refers to text data obtained by performing speech recognition on audio data in a video. For example, the automatic speech recognition technology (ASR, Automatic Speech Recognition) may be used to recognize audio data in a video clip to obtain audio text. Audio data may also be referred to as speech data. For example, the voice data in the video clip can be input into a trained speech recognition model for speech recognition to obtain the audio text corresponding to the voice data. Among them, automatic speech recognition technology is a technology for converting speech into text. The speech recognition model is used to recognize the voice data and obtain text data. The server can extract the bullet screen in the video clip to obtain the bullet screen text. The image content can include each video frame in the video clip, and the audio content can include each audio frame included in the audio data in the video clip. The audio text can also be called ASR text.

[0075] In some embodiments, the server may extract at least one of text content, image content or audio content from the video clip. The server may perform text feature extraction on the text content to obtain text content features, perform image feature extraction on the image content to obtain image content features, perform audio feature extraction on the audio content to obtain audio content features, and obtain video clip features corresponding to the video clip based on at least one of the text content features, image content features or audio content features. For example, any one or more of the text content features, image content features and audio content features may be used as video clip features, or feature fusion may be performed on the text content features, image content features and audio content features, and the fused features may be used as video clip features. Fusion may be, for example, a multiplication operation or a splicing process, and splicing means connecting features together in sequence.

[0076] In some embodiments, the server can use a trained text feature extraction network to extract text features. For example, the server can use a trained text feature extraction network to extract features from video search information to obtain search information features corresponding to the video search information, or use a text feature extraction network to extract text features from text content to obtain text content features corresponding to the text content. The text feature extraction network is used to extract features of text data. Among them, the text feature extraction network can be a neural network model, for example, it can be BERT (Bidirectional Encoder Representations from Transformers).

[0077] In some embodiments, the server can use a trained audio feature extraction network to extract audio features from each audio frame in the audio content, and obtain audio frame features corresponding to each audio frame. The server can obtain audio content features corresponding to the audio content based on each audio frame feature. The audio feature extraction network can be a neural network model, for example, it can be a VGGish model. The VGGish model is a VGG (Visual Geometry Group) model based on tensorflow. The VGGish model can extract a semantic and meaningful 128-dimensional high-dimensional feature vector from an audio waveform. Tensorflow is a deep learning framework.

[0078] In some embodiments, the server may obtain each audio frame from the audio content presented in the video clip, extract audio features corresponding to each audio frame, use the extracted features as the audio frame features corresponding to the audio frame, and obtain audio content features corresponding to the audio content based on the features of each audio frame. For example, the features of each audio frame may be fused, and the fused features may be used as the audio features corresponding to the audio content. For example, the server may use a trained feature fusion network to fuse the features of each audio frame to obtain audio content features. The feature fusion network may be a neural network model, such as NeXtVlad (NeXt Vector of Local AggregatedDescriptors), where Vlad in NeXtVlad is the abbreviation of Vector of locally aggregated descriptors, and its Chinese name is Local Aggregate Vector.

[0079] In some embodiments, the server may use a trained image feature extraction network to extract image features. For example, the server may use a trained image feature extraction network to extract features from candidate display video frames to obtain candidate video frame features corresponding to the candidate display video frames. The image feature extraction network is used to extract image features. The image feature extraction network may be any neural network model used to extract image features, such as an efficientNet network.

[0080] In some embodiments, the server may extract image features from each video frame in the video clip, and use the extracted features as video frame features corresponding to the video frame. The server may perform feature fusion on each video frame feature, and use the fused features as image content features corresponding to the image content. Feature fusion may be performed using a feature fusion network. Video frames may also be referred to as image frames, and video frame features may also be referred to as image frame features.

[0081] In some embodiments, the server may calculate the segment feature correlation using a trained segment correlation detection model. The segment correlation detection model may determine the segment feature correlation between the video search information and the video segment based on the video search information and the video segment. The server inputs the video search information and the video segment into the segment correlation detection model to perform correlation calculation to obtain the segment feature correlation between the video search information and the video segment. The segment correlation detection model may include at least one of a text feature extraction network, an audio feature extraction network, an image feature extraction network, or a feature fusion network. The text feature extraction network, the audio feature extraction network, the image feature extraction network, and the feature fusion network in the segment correlation detection model may be obtained through joint training.

[0082] like Figure 4As shown, a structural diagram of a segment relevance detection model for calculating segment feature relevance is shown. The segment relevance detection model includes a first text feature extraction network, a second text feature extraction network, an audio feature extraction network, an image feature extraction network, an audio feature fusion network, an image feature fusion network, a multi-dimensional feature fusion unit, and a segment feature relevance detection unit. Among them, the first text feature extraction network is used to extract text features of video search information, and the second text feature extraction network is used to extract text features of text content extracted from video segments. For example, feature extraction can be performed on image text, audio text, and bullet text respectively to obtain image text features corresponding to image text, audio text features corresponding to audio text, and bullet text features corresponding to bullet text. Based on at least one of the image text features, audio text features, or bullet text features, text content features corresponding to the text content are obtained. The audio feature extraction network is used to extract audio features of each audio frame included in the audio content extracted from the video segment to obtain audio frame features corresponding to each audio frame. The image feature extraction network is used to extract image features of each image frame included in the image content extracted from the video segment to obtain image frame features corresponding to each image frame. The image content may include, for example, image frames 1 to image frame I in video clip A, and the audio content may include, for example, audio frame 1 to audio frame J in video clip A. Wherein, I and J are positive integers greater than or equal to 1. The audio feature fusion network is used to fuse the audio frame features corresponding to each audio frame included in the audio content, and obtain the audio content features corresponding to the audio content. The image feature fusion network is used to fuse the image frame features corresponding to each image frame included in the image content, and obtain the image content features corresponding to the image content. The multi-dimensional feature fusion unit is used to fuse two or more features of text content features, audio content features or image content features, and obtain the video clip features corresponding to the video clip. The clip feature relevance detection unit is used to detect the relevance between the video clip features and the search information features, and obtain the clip feature relevance.

[0083] In some embodiments, each network of the segment relevance detection model may be obtained through joint training. For example, the server may obtain a training video segment and training search information, and use the training video segment and training search information to train each network in the segment relevance detection model to be trained, so as to obtain a trained segment relevance detection model. There are multiple training video segments. The training search information may be one or more. The server may input the training video segment and the training search information into the segment relevance detection model, obtain the predicted segment relevance between the training video segment output by the segment relevance detection model and the training search information, obtain the real relevance between the training video segment and the training search information, determine the model loss value based on the relevance difference between the predicted segment relevance and the real relevance, and the model loss value is positively correlated with the relevance difference. The server may adjust the model parameters of the segment relevance detection model in the direction of decreasing the model loss value until the model convergence condition is met, and use the segment relevance detection model that meets the model convergence condition as the trained segment relevance detection model.

[0084] S208: Select a target display video frame related to the video search information from the candidate display video frame set based on the candidate information relevance.

[0085] The target display video frame is at least one of the video frames in the candidate display video frame set that is related to the video search information, for example, it may be a candidate display video frame in the candidate display video frame set that has the greatest candidate information relevance to the video search information.

[0086] Specifically, the server may select a candidate display video frame whose candidate information relevance is greater than an information relevance threshold from the candidate display video frame set, and use at least one of the candidate display video frames whose candidate information relevance is greater than the information relevance threshold as a target display video frame. The information relevance threshold may be preset or set as required, for example, may be determined based on the relevance between the original display video frame corresponding to the search video and the video search information.

[0087] In some embodiments, the server may first screen the candidate display video frames in the candidate display video frame set according to the candidate information relevance, for example, a first relevance threshold may be obtained, and the candidate display video frames whose candidate information relevance is greater than the first relevance threshold are used as screened display video frames, and at least one of the screened display video frames is used as a target display video frame. For example, the information relevance threshold may be determined according to the relevance between the original display video frame and the video search information, and at least one of the screened display video frames whose candidate information relevance is greater than the information relevance threshold is used as a target display video frame. The first relevance threshold is different from the information relevance threshold.

[0088] In some embodiments, the server may arrange the candidate display video frames in the candidate display video frame set in descending order of candidate information relevance to obtain a candidate display video frame sequence, and use at least one of the video frames in the candidate display video frame sequence whose video frame ranking is before the ranking threshold as the target display video frame. The greater the candidate information relevance, the higher the ranking of the candidate display video frame in the candidate display video frame sequence. Video frame ranking refers to the ranking of the candidate display video frames in the candidate display video frame sequence, and the ranking threshold can be preset or set as needed.

[0089] In some embodiments, the server searches for multiple search videos based on the video search information, and the server can obtain a set of candidate display video frames corresponding to each search video, and selects a target display video frame corresponding to each search video from the set of candidate display video frames corresponding to each search video. The similarity between the target display video frames corresponding to each search video can be less than a similarity threshold, and the similarity can be preset or set as needed.

[0090] S210, sending video search results, where the video search results include a target display video frame.

[0091] The video search results may include a target display video frame corresponding to the search video, and may also include a video identifier of the search video, where the video identifier is used to uniquely identify the search video. The video identifier may be, for example, a video name. Each search video corresponds to a video search result. The target display video frame corresponding to the search video may be one or more, and the video search results may include one or more target display video frames. Multiple means at least two.

[0092] Specifically, the server can generate a video search result corresponding to the search video based on the target display video frame corresponding to the search video, and return the video search result to the terminal corresponding to the video search information. The terminal receives the video search result returned by the server, obtains the target display video frame corresponding to the search video from the video search result, and the terminal can display the target display video frame. For example, the terminal can display a search result display area in the video search interface, and the search result display area is used to display the target display video frame in the video search result. Among them, the terminal corresponding to the video search information refers to the terminal that sends the video search information to the server, for example, it can be a terminal that sends a video search request carrying the video search information to the server, for example, it can be Figure 1 Terminal 102 in.

[0093] In some embodiments, the terminal may display the target display video frame as the cover image of the search video. Figure 5As shown, a video search interface 502 is displayed, and a search information input area 504, a search confirmation control 506 and a search result display area 508 are displayed on the video search interface 502. When the terminal obtains a trigger operation on the search confirmation control 506, the video search information "abc video" in the search information input area 504 can be sent to the server. The server searches for "abc video" and obtains two videos, and the names of the videos are "abc video highlights" and "abc video introduction", and the server determines that the target display video frame of "abc video highlights" is picture A, and the target display video frame of "abc video introduction" is picture B. The server returns the target display video frames of the two videos to the terminal, and the terminal displays the target display video frames as cover images of the videos in the search result display area 508, that is, picture A is displayed as the cover image of "abc video highlights", and picture B is displayed as the cover image of "abc video introduction".

[0094] In some embodiments, the terminal may display the target display video frame as preview information. When the terminal obtains the preview information viewing operation corresponding to the search video, the terminal may display the target display video frame corresponding to the search video. The preview information viewing operation is used to trigger the display of preview information. The preview information may include one or more target display video frames corresponding to the search video. For example, the video search results may include the video name, the target display video frame corresponding to the search video, and the cover image corresponding to the search video. The terminal may display the cover image corresponding to each search video. When the terminal obtains the preview information viewing operation corresponding to the search video, the preview information viewing operation may be, for example, a focus operation on the cover image of the search video. For example, when the mouse is located on the cover image, it is determined that the preview information viewing operation is obtained, and the target display video frame is displayed in the preview information display area of ​​the cover image. The preview information display area is used to display the preview information. The position of the preview information display area may be set as needed or may be preset, for example, it may be an area located above the cover image. Figure 6 As shown, a video search interface 602 is displayed, and the terminal displays the cover image of the searched video "abc video introduction" and the cover image of the video "abc video highlights" in the search result display area 604. Picture A1, Picture A2 and Picture A3 are the target display video frames of the video "abc video introduction". When the terminal detects that the mouse is located on the cover image of "abc video introduction", the target display video frame corresponding to the video "abc video introduction" is displayed in the preview information display area 606 corresponding to the video "abc video introduction", that is, Picture A1, Picture A2 and Picture A3 are displayed.

[0095] In the above-mentioned video search method, video search information is obtained, video search is performed based on the video search information to obtain a search video, a candidate display video frame set is obtained from the search video, the candidate display video frame set includes multiple candidate display video frames, the information relevance between the candidate display video frame and the video search information is obtained as the candidate information relevance, and a target display video frame related to the video search information is selected from the candidate display video frame set based on the candidate information relevance, and the video search result is sent, and the video search result includes the target display video frame, so that the video frames in the searched video with a greater relevance to the video search information are returned to the terminal, thereby improving the relevance of the video search results and the video search information, thereby improving the effectiveness of the video search results.

[0096] Since different users may pay attention to different plot points of the same video, even the same user may pay attention to different plot points of the same video at different time points. If a fixed video image is used as the cover image of the video, that is, the cover image of the video is fixed as a video image, the flexibility of the cover image is reduced, and the user experience is reduced. In the embodiment of the present application, the cover image of the video can be determined according to the user's search information, so that a video image with a high degree of relevance to the search information can be obtained as the video cover image, so that when the video is displayed with the cover image, the user can intuitively understand the content in the video that the user is interested in, which increases the user's intention to click on the video, thereby increasing the video click-through rate.

[0097] In some embodiments, selecting a target display video frame related to video search information from a set of candidate display video frames based on candidate information relevance includes: obtaining an original display video frame corresponding to the search video; obtaining the information relevance between the original display video frame and the video search information as the original information relevance; determining a relative difference value of the candidate information relevance relative to the original information relevance, selecting from the set of candidate display video frames a candidate display video frame having a relative difference value greater than a difference threshold, and using at least one of the candidate display video frames having a relative difference value greater than the difference threshold as a target display video frame related to the video search information.

[0098] The original display video frame refers to the display video frame used at the current time of the search video. The original information relevance refers to the relevance between the original display video frame and the video search information. The relative difference value refers to the difference value between the candidate information relevance and the original information relevance. For example, the candidate information relevance and the original information relevance can be subtracted, and the result of the subtraction is used as the relative difference value. The difference threshold can be preset or set as needed, for example, it can be 0 or 0.1.

[0099] Specifically, the server can extract image features from the original displayed video frame, use the extracted features as the original video frame features, calculate the correlation between the original video frame features and the search information features, use the calculated correlation as the original feature correlation corresponding to the original video frame features, and obtain the original information correlation based on the original features. The original information correlation is positively correlated with the original feature correlation. For example, the original feature correlation can be used as the original information correlation, or the original feature correlation can be adjusted and the adjusted correlation can be used as the original information correlation.

[0100] In some embodiments, the server may obtain the video interactivity corresponding to the original display video frame. The video interactivity is used to reflect the degree of interaction between the user and the search video when the original display video frame is used as the display video frame of the search video for display. The video interactivity is positively correlated with the degree of interaction. Interaction refers to the interactive behavior between the user and the search video. The interactive behavior may include at least one of click, forward, comment or like. The degree of interaction may be represented by the frequency or number of times the interactive behavior occurs. For example, the degree of interaction may be positively correlated with the frequency of the interactive behavior, for example, it may be positively correlated with the click-through rate.

[0101] In some embodiments, the server may use the candidate information relevance minus the original information relevance, and use the result obtained by subtracting the candidate information relevance from the original information relevance as the relative difference value. The server may compare the relative difference value with the difference threshold, and when it is determined that the relative difference value is greater than the difference threshold, use the corresponding candidate display video frame as the target display video frame.

[0102] In this embodiment, at least one of the candidate display video frames whose relative difference value is greater than the difference threshold is used as a target display video frame related to the video search information, so that the correlation between the obtained target display video frame and the video search information can be greater than the correlation between the original display video frame and the video search information, thereby improving the correlation between the obtained target display video frame and the video search information.

[0103] In some embodiments, obtaining the information correlation between the original display video frame and the video search information as the original information correlation includes: obtaining the feature correlation between the original display video frame and the video search information as the original feature correlation; obtaining the video interactivity corresponding to the original display video frame, the video interactivity is the video interactivity of the search video when the original display video frame is displayed as a video search result of the search video; obtaining the original information correlation between the original display video frame and the video search information based on the video interactivity and the original feature correlation; the original information correlation is positively correlated with the video interactivity and the original feature correlation.

[0104] Among them, the original feature relevance refers to the correlation between the features corresponding to the original display video frame and the features corresponding to the video search information. Video interactivity is used to reflect the video interactivity between the user and the search video, and the video interactivity is positively correlated with the video interactivity. Video interactivity refers to the degree of interaction between the user and the search video. When displaying the display video frame corresponding to the search video, for example, when displaying the original display video frame of the search video, the number or frequency of interactive operations triggered by the original display video frame can reflect the video interactivity. The more times or the higher the frequency of interactive operations triggered by the original display video frame, the higher the video interactivity. The original information relevance is positively correlated with the video interactivity, and the original information relevance is positively correlated with the original feature relevance.

[0105] Specifically, the server may extract image features from the original display video frame, use the extracted features as original video frame features, calculate the correlation between the original video frame features and the search information features, and use the calculated results as the original feature correlation.

[0106] In some embodiments, the server may perform at least one of a linear operation or a nonlinear operation on the video interactivity and the original feature correlation, and use the result of the operation as the original information correlation between the original display video frame and the video search information. The linear operation may include at least one of a weighted operation or a multiplication operation. The nonlinear operation may include at least one of a logarithmic operation, an exponential operation, or a square root operation. For example, the video interactivity and the original feature correlation may be weighted, and the result obtained by the weighted calculation may be used as the original information correlation, or the video interactivity and the original feature correlation may be multiplied, and the result obtained by the multiplication may be used as the original information correlation.

[0107] In this embodiment, the original information relevance between the original display video frame and the video search information is obtained based on the video interactivity and the original feature relevance. Since the original information relevance is positively correlated with the video interactivity and the original feature relevance, the original information relevance can reflect the degree of interaction between the user and the search video, and can reflect the degree of correlation between the video search information and the original display video frame, thereby improving the accuracy of the original information relevance.

[0108] In some embodiments, obtaining the video interactivity corresponding to the original display video frame includes: obtaining the video play possibility corresponding to the search video when the original display video frame is displayed as the video search result of the search video; obtaining the video play completion corresponding to the search video when the original display video frame is displayed as the video search result of the search video; obtaining the video interactivity corresponding to the original display video frame based on the video play possibility and the video play completion; the video interactivity is positively correlated with the video play possibility and the video play completion.

[0109] Among them, the video play possibility corresponding to the search video refers to the play possibility of the search video being played by the user. The video play possibility is the video play possibility obtained by statistics when the original display video frame is displayed as the video search result of the search video. Taking the original display video frame as the original cover image as an example, when the search result of the search video is displayed with the original cover image, it can represent the possibility of the user seeing the cover image and clicking the search result of the video to play the search video. The video play possibility can be the possibility of the user playing the search video when the original display video frame corresponding to the search video is displayed in the historical time period. For example, when the original display video frame corresponding to the search video is displayed in the historical time period, the possibility of the user clicking to play the search video can be determined according to the number of users who play the search video. For example, the number of users who display the original display video frame of the search video in the historical time period can be counted, and the number is used as the total number of users. The number of users who play the search video among the users who display the original display video frame of the search video is determined, and the number is used as the number of plays. The ratio of the number of plays to the total number of users is calculated, and the calculated ratio is used as the video play possibility. For example, 100 users search for the search video, and the user's terminal displays the original cover image corresponding to the search video. Among the 100 users, 30 users play the search video after seeing the original cover image. Then, the number of plays is 30, the total number of users is 100, and the video play possibility is 30 / 100=30%. The historical time period is a historical time period, which can be determined as needed.

[0110] The video playback completion represents the ratio of the playback time of the searched video to the total video playback time. It can be obtained by counting the playback time of one or more users. For example, it can be calculated as the ratio between the average user playback time and the total video playback time as the video playback completion. The total video playback time refers to the total time of the searched video, and the user playback time refers to the time the user plays the video. For example, if the searched video is a 10-minute video, the total video playback time is 10 minutes. Assuming that the user only watches 5 minutes of the searched video, the user playback time is 5 minutes, and the video playback completion is 5 / 10=50%. The average user playback time refers to the average playback time of each user. For example, if 500 users play the video, the average playback time of these 500 users is calculated, and the calculated average value is used as the average user playback time. Video interactivity is positively correlated with video playback possibility, and video interactivity is positively correlated with video playback completion.

[0111] Specifically, the server may perform at least one of linear or nonlinear operations on the video playback possibility and the video playback completion, and use the result of the operation as the video interactivity. For example, the server may perform a weighted operation on the video playback possibility and the video playback completion, and use the result of the weighted operation as the video interactivity. Alternatively, the server may perform a multiplication operation on the video playback possibility and the video playback completion, and use the result of the multiplication operation as the video interactivity. For example, video interactivity = video playback possibility × video playback completion.

[0112] In this embodiment, the video interactivity corresponding to the original display video frame is obtained based on the video playback possibility and the video playback completion degree. Since the video interactivity is positively correlated with the video playback possibility and the video playback completion degree, the video interactivity can reflect the playback status of the searched video when the original display video frame is displayed, thereby improving the accuracy of the video interactivity.

[0113] In some embodiments, obtaining the information correlation between the candidate display video frame and the video search information as the candidate information correlation includes: obtaining the feature correlation between the candidate display video frame and the video search information as the frame feature correlation; obtaining the feature correlation between the video clip and the video search information as the clip feature correlation, wherein the candidate display video frame is obtained from the video clip, and the video clip is obtained by segmenting the search video; based on the frame feature correlation and the clip feature correlation, obtaining the information correlation between the candidate display video frame and the video search information as the candidate information correlation, and the candidate information correlation is positively correlated with the frame feature correlation and the clip feature correlation.

[0114] Among them, feature correlation refers to the correlation between features, and frame feature correlation refers to the correlation between the candidate video frame feature corresponding to the candidate display video frame and the search information feature corresponding to the video search information. The candidate video frame feature is the feature obtained by extracting the feature of the candidate display video frame. The search information feature is the feature obtained by extracting the feature of the video search information. The segment feature correlation refers to the correlation between the video segment feature corresponding to the video segment and the search information feature corresponding to the video search information. The video segment feature is the feature obtained by extracting the feature of the video segment.

[0115] Specifically, the server may calculate the correlation between the candidate video frame features and the search information features, and use the calculated correlation as the frame feature correlation corresponding to the candidate display video frame, extract features from the video clip to obtain video clip features corresponding to the video clip, calculate the correlation between the video clip features and the search information features, and use the calculated correlation as the clip feature correlation, perform a product operation on the frame feature correlation and the clip feature correlation, and use the result of the product operation as the candidate information correlation.

[0116] For example, the search video is segmented into N video segments, namely, video segment 1 to video segment N, and the key frames in each video segment are used as candidate display video frames. Assume that video segment i includes M key frames, the candidate display video frame is the k-th key frame in video segment i, the segment feature correlation corresponding to video segment i is P_qs[i], and the frame feature correlation corresponding to the k-th key frame in video segment i is P_qf[k]. Then, the candidate information correlation P_d[k] corresponding to the candidate display video frame (i.e., the k-th key frame in video segment i) is P_d[k]=P_qs[i]*P_qf[k]. i is greater than or equal to 1 and less than or equal to N, and k is greater than or equal to 1 and less than or equal to M.

[0117] In this embodiment, based on the frame feature correlation and the segment feature correlation, the information correlation between the candidate display video frame and the video search information is obtained as the candidate information correlation. Since the candidate information correlation is positively correlated with the frame feature correlation and the segment feature correlation, the candidate information correlation can reflect both the correlation between the video segment in which the video frame is located and the search information, and the correlation between the video frame itself and the search information, thereby improving the accuracy of the candidate information correlation.

[0118] In some embodiments, obtaining a set of candidate display video frames from a search video includes: obtaining a set of video segments obtained by segmenting the search video frame, the video segment set including multiple video segments; performing feature extraction on each video frame in a video frame sequence corresponding to the video segment to obtain a video frame feature sequence, and obtaining key frame detection results corresponding to each video frame in the video frame sequence based on the video frame feature sequence; based on the key frame detection results corresponding to each video frame in the video frame sequence, extracting the key frames corresponding to the video segment from the video frame sequence as candidate display video frames in the set of candidate display video frames.

[0119] The video frame feature is an image feature obtained by extracting image features from a video frame. The video frame feature sequence includes multiple video frame features, and each video frame feature in the video frame feature sequence is arranged according to the order of the video frame in the video clip. The higher the order of the video frame in the video clip, the higher the order of the video feature corresponding to the video frame in the video frame feature sequence. The key frame detection result may include the key frame probability, which refers to the probability that the video frame is a key frame. The key frame detection result may also include annotation information.

[0120] Specifically, the server can determine the key frame probability corresponding to each video frame according to the video frame feature sequence, the key frame probability refers to the probability that the video frame is a key frame, and the video frame whose key frame probability is greater than the probability threshold is determined as a key frame, and the video frame whose key frame probability is less than the probability threshold is determined as a non-key frame, and the key frame is used as a candidate display video frame corresponding to the search video. The probability threshold can be preset or set as needed.

[0121] In some embodiments, the server may determine the annotation information corresponding to the video frame based on the key frame probability, determine whether the video frame is a key frame based on the annotation information, and determine that the video frame is a key frame when the annotation information is positive annotation information, and determine that the video frame is a non-key frame when the annotation information is negative annotation information. For example, the server may use a trained key frame detection network to extract features of each video frame in a video frame sequence, obtain video frame features corresponding to each video frame, form a video frame feature sequence, obtain the key frame probability corresponding to each video frame based on the video frame feature sequence, and determine the annotation information corresponding to the video frame based on the key frame probability.

[0122] In this embodiment, key frame detection results corresponding to each video frame in the video frame sequence are obtained according to the video frame feature sequence, so that the process of obtaining the key frame detection results utilizes the order of the video frames in the sequence, thereby improving the accuracy of key frame detection.

[0123] In some embodiments, there are multiple search videos, and selecting target display video frames related to the video search information from the candidate display video frame set based on the candidate information relevance includes: selecting candidate display video frames related to the video search information from the candidate display video frame set based on the candidate information relevance to form a selected display video frame set corresponding to the search video; selecting target display video frames corresponding to each search video from the selected display video frame sets corresponding to each search video; wherein the video frame difference between the target display video frames corresponding to each search video is greater than a difference threshold.

[0124] The selected display video frame set includes a plurality of candidate display video frames. The candidate display video frames in the selected display video frame set are selected from the candidate display video frame set based on the candidate information relevance. The video frame difference refers to the difference between different video frames, which is used to reflect the difference between different video frames. The greater the video frame difference, the greater the difference between the video frames. The difference threshold can be preset or set as needed.

[0125] Specifically, the server may obtain the relative difference values ​​corresponding to each candidate display video frame in the candidate display video frame set, and form the candidate display video frames whose relative difference values ​​are greater than the difference value threshold into the selected display video frame set. For example, the candidate display video frame set is DC_List_1, the selected display video frame set is DC_List_2, and the video frames in DC_List_2 are selected from DC_List_1.

[0126] In some embodiments, the server may sequentially determine the target display video frames corresponding to each search video. For example, the server may arrange each search video to obtain a search video sequence, and sequentially determine the target display video frames corresponding to each search video according to the order of the search video in the search video sequence. The higher the order of the search video in the search video sequence, the higher the order of the target display video frame. For search videos whose target display video frames are not determined, the server may obtain each search video whose target display video frames are determined as comparison videos, obtain the target display video frames corresponding to each comparison video, select a video frame from a set of selected display video frames corresponding to the search video whose target display video frames are not determined, calculate the difference between the selected video frame and the target display video frame of the comparison video, and obtain the video frame difference. When the video frame difference is greater than the difference threshold, the selected video frame may be used as the target display video frame of the search video whose target display video frame is not determined. The server may calculate the similarity between different video frames, and determine the video frame difference between different video frames based on the calculated similarity. The video frame difference is negatively correlated with the similarity. For example, the cosine similarity between different video frames can be calculated, and the video frame difference can be obtained based on the cosine similarity. The video frame difference is negatively correlated with the cosine similarity. For example, the video frame difference can be a result obtained by subtracting the cosine similarity from a preset value, and the preset value can be 1, for example.

[0127] Among them, negative correlation means that when other conditions remain unchanged, the two variables change in opposite directions. When one variable changes from large to small, the other variable changes from small to large. It can be understood that the negative correlation here means that the direction of change is opposite, but it does not require that when one variable changes a little, the other variable must also change.

[0128] In some embodiments, the server may obtain the candidate information relevance corresponding to each candidate display video frame in the selected display video frame set, and arrange each candidate display video frame in the selected display video frame set in descending order of the candidate information relevance to obtain a selected display video frame sequence. The greater the candidate information relevance, the higher the ranking of the candidate display video frame in the selected display video frame sequence. The server may sequentially obtain video frames from the selected display video frame sequence, and calculate the difference between the obtained video frames and the target display video frame of the comparison video.

[0129] In the present embodiment, a target display video frame corresponding to each search video is selected from a set of selected display video frames corresponding to each search video. Since the video frame difference between the target display video frames corresponding to each search video is greater than a difference threshold, there are large differences between the target display video frames obtained for each search video. Therefore, when the target display video frames corresponding to each search video are displayed, the diversity of the target display video frames can be improved.

[0130] In some embodiments, selecting a target display video frame corresponding to each search video from a set of selected display video frames corresponding to each search video includes: determining a search video of a target display video frame to be selected as the current video; obtaining target display video frames corresponding to each comparison video to form a comparison video frame set, wherein the comparison video is a search video of a determined target display video frame; selecting a video frame whose video frame difference with a target display video frame in the comparison video frame set is greater than a difference threshold from the set of selected display video frames corresponding to the current video, and using the video frame whose difference is greater than the difference threshold as the target display video frame corresponding to the current video.

[0131] Among them, the current video can be any search video in each search video whose target display video frame has not been determined. The comparison video refers to the search video in each search video whose target display video frame has been determined. When each search video has not determined the target display video frame, the current video has no comparison video frame. At this time, the target display video frame corresponding to the current video can be determined according to the relevance of the candidate information. For example, the video frame with the largest candidate information relevance can be selected from the selected display video frame set corresponding to the current video as the target display video frame corresponding to the current video. The comparison video frame set is a set composed of the target display video frames of the comparison video.

[0132] Specifically, the server may randomly select a search video from each search video of an undetermined target display video frame as the current video, or the server may arrange each search video to obtain a search video sequence, and sequentially obtain search videos of undetermined target display video frames from the search video sequence according to the order of the search videos in the search video sequence as the current video. The server may obtain a search video of a determined target display video frame and use the search video as a comparison video corresponding to the current video.

[0133] In some embodiments, the server can use any one of the video frames in the selected display video frame set corresponding to the current video, whose video frame difference with the target display video frame in the comparison video frame set is greater than a difference threshold, as the target display video frame corresponding to the current video. For example, the video frame with the greatest candidate information relevance can be used as the target display video frame.

[0134] In some embodiments, when the video frame differences between the selected display video frame corresponding to the current video and each target display video frame in the comparison video frame set are greater than the difference threshold, the selected display video frame is used as the target display video frame corresponding to the current video. The selected display video frame corresponding to the current video refers to the candidate display video frame in the selected display video frame corresponding to the current video.

[0135] In some embodiments, when there is a target display video frame in the comparison video frame set whose video frame difference with the selected display video frame is less than a difference threshold, the selected display video frame is not used as the target display video frame corresponding to the current video.

[0136] In this embodiment, from the selected display video frame set corresponding to the current video, video frames whose video frame difference with the target display video frames in the comparison video frame set is greater than a difference threshold are selected, and the video frames greater than the difference threshold are used as the target display video frames corresponding to the current video, thereby improving the distinction between the target display video frames between different search videos and improving the diversity of the target display video frames.

[0137] In some embodiments, a video frame whose video frame difference with a target display video frame in a comparison video frame set is greater than a difference threshold is selected from a selected display video frame set corresponding to the current video, and the video frame greater than the difference threshold is used as the target display video frame corresponding to the current video, including: obtaining the current display video frame from the selected display video frame set corresponding to the current video in descending order of candidate information relevance; obtaining the current video frame difference between the current display video frame and the target display video frame in the comparison video frame set; when the current video frame difference corresponding to each target display video frame in the comparison video frame set is greater than the difference threshold, using the current display video frame as the target display video frame corresponding to the current video, otherwise returning to the step of obtaining the current display video frame from the selected display video frame set corresponding to the current video in descending order of candidate information relevance.

[0138] The current display video frame may be any video frame in the selected display video frame set of the current video. The current video frame difference refers to the video frame difference between the current display video frame and the target display video frame in the comparison video frame set.

[0139] Specifically, the server can preferentially obtain a video frame with a larger candidate information relevance from the selected display video frame set corresponding to the current video as the current display video frame. For example, the selected display video frame set corresponding to the current video includes video frame 1, video frame 2 and video frame 3, and the candidate information relevance of video frame 1 is greater than the candidate information relevance of video frame 2, and the candidate information relevance of video frame 2 is greater than the candidate information relevance of video frame 3. Then, video frame 1 is preferentially selected as the current display video frame, followed by video frame 2, and finally video frame 3 is selected as the current display video frame. Of course, when video frame 1 has been determined as the target display video frame of the current video, it is no longer necessary to select video frame 2 and video frame 3 as the current display video frame.

[0140] In some embodiments, the server may calculate the difference between the current video frame and each target display video frame in the comparison video frame set to obtain the difference between each current video frame. When the difference between each current video frame is greater than a difference threshold, the current display video frame is used as the target display video frame corresponding to the current video.

[0141] In some embodiments, the server arranges the selected display video frames to obtain a selected display video frame sequence, which may also be referred to as a selected display video frame list. In the selected display video frame list, selected display video frames with greater candidate information relevance are arranged before selected display video frames with less candidate information relevance. The server determines the target display video frame from the selected display video frame sequence according to the arrangement order.

[0142] For example, assuming that the search videos are arranged into a search video list of [search video 1, search video 2, search video 3], search video 1 corresponds to selected display video frame sequence 1, search video 2 corresponds to selected display video frame sequence 2, and search video 3 corresponds to selected display video frame sequence 3. First, determine the target display video frame for search video 1, and take the selected display video frame arranged in the first place in the selected display video frame sequence 1 as the target display video frame, recorded as target display video frame 1; secondly, determine the target display video frame for search video 2, obtain the video frame difference between the selected display video frame arranged in the first place in the selected display video frame sequence 2 and the target display video frame 1, when the video frame difference is greater than the difference threshold, then take the selected display video frame arranged in the first place in the selected display video frame sequence 2 as the target display video frame for search video 2, otherwise, obtain the video frame difference between the selected display video frame arranged in the second place in the selected display video frame sequence 2 and the target display video frame 1, until the video frame difference is greater than the difference threshold, then The target display video frame is recorded as target display video frame 2; finally, the target display video frame of the search video 3 is determined, and the video frame difference between the selected display video frame arranged in the first position in the selected display video frame sequence 3 and the target display video frame 1 is obtained, which is recorded as video frame difference 1, and the video frame difference between the selected display video frame arranged in the first position in the selected display video frame sequence 3 and the target display video frame 2 is obtained, which is recorded as video frame difference 2. When the video frame difference 1 is greater than the difference threshold and the video frame difference 2 is greater than the difference threshold, the selected display video frame arranged in the first position in the selected display video frame sequence 3 is used as the target display video frame of the search video 3, otherwise, the video frame difference 1 between the selected display video frame arranged in the second position in the selected display video frame sequence 3 and the target display video frame 1 is obtained, and the video frame difference 2 between the selected display video frame arranged in the first position in the selected display video frame sequence 3 and the target display video frame 2 is obtained, until the video frame difference 1 is greater than the difference threshold and the video frame difference 2 is greater than the difference threshold.

[0143] In this embodiment, the current display video frame is obtained from the set of selected display video frames corresponding to the current video in order of candidate information relevance from large to small. When the difference between the current video frames corresponding to the target display video frames in the comparison video frame set is greater than the difference threshold, the current display video frame is used as the target display video frame corresponding to the current video. Otherwise, the step of obtaining the current display video frame from the set of selected display video frames corresponding to the current video in order of candidate information relevance from large to small is returned to the step of obtaining the current display video frame from the set of selected display video frames corresponding to the current video in order of candidate information relevance from large to small. In this way, a video frame with a large candidate information relevance and a large difference from the determined target display video frame can be used as the target display video frame of the current video, thereby improving the diversity of the target display video frames and improving the relevance of the target display video frames with the video search information.

[0144] In some embodiments, determining the search video of the target display video frame to be selected as the current video includes: determining the search result ranking corresponding to each search video; and determining the search video of the target display video frame to be selected as the current video in sequence from the multiple search videos obtained by the search according to the search result ranking.

[0145] Among them, the search result sorting refers to the sorting of the search videos in each search video sequence. The search video sequence is a sequence obtained by arranging the search videos. For example, it can be arranged in the order of the time when the videos are searched, for example, the video searched first is arranged before the video searched later.

[0146] Specifically, the server can determine the search video for the target display frame to be selected from each search video according to the sorting of the search results, and use it as the current video. For example, the server can determine the search video for the target display frame to be selected in the order of the search results from front to back, and prioritize the search video with a higher sorting in the search results as the current video.

[0147] In this embodiment, the search videos for the target display video frames to be selected are determined in turn from the multiple search videos obtained by the search according to the order of the search results, and are used as the current video, so that the target display video frames corresponding to the respective search videos can be determined in an orderly manner, thereby improving the efficiency of video search.

[0148] In some embodiments, Figure 7 As shown, a video search method is provided, which is applied to Figure 1The terminal 102 in the example is used for explanation, and the steps include: S702, displaying a search information input area; S704, receiving video search information through the search information input area; S706, triggering a video search based on the video search information in response to a search operation on the search information input area; S708, displaying a video search result corresponding to the searched video obtained by the search, the video search result includes a target display video frame related to the video search information in the search video, and the target display video frame is displayed as a video display frame in the video search result.

[0149] The search information input area is used to receive video search information input or selected by the user. The video display frame is a video frame used for display, for example, the target display video frame can be displayed as the cover image of the search video.

[0150] Specifically, the terminal can display a video search interface and a search information input area in the video search interface. The terminal can also display a search confirmation control on the video search interface. When a trigger operation on the search confirmation control is obtained, it is determined that a search operation for the search information input area is obtained. The terminal responds to the trigger operation on the search response control, obtains the video search information received in the search information input area, generates a video search request carrying the video search information, and sends the video search request to the server.

[0151] In some embodiments, the server responds to a video search request sent by a terminal, extracts video search information from the video search request, searches for a video that matches the video search information as the search video, uses the above-mentioned video search method to determine the target display video frames corresponding to each search video, obtains the video search information, performs a video search based on the video search information, and obtains the search video. For example, the server can obtain a set of candidate display video frames from the search video, where the candidate display video frame set includes multiple candidate display video frames, obtains information relevance between the candidate display video frames and the video search information as the candidate information relevance, selects a target display video frame related to the video search information from the candidate display video frame set based on the candidate information relevance, generates a video search result corresponding to the search video based on the target display video frame, and returns the video search result to the terminal.

[0152] In some embodiments, the terminal receives the video search results returned by the server, obtains the target display video frame corresponding to the search identification from the video search results, and displays the target display video frames corresponding to each search video. For example, the terminal can display a search result display area in the video search interface, and the search result display area is used to display the target display video frame in the video search results. When the terminal obtains a trigger operation for the displayed target display video frame, the search video corresponding to the target display video frame can be played. For example, the position of the target display video frame in the search video can be used as the starting playback position for playback.

[0153] In the above-mentioned video search method, a search information input area is displayed, video search information is received through the search information input area, and in response to a search operation on the search input area, a video search based on the video search information is triggered, and video search results corresponding to the searched video obtained by the search are displayed. The video search results include target display video frames related to the video search information in the search video. The target display video frames are displayed as video display frames in the video search results, thereby improving the relevance of the video search results to the video search information and improving the effectiveness of the video search results.

[0154] In some embodiments, a video search method is provided, comprising the following steps:

[0155] 1. The terminal displays the search information input area.

[0156] 2. The terminal receives video search information through the search information input area.

[0157] 3. In response to the search operation on the search information input area, the terminal triggers a video search based on the video search information and sends a video search request carrying the video search information to the server.

[0158] 4. The server responds to the video search request, obtains video search information from the video search request, performs a video search based on the video search information, and assembles the searched videos into a search video set.

[0159] 5. The server segments the search video in the search video set to obtain a set of video clips corresponding to the search video.

[0160] 6. The server extracts key frames from each video segment in the video segment set, and composes a candidate cover image set with the key frames extracted from each video segment.

[0161] 7. The server performs feature extraction on the video clips in the video clip set to obtain video clip features corresponding to the video clips, performs feature extraction on the candidate cover images in the candidate cover image set to obtain candidate cover features, and performs feature extraction on the video search information to obtain video search features.

[0162] 8. The server calculates the correlation between the video clip features and the video search features to obtain the clip feature correlation corresponding to the video clip, and calculates the correlation between the candidate cover features and the video search features to obtain the frame feature correlation corresponding to the candidate cover image.

[0163] 9. The server obtains the segment feature correlation corresponding to the video segment where the candidate cover image is located, multiplies the segment feature correlation with the frame feature correlation corresponding to the candidate cover image, and uses the result of the multiplication as the candidate information correlation corresponding to the candidate cover image.

[0164] 10. The server obtains the original cover image corresponding to the searched video, extracts features from the original cover image to obtain original cover features, calculates the correlation between the original cover features and the video search features, and obtains the cover feature correlation corresponding to the original cover image.

[0165] 11. When the server obtains the original cover image in the historical time period and displays it as the cover image of the search video, the video play possibility and video play completion degree obtained by the search video are multiplied by the video play possibility and the video play completion degree to obtain the video interactivity corresponding to the original cover image.

[0166] 12. The server multiplies the video interactivity by the cover feature relevance, and uses the result of the operation as the original information relevance corresponding to the original cover image.

[0167] 13. The server compares the candidate information relevance of the candidate cover image corresponding to the search video with the corresponding original information relevance. When the candidate information relevance is greater than the original information relevance, the candidate cover image is used as the selected cover image corresponding to the search video to form a selected cover image set.

[0168] 14. The server obtains a set of selected cover images corresponding to each search video, and selects a target cover image corresponding to each search video from each set of selected cover images, wherein the video frame difference between the target cover images corresponding to each search video is greater than a difference threshold.

[0169] 15. The server generates a video search result corresponding to the search video based on the target cover image corresponding to the search video, the video search result includes the target cover image, and sends the video search result to the terminal.

[0170] 16. The terminal receives the video search results returned by the server and displays the target cover image in the video search results.

[0171] like Figure 8 As shown, a schematic diagram of a video search method in some embodiments is shown. Figure 8 The video platform in the video search function can perform video search. The terminal can display the search information input area through the interface of the video platform, so that the user can search for videos in the video platform. The terminal obtains the video search information input on the video platform. The video search information is Figure 8 The user search query in the video is sent to the server. The server searches for the search video according to the user search query, segments the search video, obtains the video segment corresponding to the search video, calculates the correlation between the user search query and the video segment, obtains the segment feature correlation corresponding to the video segment, obtains the key frame from the video segment, calculates the correlation between the key frame of the video segment and the user search query, obtains the frame feature correlation, multiplies the frame feature correlation with the segment feature correlation, and obtains the information correlation corresponding to the key frame. The server obtains the original cover image corresponding to the search video, calculates the correlation between the original cover image and the user search query, and obtains the original feature correlation. "Calculation of the posterior effect of the original video cover image" refers to obtaining the video interactivity corresponding to the original cover image, multiplying the video interactivity with the original feature correlation, and obtaining the original information correlation corresponding to the original cover image. "Construction of dynamic cover image candidates for video search relevance" means using the original information relevance as the screening threshold, comparing the information relevance corresponding to the key frame with the original information relevance, and when the information relevance corresponding to the key frame is greater than the original information relevance or the difference between the information relevance corresponding to the key frame and the original information relevance is greater than the threshold, the key frame is used as the selected cover image for the search video. The video list is a list obtained by arranging the various search videos, and the "dynamic diversity of search result video list" is used to determine the target cover images corresponding to each search video in the video list from the selected cover images corresponding to each search video, and make the difference between each target cover image greater than the difference threshold. The server can return the target cover images corresponding to each search video to the terminal, and the terminal can display each target cover image. When a trigger operation on the target cover image, such as a click operation, is obtained, the corresponding search video is played.

[0172] In this embodiment, the display of videos under different search information is optimized, so that when videos are displayed under different search contexts, images in the video that are more relevant to the search information can be used as the cover image of the video, so that the part of the video that is more relevant to the search information can be intuitively displayed, improving the display effect of the cover image and further improving the video click efficiency. In addition, the cover images corresponding to the videos in the video list in the search results are dynamically diversified, so that each cover image has a large difference, reducing the similarity of the cover images corresponding to the videos in the video list in the search results, improving the diversity of the cover images, and enhancing the user's desire to browse the displayed videos, improving the click-through rate and play rate of the video, and improving the conversion ability of the search results such as the play.

[0173] It should be understood that although Figure 2-Figure 8 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 8 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0174] In some embodiments, Fig. 9 As shown, a video search device is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: a search video obtaining module 902, a candidate display video frame set obtaining module 904, a candidate information relevance obtaining module 906, a target display video frame obtaining module 908 and a video search result sending module 910, wherein:

[0175] The video search obtaining module 902 is used to obtain video search information, perform video search based on the video search information, and obtain the search video;

[0176] A candidate display video frame set obtaining module 904 is used to obtain a candidate display video frame set from the search video, where the candidate display video frame set includes multiple candidate display video frames;

[0177] A candidate information relevance obtaining module 906 is used to obtain the information relevance between the candidate display video frame and the video search information as the candidate information relevance;

[0178] A target display video frame obtaining module 908 is used to select a target display video frame related to the video search information from the candidate display video frame set based on the candidate information relevance;

[0179] The video search result sending module 910 is used to send the video search result, where the video search result includes a target display video frame.

[0180] The above-mentioned video search device obtains video search information, performs video search based on the video search information to obtain the search video, obtains a candidate display video frame set from the search video, the candidate display video frame set includes multiple candidate display video frames, obtains the information relevance between the candidate display video frames and the video search information as the candidate information relevance, selects the target display video frame related to the video search information from the candidate display video frame set based on the candidate information relevance, sends the video search result, the video search result includes the target display video frame, thereby returning the video frames in the searched video that are more relevant to the video search information to the terminal, thereby improving the relevance of the video search results and the video search information, thereby improving the effectiveness of the video search results.

[0181] In some embodiments, the target display video frame obtaining module includes: an original display video frame obtaining unit, used to obtain the original display video frame corresponding to the search video; an original information relevance obtaining unit, used to obtain the information relevance between the original display video frame and the video search information as the original information relevance; a first target display video frame obtaining unit, used to determine the relative difference value of the candidate information relevance relative to the original information relevance, select from the candidate display video frame set the candidate display video frame whose relative difference value is greater than the difference threshold, and use at least one of the candidate display video frames whose relative difference value is greater than the difference threshold as the target display video frame related to the video search information.

[0182] In some embodiments, the original information relevance acquisition unit is also used to obtain the feature correlation between the original display video frame and the video search information as the original feature correlation; obtain the video interactivity corresponding to the original display video frame, the video interactivity is the video interactivity of the search video when the original display video frame is displayed as a video search result of the search video; obtain the original information correlation between the original display video frame and the video search information based on the video interactivity and the original feature correlation; the original information correlation is positively correlated with the video interactivity and the original feature correlation.

[0183] In some embodiments, the original information relevance acquisition unit is also used to obtain the video playback possibility corresponding to the search video when the original display video frame is displayed as the video search result of the search video; obtain the video playback completion corresponding to the search video when the original display video frame is displayed as the video search result of the search video; obtain the video interactivity corresponding to the original display video frame based on the video playback possibility and the video playback completion; the video interactivity is positively correlated with the video playback possibility and the video playback completion.

[0184] In some embodiments, the candidate information relevance obtaining module includes: a frame feature correlation obtaining unit, used to obtain the feature correlation between the candidate display video frame and the video search information as the frame feature correlation; a segment feature correlation obtaining unit, used to obtain the feature correlation between the video segment and the video search information as the segment feature correlation, wherein the candidate display video frame is obtained from the video segment, and the video segment is obtained by segmenting the search video; the candidate information relevance obtaining unit is used to obtain the information correlation between the candidate display video frame and the video search information based on the frame feature correlation and the segment feature correlation as the candidate information relevance, and the candidate information relevance is positively correlated with the frame feature correlation and the segment feature correlation.

[0185] In some embodiments, the module for obtaining a candidate display video frame set includes: a video segment set obtaining unit, which is used to obtain a video segment set obtained by segmenting the search video frame, wherein the video segment set includes multiple video segments; a key frame detection result obtaining unit, which is used to extract features of each video frame in a video frame sequence corresponding to the video segment to obtain a video frame feature sequence, and obtain key frame detection results corresponding to each video frame in the video frame sequence based on the video frame feature sequence; and a candidate display video frame obtaining unit, which is used to extract key frames corresponding to the video segment from the video frame sequence based on the key frame detection results corresponding to each video frame in the video frame sequence, as candidate display video frames in the candidate display video frame set.

[0186] In some embodiments, there are multiple search videos, and the target display video frame obtaining module includes: a display video frame set selection composition unit, which is used to select candidate display video frames related to the video search information from the candidate display video frame set based on the candidate information relevance to form a selected display video frame set corresponding to the search video; a second target display video frame obtaining unit, which is used to select target display video frames corresponding to each search video from the selected display video frame sets corresponding to each search video; wherein the video frame difference between the target display video frames corresponding to each search video is greater than a difference threshold.

[0187] In some embodiments, the second target display video frame obtaining unit is also used to determine the search video of the target display video frame to be selected as the current video; obtain the target display video frames corresponding to each comparison video to form a comparison video frame set, and the comparison video is the search video of the determined target display video frame; from the selected display video frame set corresponding to the current video, select a video frame whose video frame difference with the target display video frame in the comparison video frame set is greater than a difference threshold, and use the video frame greater than the difference threshold as the target display video frame corresponding to the current video.

[0188] In some embodiments, the second target display video frame obtaining unit is also used to obtain the current display video frame from the selected display video frame set corresponding to the current video in order of candidate information relevance from large to small; obtain the current video frame difference between the current display video frame and the target display video frame in the comparison video frame set; when the current video frame difference corresponding to each target display video frame in the comparison video frame set is greater than the difference threshold, the current display video frame is used as the target display video frame corresponding to the current video, otherwise, the step of obtaining the current display video frame from the selected display video frame set corresponding to the current video in order of candidate information relevance from large to small is returned.

[0189] In some embodiments, the second target display video frame obtaining unit is also used to include: determining the search result ranking corresponding to each search video; and determining the search video of the target display video frame to be selected from the multiple search videos obtained by the search according to the search result ranking as the current video.

[0190] In some embodiments, Fig.10 As shown, a video search device is provided, which can be a software module or a hardware module, or a combination of the two to form a part of a computer device. The device specifically includes: a search information input area display module 1002, a video search information receiving module 1004, a video search trigger module 1006 and a video search result display module 1008, wherein:

[0191] Search information input area display module 1002, used to display the search information input area;

[0192] The video search information receiving module 1004 is used to receive video search information through the search information input area;

[0193] A video search triggering module 1006, configured to trigger a video search based on the video search information in response to a search operation on the search information input area;

[0194] The video search result display module 1008 is used to display the video search results corresponding to the searched video. The video search results include target display video frames related to the video search information in the searched video. The target display video frames are displayed as video display frames in the video search results.

[0195] The above-mentioned video search device displays a search information input area, receives video search information through the search information input area, triggers a video search based on the video search information in response to a search operation on the search input area, and displays video search results corresponding to the searched video obtained by the search. The video search results include target display video frames related to the video search information in the search video. The target display video frames are displayed as video display frames in the video search results, thereby improving the relevance of the video search results to the video search information and improving the effectiveness of the video search results.

[0196] For the specific definition of the video search device, please refer to the definition of the video search method above, which will not be repeated here. Each module in the above-mentioned video search device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0197] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a video search method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0198] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Fig.12As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to the video search method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video search method is implemented.

[0199] Those skilled in the art will understand that Fig.11 and Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0200] In some embodiments, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0201] In some embodiments, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0202] In some embodiments, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0203] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0204] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0205] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A video search method, characterized in that: The method comprises: Acquire video search information, perform video search based on the video search information, and obtain search videos; Acquire a candidate display video frame set from each video segment obtained by segmenting the search video, wherein the candidate display video frame set includes a plurality of candidate display video frames; Obtaining a frame feature correlation between the candidate display video frame and the video search information, and obtaining a segment feature correlation between the video segment to which the candidate display video frame belongs and the video search information, and obtaining a candidate information correlation between the candidate display video frame and the video search information based on the frame feature correlation and the segment feature correlation; Obtaining the video playback completion degree corresponding to the search video when the original display video frame of the search video is displayed as the video search result of the search video, wherein the video playback completion degree represents the ratio of the playback time of the search video to the total video time; Acquire a feature correlation between the original display video frame and the video search information as an original feature correlation; Based on the video playback completion degree and the original feature relevance, obtaining the original information relevance between the original display video frame and the video search information; Based on the original information relevance and the candidate information relevance, a target display video frame related to the video search information is selected from the candidate display video frame set; The video search result is sent, where the video search result includes the target display video frame.

2. The method according to claim 1, characterized in that The step of selecting a target display video frame related to the video search information from the candidate display video frame set based on the original information relevance and the candidate information relevance includes: Determining a relative difference value of the candidate information relevance relative to the original information relevance; Selecting a candidate display video frame whose relative difference value is greater than a difference threshold from the candidate display video frame set; At least one of the candidate display video frames whose relative difference value is greater than the difference threshold is used as a target display video frame related to the video search information.

3. The method according to claim 1, characterized in that The obtaining, based on the video playback completion degree and the original feature relevance, the original information relevance between the original display video frame and the video search information includes: Based on the video playback completion degree, obtaining the video interactivity corresponding to the original displayed video frame; Based on the video interactivity and the original feature relevance, the original information relevance between the original display video frame and the video search information is obtained; the original information relevance is positively correlated with the video interactivity and the original feature relevance.

4. The method according to claim 3, characterized in that The obtaining of the video interactivity corresponding to the original displayed video frame based on the video playback completion degree includes: Obtaining a video playability corresponding to the search video when the original display video frame is displayed as a video search result of the search video; The video interactivity corresponding to the original display video frame is obtained based on the video play possibility and the video play completion; the video interactivity is positively correlated with the video play possibility and the video play completion.

5. The method according to claim 1, characterized in that The candidate information relevance is positively correlated with the frame feature relevance and the segment feature relevance.

6. The method according to claim 1, characterized in that The step of obtaining a candidate display video frame set from each video segment obtained by segmenting the search video comprises: Acquire a video segment set obtained by segmenting the search video frame, wherein the video segment set includes a plurality of video segments; Extracting features from each video frame in the video frame sequence corresponding to the video clip to obtain a video frame feature sequence, and obtaining a key frame detection result corresponding to each video frame in the video frame sequence based on the video frame feature sequence; Based on the key frame detection results corresponding to each video frame in the video frame sequence, the key frame corresponding to the video clip is extracted from the video frame sequence as a candidate display video frame in the candidate display video frame set.

7. The method according to claim 1, characterized in that The search video is multiple, and the method further includes: Selecting candidate display video frames related to the video search information from the candidate display video frame set based on the relevance of the candidate information to form a selected display video frame set corresponding to the search video; From the selected display video frame sets corresponding to each of the search videos, target display video frames corresponding to each of the search videos are selected; wherein the video frame difference between the target display video frames corresponding to each of the search videos is greater than a difference threshold.

8. The method according to claim 7, characterized in that The selecting and obtaining target display video frames corresponding to each of the search videos from the selected display video frames sets corresponding to each of the search videos respectively comprises: Determine the search video of the target display video frame to be selected as the current video; Obtaining target display video frames corresponding to each comparison video to form a comparison video frame set, wherein the comparison video is a search video of the determined target display video frame; From the selected display video frame set corresponding to the current video, select a video frame whose video frame difference with the target display video frame in the comparison video frame set is greater than a difference threshold, and use the video frame greater than the difference threshold as the target display video frame corresponding to the current video.

9. The method according to claim 8, characterized in that The step of selecting, from the selected display video frame set corresponding to the current video, a video frame whose video frame difference with a target display video frame in the comparison video frame set is greater than a difference threshold, and using the video frame greater than the difference threshold as the target display video frame corresponding to the current video comprises: Obtaining the current display video frame from the set of selected display video frames corresponding to the current video in descending order of the relevance of the candidate information; Obtaining a current video frame difference between a current display video frame and a target display video frame in the comparison video frame set; When the difference between the current video frames corresponding to the target display video frames in the comparison video frame set is greater than the difference threshold, the current display video frame is used as the target display video frame corresponding to the current video; otherwise, the process returns to the step of obtaining the current display video frame from the selected display video frame set corresponding to the current video in order of the candidate information relevance from large to small.

10. The method according to claim 8, characterized in that The search video of the target display video frame to be selected as the current video includes: Determine the search result ranking corresponding to each of the search videos; According to the search result sorting, the search video of the target display video frame to be selected is determined in turn from the multiple search videos obtained by the search as the current video.

11. A video search method, characterized in that: The method comprises: Display search information input area; receiving video search information through the search information input area; In response to a search operation on the search information input area, triggering a video search based on the video search information; Displaying a video search result corresponding to the searched video obtained by the search, wherein the video search result includes a target display video frame in the searched video that is related to the video search information, and the target display video frame is displayed as a video display frame in the video search result; The step of obtaining the target display video frame includes: Acquire a set of candidate display video frames from each video segment obtained by segmenting the search video; Obtaining a frame feature correlation between the candidate display video frame and the video search information, and obtaining a segment feature correlation between the video segment to which the candidate display video frame belongs and the video search information, and obtaining a candidate information correlation between the candidate display video frame and the video search information based on the frame feature correlation and the segment feature correlation; Obtaining the video playback completion degree corresponding to the search video when the original display video frame of the search video is displayed as the video search result of the search video, wherein the video playback completion degree represents the ratio of the playback time of the search video to the total video time; Acquire a feature correlation between the original display video frame and the video search information as an original feature correlation; Based on the video playback completion degree and the original feature relevance, obtaining the original information relevance between the original display video frame and the video search information; Based on the original information relevance and the candidate information relevance, a target display video frame related to the video search information is selected from the candidate display video frame set.

12. A video search device, characterized in that: The device comprises: A video search module is used to obtain video search information, perform video search based on the video search information, and obtain the search video; A candidate display video frame set obtaining module is used to obtain a candidate display video frame set from each video segment obtained by segmenting the search video, wherein the candidate display video frame set includes multiple candidate display video frames; A candidate information relevance obtaining module is used to obtain the frame feature relevance between the candidate display video frame and the video search information, and obtain the segment feature relevance between the video segment to which the candidate display video frame belongs and the video search information, and obtain the candidate information relevance between the candidate display video frame and the video search information based on the frame feature relevance and the segment feature relevance; A target display video frame obtaining module is used to obtain the video playback completion corresponding to the search video when the original display video frame of the search video is displayed as the video search result of the search video, and the video playback completion represents the ratio of the playback time of the search video to the total video time; obtain the feature correlation between the original display video frame and the video search information as the original feature correlation; based on the original feature correlation and the video playback completion, obtain the original information correlation between the original display video frame and the video search information; based on the original information correlation and the candidate information correlation, select a target display video frame related to the video search information from the candidate display video frame set; The video search result sending module is used to send the video search result, and the video search result includes the target display video frame.

13. The device according to claim 12, characterized in that The target display video frame obtaining module is also used for: Determining a relative difference value of the candidate information relevance relative to the original information relevance; Selecting a candidate display video frame whose relative difference value is greater than a difference threshold from the candidate display video frame set; At least one of the candidate display video frames whose relative difference value is greater than the difference threshold is used as a target display video frame related to the video search information.

14. The device according to claim 12, characterized in that The target display video frame obtaining module is also used for: Based on the video playback completion degree, obtaining the video interactivity corresponding to the original displayed video frame; Based on the video interactivity and the original feature relevance, the original information relevance between the original display video frame and the video search information is obtained; the original information relevance is positively correlated with the video interactivity and the original feature relevance.

15. The device according to claim 14, characterized in that The target display video frame obtaining module is also used for: Obtaining a video playability corresponding to the search video when the original display video frame is displayed as a video search result of the search video; The video interactivity corresponding to the original display video frame is obtained based on the video play possibility and the video play completion; the video interactivity is positively correlated with the video play possibility and the video play completion.

16. The device according to claim 12, characterized in that The candidate information relevance is positively correlated with the frame feature relevance and the segment feature relevance.

17. The device according to claim 12, characterized in that The module for obtaining the candidate display video frame set is further used for: Acquire a video segment set obtained by segmenting the search video frame, wherein the video segment set includes a plurality of video segments; Extracting features from each video frame in the video frame sequence corresponding to the video clip to obtain a video frame feature sequence, and obtaining a key frame detection result corresponding to each video frame in the video frame sequence based on the video frame feature sequence; Based on the key frame detection results corresponding to each video frame in the video frame sequence, the key frames corresponding to the video clip are extracted from the video frame sequence as candidate display video frames in the candidate display video frame set.

18. The device according to claim 12, characterized in that The search video is multiple, and the target display video frame obtaining module is also used for: Selecting candidate display video frames related to the video search information from the candidate display video frame set based on the relevance of the candidate information to form a selected display video frame set corresponding to the search video; From the selected display video frame sets corresponding to each of the search videos, target display video frames corresponding to each of the search videos are selected; wherein the video frame difference between the target display video frames corresponding to each of the search videos is greater than a difference threshold.

19. The device according to claim 18, characterized in that The target display video frame obtaining module is also used for: Determine the search video of the target display video frame to be selected as the current video; Obtaining target display video frames corresponding to each comparison video to form a comparison video frame set, wherein the comparison video is a search video of the determined target display video frame; From the selected display video frame set corresponding to the current video, select a video frame whose video frame difference with the target display video frame in the comparison video frame set is greater than a difference threshold, and use the video frame greater than the difference threshold as the target display video frame corresponding to the current video.

20. The device according to claim 19, characterized in that The target display video frame obtaining module is also used for: Obtaining the current display video frame from the set of selected display video frames corresponding to the current video in descending order of the relevance of the candidate information; Obtaining a current video frame difference between a current display video frame and a target display video frame in the comparison video frame set; When the difference between the current video frames corresponding to the target display video frames in the comparison video frame set is greater than the difference threshold, the current display video frame is used as the target display video frame corresponding to the current video; otherwise, the process returns to the step of obtaining the current display video frame from the selected display video frame set corresponding to the current video in order of the candidate information relevance from large to small.

21. The device according to claim 19, characterized in that The target display video frame obtaining module is also used for: Determine the search result ranking corresponding to each of the search videos; According to the search result sorting, the search video of the target display video frame to be selected is determined in turn from the multiple search videos obtained by the search as the current video.

22. A video search device, characterized in that: The device comprises: Search information input area display module, used to display the search information input area; A video search information receiving module, used for receiving video search information through the search information input area; A video search triggering module, configured to trigger a video search based on the video search information in response to a search operation on the search information input area; A video search result display module is used to display the video search results corresponding to the searched video, wherein the video search results include target display video frames in the search video that are related to the video search information, and the target display video frames are displayed as video display frames in the video search results; wherein the step of obtaining the target display video frames includes: obtaining a set of candidate display video frames from each video segment obtained by segmenting the search video; obtaining a frame feature correlation between the candidate display video frames and the video search information, and obtaining a segment feature correlation between the video segment to which the candidate display video frames belong and the video search information, and obtaining a set of candidate display video frames based on the frame feature correlation and the segment feature correlation. The method comprises the following steps: obtaining the candidate information correlation between the search video search information and the original display video frame; obtaining the video playback completion degree corresponding to the search video when the original display video frame of the search video is displayed as the video search result of the search video, and the video playback completion degree represents the ratio of the playback time of the search video to the total video time; obtaining the feature correlation between the original display video frame and the video search information as the original feature correlation; obtaining the original information correlation between the original display video frame and the video search information based on the video playback completion degree and the original feature correlation; and selecting a target display video frame related to the video search information from the candidate display video frame set based on the original information correlation and the candidate information correlation.

23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

24. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

25. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images