Video retrieval method, device, electronic device and storage medium

Abstract

The present application is applicable to the field of retrieval technology, and provides a video retrieval method, device, electronic device and storage medium, including: determining a first video frame sequence based on a reference action video, wherein the reference action video is a video showing human body movements, and the first video frame sequence includes multiple first video frames containing human body movements, and all the first video frames contain multiple human body movements; based on the first video frame sequence, searching the video to be retrieved to obtain a video retrieval result, wherein the video retrieval result is used to indicate whether there is a video frame sequence matching the first video frame sequence in the video to be retrieved. The present application can improve the efficiency of video retrieval.

Description

Technical Field

[0001] The present application belongs to the field of retrieval technology, and in particular, relates to a video retrieval method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Video-based video retrieval technology is a technology that searches and matches related video clips in other videos based on a given video. It has broad application prospects in fields such as copyright protection or action evaluation.

[0003] Currently, when performing video-based video retrieval, the similarity between the video frames of the entire reference video and the video frames of the video to be retrieved is usually calculated, which has high requirements on computing resources and requires a large amount of computing resources of the device during the retrieval process, which is not conducive to practical applications. Summary of the invention

[0004] The embodiments of the present application provide a video retrieval method, device, electronic device and storage medium, which can improve the efficiency of video retrieval.

[0005] In a first aspect, an embodiment of the present application provides a video retrieval method, comprising:

[0006] Determine a first video frame sequence according to a reference action video, wherein the reference action video is a video showing human body actions, and the first video frame sequence includes a plurality of first video frames containing human body actions, and all the first video frames contain a plurality of human body actions;

[0007] The video to be retrieved is retrieved based on the first video frame sequence to obtain a video retrieval result, where the video retrieval result is used to indicate whether there is a video frame sequence matching the first video frame sequence in the video to be retrieved.

[0008] In a possible implementation manner of the first aspect, before searching for a video to be searched based on the first video frame sequence to obtain a video search result, the method further includes:

[0009] Determine a second video frame sequence according to the video to be retrieved, wherein the second video frame included in the second video frame sequence is a video frame containing a human body action in the video to be retrieved;

[0010] Correspondingly, searching the video to be searched based on the first video frame sequence to obtain a video search result includes:

[0011] The second video frame sequence is retrieved based on the first video frame sequence to obtain the video retrieval result.

[0012] In a possible implementation manner of the first aspect, the searching the second video frame sequence based on the first video frame sequence to obtain the video retrieval result includes:

[0013] For each of the first video frames, searching the second video frame sequence based on the human body motion in the first video frame to obtain a frame search result;

[0014] The video retrieval result is determined according to each of the frame retrieval results.

[0015] In a possible implementation manner of the first aspect, before searching the second video frame sequence based on the human body motion in the first video frame for each of the first video frames to obtain a frame search result, the method further includes:

[0016] For each of the first video frames, extracting a human body motion in the first video frame based on preset skeleton key points to obtain a first feature;

[0017] For each of the second video frames, extracting a human body motion in the second video frame based on the skeleton key points to obtain a second feature;

[0018] Correspondingly, for each of the first video frames, searching the second video frame sequence based on the human body motion in the first video frame to obtain a frame search result includes:

[0019] For each of the first video frames, the frame retrieval result of the first video frame is determined based on the similarity between the first feature and each of the second features.

[0020] In a possible implementation manner of the first aspect, determining, for each of the first video frames, the frame retrieval result of the first video frame based on a similarity between the first feature and each of the second features includes:

[0021] For each of the first video frames, determining a posture similarity between the first video frame and each of the second video frames according to the first feature and the second feature;

[0022] Constructing a directed weight graph based on each of the posture similarities corresponding to each of the first video frames, wherein the weights in the directed weight graph are determined according to the posture similarities, and the posture similarities are negatively correlated with the weights;

[0023] Using the shortest path algorithm to solve the directed weighted graph to obtain the shortest path;

[0024] The frame retrieval result of each of the first video frames is determined based on the shortest path.

[0025] In a possible implementation manner of the first aspect, determining the video retrieval result according to each of the frame retrieval results includes:

[0026] Determine a candidate video frame sequence from the video to be retrieved according to the first and last two target frames, and determine a reference video frame sequence from the reference action video according to the first and last two first video frames, wherein the target frame is a second video frame that matches the first video frame as indicated by the frame retrieval result;

[0027] Performing an alignment process on the candidate video frame sequence based on the reference video frame sequence to obtain the aligned candidate video frame sequence, wherein the number of video frames in the aligned candidate video frame sequence is the same as the number of video frames in the reference video frame sequence;

[0028] The video retrieval result is determined according to the aligned candidate video frame sequence.

[0029] In a possible implementation manner of the first aspect, determining the first video frame sequence according to the reference action video includes:

[0030] Determining a plurality of reference frames containing a human body according to the reference action video to obtain a reference frame sequence;

[0031] Determine a starting frame of a sliding window, wherein the length of the sliding window is greater than 1;

[0032] Determine whether the similarities between the reference frames in the sliding window are greater than or equal to a similarity threshold;

[0033] In the case where it is determined that each of the similarities is greater than or equal to the similarity threshold, determining a first video frame based on the reference frame within the sliding window, and determining a new starting frame based on the next reference frame outside the sliding window;

[0034] Based on the new starting frame, the steps of judging whether the similarities between the reference frames in the sliding window are greater than or equal to a similarity threshold and determining the new starting frame are performed until the judgment of all the reference frames is completed.

[0035] In a second aspect, an embodiment of the present application provides a video retrieval device, including:

[0036] A first video frame sequence acquisition module is used to determine a first video frame sequence according to a reference action video, wherein the reference action video is a video showing human body actions, and the first video frame sequence includes a plurality of first video frames containing human body actions, and all the first video frames contain a plurality of human body actions;

[0037] The retrieval module is used to retrieve the video to be retrieved based on the first video frame sequence to obtain a video retrieval result, wherein the video retrieval result is used to indicate whether there is a video frame sequence matching the first video frame sequence in the video to be retrieved.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the video retrieval method described in the first aspect when executing the computer program.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video retrieval method described in the first aspect are implemented.

[0040] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the video retrieval method described in the first aspect above.

[0041] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0042] In the embodiment of the present application, multiple first video frames are determined from the reference action video to obtain a first video frame sequence, and then the video to be retrieved is retrieved based on the first video frame sequence to retrieve whether there is a video frame sequence matching the first video frame sequence in the video to be retrieved. There is no need to perform video retrieval based on the entire reference action video, which can effectively reduce the amount of data processing for video retrieval, thereby reducing the computing resources required in the retrieval process and reducing the computing power requirements for the device, improving the efficiency of video retrieval, and facilitating practical applications. At the same time, the first video frame contains video frames of human body movements, and all first video frames contain multiple human body movements, that is, the first video frame sequence can reflect different human body movements. Retrieving the video to be retrieved based on each first video frame in the first video frame sequence can reduce the impact of missing some repeated human body movements in the video to be retrieved on the video retrieval of the action stream, thereby ensuring the video retrieval effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0044] Figure 1 It is a flowchart of a video retrieval method provided by an embodiment of the present application;

[0045] Figure 2is a schematic diagram of a first video frame sequence provided in an embodiment of the present application;

[0046] Figure 3 is a schematic diagram of a skeleton key point provided in an embodiment of the present application;

[0047] Figure 4 It is a structural schematic diagram of a directed weight graph provided in an embodiment of the present application;

[0048] Figure 5 is a schematic diagram of a reference frame sequence provided in an embodiment of the present application;

[0049] Figure 6 is a schematic diagram of the structure of a video retrieval device provided in an embodiment of the present application;

[0050] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0052] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0053] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0054] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0055] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0056] Embodiment 1:

[0057] Figure 1 A schematic diagram of a video retrieval method provided in an embodiment of the present application is shown, and is described in detail as follows:

[0058] Step S101 : determining a first video frame sequence according to a reference action video, wherein the first video frame sequence includes a plurality of first video frames containing human body actions, and all the first video frames contain a plurality of human body actions.

[0059] The above-mentioned reference action video frames are used to display standard human body actions. The method for obtaining the reference action video can be set according to the actual application scenario and is not specifically limited here.

[0060] As an example, when the above-mentioned video retrieval method is applied to a video action evaluation scenario, the reference action video may be an automatically generated standard action video or an artificially recorded standard action video.

[0061] As another example, when the above-mentioned video retrieval method is applied to an action video retrieval scenario, the reference action video can be an action video input by the user, or an action video automatically generated based on a text description input by the user, or an action video that the user is currently watching or has watched in the past.

[0062] Since the amount of video data is usually large, directly performing video retrieval based on the entire video usually requires a large amount of computing resources. Therefore, in order to reduce the computing resources required for video retrieval, after obtaining the reference action video, you can first select a video frame containing human body movements from the reference action video as the first video frame to obtain a first video frame sequence containing multiple first video frames.

[0063] It should be noted that the embodiments of the present application are directed to a continuous stream reference action video, that is, the reference action video shows a series of human body movements, and this series of human body movements usually includes multiple human body movements. Therefore, in the embodiments of the present application, all the first video frames contain multiple human body movements, that is, among all the first video frames, the human body movements in at least two first video frames are different human body movements.

[0064] It should be noted that, in some embodiments, the human body motions in two or more consecutive first video frames in the first video frame sequence may be the same human body motions or human body motions whose similarity is greater than or equal to a similarity threshold.

[0065] It should also be pointed out that, in order to ensure the accuracy of video retrieval, the first video frame sequence obtained in the embodiment of the present application is a time sequence, and the order of each first video frame in the first video frame sequence is determined according to their order in the reference action video.

[0066] For example, assuming that the first video frame sequence includes the following five first video frames in sequence: frame A, frame B, frame C, frame D and frame E. Assuming that the human body actions in frames A, B, D and E are all action M1, then the human body action in frame C will not be action M1, but other actions (such as action M2). At this time, all first video frames contain action M1 and action M2.

[0067] In some embodiments, taking into account the case where human body motion changes slowly, some first video frames may be video frames that are continuous in time and have a high degree of human body motion similarity. Therefore, in order to further reduce the amount of data processing for video retrieval, all video frames containing human body motion (assuming they are called human body video frames) can be first determined from the reference action video, and then multiple first video frames can be determined from all human body video frames through clustering, random sampling, or stratified sampling methods, without taking all human body video frames as first video frames, thereby reducing the amount of data in the first video frame sequence, that is, reducing the amount of data processing for subsequent video retrieval.

[0068] Optionally, when determining a video frame containing human body movement from a reference action video, the video frame corresponding to the reference action video can be first obtained through frame segmentation processing, and then the video frame containing human body movement can be determined from each video frame through target recognition algorithm or manual recognition, so as to obtain the required first video frame.

[0069] Optionally, when determining multiple first video frames based on a reference video frame sequence to obtain a first video frame sequence consisting of multiple first video frames, the multiple first video frames can be determined by similarity comparison or clustering, etc., which can be specifically set according to the actual application scenario, and the embodiment of the present application does not impose any specific restrictions on this.

[0070] For example, assuming that multiple first video frames are determined by similarity comparison, and assuming that there are 7 human body video frames, in the process of determining the first video frame, the first human body video frame is used as the first first video frame (i.e. Figure 2After frame A in the figure, the next human body video frame is taken as the current human body video frame, and the similarity between the next human body video frame and the reference video frame (which can be the previous human body video frame or the most recently determined first video frame) is compared; if it is determined that the similarity between the current human body video frame and the reference video frame is less than the similarity threshold, the current human body video frame can be taken as the first video frame, and then the next human body video frame can be taken as the current human body video frame for further judgment; if it is determined that the similarity between the current human body video frame and the reference video frame is greater than or equal to the similarity threshold, the next human body video frame can be taken as the current human body video frame for further judgment.

[0071] like Figure 2 As shown, assuming that the similarity between the fourth human body video frame and the first first video frame (i.e., frame A) is less than the similarity threshold, the fourth human body video frame can be used as the second first video frame (i.e., Figure 2 ), assuming that the similarity between the sixth human body video frame and the second first video frame (i.e., frame B) is less than the similarity threshold, the sixth human body video frame is used as the third first video frame (i.e., Figure 2 Assuming that the similarity between the seventh human body video frame and frame C is less than the similarity threshold, the seventh human body video frame is used as the fourth first video frame (i.e. Figure 2 Frame D).

[0072] In an embodiment of the present application, multiple video frames containing human body movements are determined from a reference sample video as first video frames. Subsequently, video retrieval only needs to be performed based on a first video frame sequence containing multiple first video frames, without the need to perform retrieval based on the entire reference action video, which can reduce the amount of data processing in the video retrieval process.

[0073] Step S102: searching the video to be searched based on the first video frame sequence to obtain a video search result, wherein the video search result is used to indicate whether there is a video frame sequence matching the first video frame sequence in the video to be searched.

[0074] In some embodiments, taking into account the different levels of different people in application scenarios such as action evaluation, the human body movements contained in the videos to be evaluated (i.e., the videos to be retrieved) may be quite different from the standard movements. Therefore, when retrieving the video to be retrieved based on the first video frame, the video to be retrieved may also be retrieved based on the first video frame sequence. A video frame sequence matching one or more first video frames in the first video frame sequence may be retrieved and used as a video frame sequence matching the first video frame. It can be understood that the video retrieval result may include multiple video frame sequences matching the first video frame sequence.

[0075] In some embodiments, the content included in the video retrieval results can be determined according to the type of actual application scenario. For example, for an application scenario of the action evaluation type, the video retrieval results may include a video frame sequence that matches the first video frame sequence and an evaluation result (such as a specific score), and the evaluation result may be determined according to the similarity between the reference action video and the matching video frame sequence, or according to the similarity between the first video frame sequence and the matching video frame sequence.

[0076] In the embodiment of the present application, multiple first video frames are determined from the reference action video to obtain a first video frame sequence, and then the video to be retrieved is retrieved based on the first video frame sequence to retrieve whether there is a video frame sequence matching the first video frame sequence in the video to be retrieved. There is no need to perform video retrieval based on the entire reference action video, which can effectively reduce the amount of data processing for video retrieval, thereby reducing the computing resources required in the retrieval process and reducing the computing power requirements for the device, improving the efficiency of video retrieval, and facilitating practical applications. At the same time, the first video frame contains video frames of human body movements, and all first video frames contain multiple human body movements, that is, the first video frame sequence can reflect different human body movements in the reference action video. Retrieving the video to be retrieved based on each first video frame in the first video frame sequence can reduce the degree of influence of the situation of missing some repeated human body movements in the video to be retrieved on the video retrieval of the action stream, thereby ensuring the video retrieval effect.

[0077] In some embodiments, before the above step S102, the method further includes:

[0078] Determine a second video frame sequence according to the video to be retrieved, wherein the second video frames included in the second video frame sequence are video frames containing human body movements in the video to be retrieved;

[0079] Correspondingly, the video to be searched is searched based on the first video frame sequence to obtain a video search result, including:

[0080] The second video frame sequence is retrieved based on the first video frame sequence to obtain the video retrieval result.

[0081] When searching a video to be retrieved, the human body movements in the video to be retrieved that match the human body movements shown in the reference action video are mainly retrieved. Therefore, in order to further reduce the amount of data processing for video retrieval, before searching, the video frame containing the human body movement can be determined from the video to be retrieved as the second video frame, and then the second video frame sequence is determined based on the obtained second video frame.

[0082] It is understandable that the second video frame sequence obtained according to the video to be retrieved is also a time sequence. If there is no video frame containing human body movements in the video frames to be retrieved, there is no need to retrieve the video to be retrieved.

[0083] Optionally, the video frame containing the human body action may be determined from each video frame corresponding to the video to be retrieved by a target recognition algorithm or manual recognition, so as to obtain the required second video frame.

[0084] In the embodiment of the present application, the video frames in the first video frame sequence and the second video frame sequence are both video frames containing human bodies. Directly searching based on these two video frame sequences can effectively reduce the amount of computation for video retrieval, thereby effectively reducing the required computing resources and improving retrieval efficiency. In addition, directly searching the second video frame sequence composed of the second video frames containing human body movements can reduce the impact of some irrelevant content in the video to be retrieved, which is conducive to improving the video retrieval effect.

[0085] In some embodiments, the above-mentioned searching the above-mentioned second video frame sequence based on the above-mentioned first video frame sequence to obtain the above-mentioned video search result includes:

[0086] For each of the first video frames, the second video frame sequence is searched based on the human body motion in the first video frame to obtain a frame search result.

[0087] The video retrieval result is determined based on each of the frame retrieval results.

[0088] To improve the retrieval effect of video retrieval, during the retrieval process, for each first video frame in the first video frame sequence, the frame retrieval result of each first video frame can be first determined from the second video frame sequence, and then the video retrieval result can be determined based on each frame search.

[0089] Optionally, when searching the second video frame sequence according to the human body motion in the first video frame, the similarity between the human body motion in the first video frame and the human body motion in the second video frame can be analyzed by a similarity algorithm such as cosine similarity or Jaccard similarity coefficient, and then the frame retrieval result of the first video frame is determined according to the similarity. Alternatively, the frame retrieval result of the first video frame can be determined according to the similarity between the first video frame input by the user and each second video frame in the second video frame sequence, and the frame retrieval result of the first video frame can also be determined according to the second video frame selected by the user.

[0090] In some embodiments, the frame retrieval result can be used to indicate whether there is a second video frame matching the first video frame in the second video frame sequence, and when there is a second video frame matching the first video frame, indicate the second video frame matching the first video frame.

[0091] In other embodiments, in application scenarios such as action evaluation, the video to be retrieved is usually a video to be evaluated that contains corresponding human body actions, but the action standards for different users are different. There is a situation where the human body actions in the video to be retrieved and the human body actions in the reference action video have a low degree of match, which may result in the inability to retrieve video frames with high similarity for evaluation. Therefore, in order to meet the needs of actual application scenarios, when retrieving the second video frame sequence according to the human body actions in the first video frame, one or more second video frames matching the first video frame will be determined from the second video frame sequence to ensure that the video frame sequence matching the reference action video can be determined according to the retrieval results of each frame, that is, to ensure that the human body actions in the video to be retrieved can be evaluated. At this time, the frame retrieval result is used to indicate the second video frame matching the first video frame, and the video retrieval result is used to indicate the video frame sequence matching the reference action video.

[0092] As an example, when determining a second video frame matching a first video frame, the first N (greater than 0) second video frames in descending order of similarity may be used as the second video frames matching the first video frame.

[0093] As another example, when determining a second video frame that matches a first video frame, a second video frame whose similarity is greater than a judgment threshold (such as 0.75) can be used as the second video frame that matches the first video frame. If there is no second video frame whose similarity is greater than the judgment threshold, the video frame with the greatest similarity is used as the second video frame that matches the first video frame. The method for determining the second video frame that matches the first video frame can be set according to the actual application scenario and is not specifically limited here.

[0094] In some embodiments, the frame retrieval result may include a frame identifier, which is the identifier of the second video frame that matches the first video frame. The frame identifier indicates the second video frame that matches the first video frame, so that the second video frame that matches the first video frame can be quickly and accurately determined based on the frame retrieval result.

[0095] In some embodiments, before searching the second video frame sequence for each of the first video frames based on the human body motion in the first video frame to obtain the frame search result, the method further includes:

[0096] For each of the first video frames, human body movements in the first video frame are extracted based on preset skeleton key points to obtain a first feature.

[0097] For each of the second video frames, human body movements in the second video frame are extracted based on the skeleton key points to obtain a second feature.

[0098] Correspondingly, for each of the first video frames, the second video frame sequence is searched based on the human body motion in the first video frame to obtain a frame search result, including:

[0099] For each of the first video frames, the frame retrieval result of the first video frame is determined based on the similarity between the first feature and each of the second features.

[0100] Specifically, considering that changes in human body motion mainly depend on changes in the human skeleton, in order to better represent human body motion in video frames, multiple skeleton key points can be set in advance, and then the human body motion in the first video frame is extracted based on the set skeleton key points to obtain the first feature, and the human body motion in the second video frame is extracted based on the set skeleton key points to obtain the second feature.

[0101] Optionally, when the first feature and the second feature are obtained based on the skeleton key point extraction, the first video frame and the second video frame can be input into a pre-trained feature extraction network (such as a recurrent neural network, an Ensem-NN network, or a graph neural network, etc.), and the feature extraction network performs feature extraction processing on the first video frame and the second video frame based on the skeleton key point to obtain the required first feature and second feature. Alternatively, the first feature and the second feature can also be extracted by manual analysis or other methods, which are not specifically limited here.

[0102] It can be understood that the first feature and the second feature can reflect the human body movement through the bones corresponding to each skeleton key point, and the skeleton is the line between two skeleton key points. Figure 3 As shown, P1, P2 and P3 are three different bone key points, the line between P1 and P2 is bone a1, and the line between P2 and P3 is bone a2.

[0103] At the same time, for the first video frame and the second video frame, human body movements are extracted based on the same skeletal key points, so that when the frame retrieval results are determined according to the first feature corresponding to the first video frame and the second feature corresponding to each second video frame, the similarity between the first feature and the second feature can be better analyzed, which is conducive to improving the accuracy of the frame retrieval results.

[0104] Optionally, the above-mentioned skeletal key points can be determined based on the video retrieval type (such as action evaluation or video retrieval, etc.) and / or the type of human body action in the reference action video (such as push-up type fitness action or grasping type fitness action), and no specific limitation is made here.

[0105] In some embodiments, when calculating the similarity between a first feature and a second feature, the similarity between each bone feature can be first calculated based on the first feature and the second feature by methods such as cosine distance or Pearson correlation coefficient to obtain the bone similarity, and then the overall similarity between the first feature and the second feature can be calculated based on each bone similarity.

[0106] In other embodiments, considering that different application scenarios may have different requirements for the matching degree of actions of different parts of the human body, for example, in the push-type action guidance of fitness action guidance, the similarity of the actions of the arms and shoulders may be required to be higher. Therefore, in order to improve the accuracy of the calculated similarity, the weights corresponding to each skeletal feature can be set. When calculating the overall similarity between the first feature and the second feature according to the similarity of each skeletal feature (i.e., the above-mentioned skeletal similarity), a weighted calculation is performed according to each skeletal similarity and its corresponding weight to obtain the final overall similarity. Among them, the weights corresponding to different skeletal features can be determined according to the video retrieval type (such as action evaluation or video retrieval, etc.) or the type of human body action in the reference action video (such as push-type fitness action or dance action), and no specific restrictions are made here.

[0107] In an embodiment of the present application, human body movements in video frames are extracted based on preset skeletal key points to obtain first features and second features that can better represent human body movements. Further, retrieval based on these first features and second features can improve the accuracy of the frame retrieval results obtained, which is conducive to improving the accuracy of the final video retrieval.

[0108] In some embodiments, for each of the first video frames, searching the second video frame sequence based on the posture feature of the human body in the first video frame to obtain a frame search result includes:

[0109] For each of the first video frames, the posture similarity between the first video frame and each of the second video frames is determined according to the posture features of the human body in the first video frame and the posture features of the human body in the second video frames.

[0110] A directed weight graph is constructed based on each of the posture similarities corresponding to each of the first video frames, the weights in the directed weight graph are determined according to the posture similarities, and the posture similarities are negatively correlated with the weights.

[0111] The shortest path algorithm is used to solve the above directed weighted graph to obtain the shortest path.

[0112] The frame retrieval results of each of the first video frames are determined based on the shortest path.

[0113] The above shortest path algorithm is a method for finding the shortest path from one node to another node in a graph. The above shortest path algorithm includes but is not limited to the Dijkstra algorithm, the Bellman-Ford algorithm, and the Floyd-Warshall algorithm.

[0114] In order to reduce the difficulty of calculation, in an embodiment of the present application, for each first video frame, the similarity (such as cosine similarity or Euclidean distance, etc.) is first calculated based on its corresponding first feature and the second feature corresponding to each second video frame to obtain the posture similarity between the first video frame and each second video frame.

[0115] After obtaining the posture similarity between each first video frame and each second video frame, a directed weight graph can be constructed according to each posture similarity. In an embodiment of the present application, when constructing a directed weight graph, a node in the directed weight graph can be determined according to the second video frame, and then the weight between two nodes in the directed weight graph can be determined according to the posture similarity between the second video frame and the first video frame.

[0116] like Figure 4 As shown, the first column of nodes in the directed weight graph includes w+1 nodes, corresponding to the 1st second video frame to the w+1th second video frame in the second video frame sequence; the second column of nodes also includes w+1 nodes, but they correspond to the 2nd second video frame to the w+2th second video frame in the second video frame sequence, and so on, the i-th row of nodes corresponds to the i-th second video frame to the w+i-th second video frame in the second video frame sequence, and the i+1-th row of nodes corresponds to the i+1-th second video frame to the w+i+1-th second video frame in the second video frame sequence.

[0117] Among them, the weighted directed edge points from the node of the previous column of nodes to the node of the next column of nodes, and the weight of the directed edge is determined according to the posture similarity between the previous node of the directed edge and the corresponding first video frame. Figure 4 As shown, a directed edge A12 points from node 1 (i.e., frame 1) in the first column to node 1 (i.e., frame 2) in the second column, indicating that frame 1 is matched with the first video frame A, and the weight of the directed edge is determined according to the posture similarity between frame 1 and the first video frame A; a directed edge A23 points from node 2 (i.e., frame 2) in the first column to node 2 (i.e., frame 3) in the second column, indicating that frame 2 is matched with the first video frame A, and the weight of the directed edge is determined according to the posture similarity between frame 2 and the first video frame A; a directed edge B23 points from node 1 (i.e., frame 2) in the second column to node 1 (i.e., frame 3) in the third column, indicating that frame 2 is matched with the first video frame B, and the weight of the directed edge B23 is determined according to the weight between frame 2 and the first video frame B.

[0118] Among them, in order to facilitate the solution of the directed weight graph, the posture similarity can be set to be negatively correlated with the weight, that is, the greater the posture similarity, the smaller the corresponding weight in the directed weight graph, so that when performing video retrieval, the directed weight graph can be directly solved by the shortest path algorithm to obtain the shortest path with the smallest weight sum. Since the weight and posture similarity in the directed weight graph are negatively correlated, that is, the greater the posture similarity, the smaller the weight, therefore, the shortest path with the smallest weight sum is also the path with the largest posture similarity sum. Based on the shortest path, the frame retrieval results of each first video frame are determined, and the retrieval of the second video frame sequence can be achieved while ensuring the maximization of the global optimal posture similarity.

[0119] In other embodiments, other optimal path algorithms such as the longest path algorithm or the gradient descent method can be used to solve the optimal path of the directed weight graph, and correspondingly, the corresponding relationship between the weight in the directed weight graph and the posture similarity can be set according to the actual application scenario. For example, when the longest path algorithm is used to solve the directed weight graph, the posture similarity and the weight of the directed weight graph can be positively correlated.

[0120] In an embodiment of the present application, a directed weight graph is constructed to represent the matching scheme between the first video frame sequence and the second video frame sequence, and the globally optimal shortest path is directly obtained through the shortest path algorithm, so that the frame retrieval results of each first video frame in the first video frame sequence can be quickly and accurately determined according to the shortest path, which is conducive to improving the efficiency and accuracy of video retrieval.

[0121] In some embodiments, determining the video retrieval result according to each of the frame retrieval results includes:

[0122] A candidate video frame sequence is determined from the video to be retrieved according to the first and last two target frames, and a reference video frame sequence is determined from the reference action video according to the first and last two first video frames, wherein the target frame is the second video frame that matches the first video frame as indicated by the frame retrieval result.

[0123] The candidate video frame sequence is aligned based on the reference video frame sequence to obtain the aligned candidate video frame sequence, wherein the number of video frames in the aligned candidate video frame sequence is the same as the number of video frames in the reference video frame sequence.

[0124] The video retrieval result is determined based on the aligned candidate video frame sequence.

[0125] To further improve the accuracy of video retrieval, after obtaining the frame retrieval results of each first video frame, we can first determine the second video frame that matches each first video frame based on the frame retrieval results, that is, determine each target frame, and then quickly determine the continuous video frames containing all the target frames from the video to be retrieved based on the first and last two target frames to obtain a candidate video frame sequence, and then determine the final video retrieval result based on the candidate video frame sequence.

[0126] Since the first video frames included in the first video frame sequence may not be all the video frames in the reference action video, in order to ensure the comprehensiveness of the video retrieval, the continuous video frames including all the first video frames are determined from the reference action video according to the first and last two first video frames in the first video frame sequence to obtain the reference video frame sequence. Then, the candidate video frame sequence is aligned with the reference video frame sequence so that the length of the candidate video frame sequence is aligned with the length of the reference video frame sequence, and then the final video retrieval result is determined based on the reference video frame sequence with the same length and the aligned candidate video frame sequence to ensure the accuracy of the video retrieval.

[0127] It can be understood that the reference video frame sequence is a sequence of continuous video frames between the video frames corresponding to the first and last first video frames in the reference action video, and the candidate video frame sequence is a sequence of continuous video frames between the video frames corresponding to the first and last target frames in the video to be retrieved.

[0128] Optionally, when performing alignment processing on the candidate video frame sequence based on the reference video frame sequence, the candidate video frame sequence may be subjected to video alignment processing, and the video alignment processing aligns the length of the candidate video frame sequence with that of the reference video frame sequence. For example, if the length of the reference video frame sequence is greater than the length of the candidate video frame sequence, the candidate video frame may be subjected to interpolation processing, that is, a new video frame is inserted into the candidate video frame sequence so that the number of video frames in the aligned candidate video frame sequence is kept consistent with the number of video frames in the reference video frame sequence.

[0129] Similarly, if the length of the reference video frame sequence is smaller than the length of the candidate video frame sequence, the candidate video frame sequence can be subjected to frame extraction, that is, one or more video frames are extracted from the candidate video frame sequence so that the number of video frames in the aligned candidate video frame sequence remains consistent with the number of video frames in the reference video frame sequence.

[0130] In some embodiments, in order to further improve the reliability of video retrieval, when the candidate video frame sequence is aligned based on the reference video frame sequence, the candidate video frame sequence can also be subjected to frame alignment to obtain an aligned candidate video frame sequence, and the frame alignment refers to making the number of video frames between any two adjacent target frames in the candidate video frame sequence the same as the number of video frames between the corresponding two first video frames in the reference video frame sequence. Through the above processing, the candidate video frame sequence is strictly aligned with the reference video frame sequence, and then, the final video retrieval result is determined based on the strictly aligned candidate video frame sequence, which can reduce the impact of missing some actions in the video frame to be retrieved on the video retrieval effect, and improve the comprehensiveness and robustness of the video retrieval.

[0131] For example, assuming that the first video frame sequence includes the following four first video frames: (A, B, C, D), according to the frame retrieval results of the four first video frames, it is determined that the target frames include: (a, b, c, d) in sequence.

[0132] Assume that the reference video frame sequence determined from the reference action video according to the first and last two first video frames (i.e., A and D) in the first video frame sequence is: (A, m1, m2, B, m3, m4, C, m5, D), and the candidate video frame sequence determined from the video to be retrieved according to the first and last two target frames is: (a, n1, b, n2, n3, n4, c, n5, d).

[0133] Among them, the number of video frames between two adjacent target frames: a and b is 1, and the number of video frames between the corresponding two first video frames: A and B is 2, then a video frame can be inserted between a and b; the number of video frames between two adjacent target frames: b and c is 3, and the number of video frames between the corresponding two first video frames: B and C is 2, then a video frame can be removed between b and c. Finally, the aligned candidate video frame sequence is obtained: (a, n1, n2, b, n3, n4, c, n5, d).

[0134] Optionally, the new video frame inserted between two adjacent target frames may be a video frame generated based on the two adjacent target frames, so that the newly inserted video frame can be well connected with the previous and next target frames.

[0135] In an embodiment of the present application, a candidate video frame sequence including all target frames is obtained from the video to be retrieved based on the two target frames located at the first and last positions in the target frames matching each first video frame, and a reference video frame sequence including all first video frames is obtained from the reference action video based on the first and last two first video frames. After the candidate video frame sequence is aligned with the reference video frame sequence, the final video retrieval result is determined based on the aligned candidate video frame sequence, which can reduce the impact of situations such as missing some actions in the video frames to be retrieved on the video retrieval, and is conducive to improving the comprehensiveness and robustness of the video retrieval.

[0136] In some embodiments, the above step S101 includes:

[0137] A plurality of reference frames including a human body are determined according to the reference action video to obtain a reference frame sequence.

[0138] A start frame of a sliding window is determined, wherein the length of the sliding window is greater than 1.

[0139] It is determined whether the similarities between the reference frames in the sliding window are greater than or equal to a similarity threshold.

[0140] When it is determined that each of the above similarities is greater than or equal to the above similarity threshold, the above first video frame is determined based on the above reference frame in the above sliding window, and the new above starting frame is determined according to the next above reference frame outside the above sliding window.

[0141] Based on the new starting frame, the steps of judging whether the similarity between the reference frames in the sliding window is greater than or equal to a similarity threshold and determining the new starting frame are performed until the judgment of all the reference frames is completed.

[0142] Optionally, when determining a reference frame containing a human body based on a reference action video, a video frame containing a human body can be found from each video frame of the reference action video through methods such as a human recognition model or manual analysis, and used as a reference frame. No specific restrictions are imposed on the specific implementation method of determining the reference frame containing a human body.

[0143] Considering that the human body movements in the reference action video are smooth and continuous, the similarity between any two adjacent video frames in a time period is usually high. Directly obtaining the first video frame from it by clustering or other methods may omit more of its video frames, which has a greater impact on the video retrieval results of application scenarios such as action evaluation. Therefore, in order to ensure the accuracy of the video retrieval results, when determining the first video frame sequence based on the reference action video, a sliding window with a preset length (the preset length is greater than 1) can be set, and the first video frame can be determined based on the video frames in the sliding window. It can be understood that when the length of the sliding window is greater than 1, the sliding window includes at least two reference frames.

[0144] When determining multiple first video frames from a reference frame sequence, the starting frame of the sliding window may be determined first. Figure 5 In the reference frame sequence shown, assuming that the length of the sliding window is 3, the starting frame determined for the first time is frame 1, and its corresponding sliding window is called window 1. In the first judgment, it is determined whether the similarities between frames 1, 2 and 3 in window 1 are all greater than or equal to the similarity threshold (such as 0.8).

[0145] Optionally, during the judgment process, it can be determined whether the similarity between frame 2 and frame A (the current starting frame) and the similarity between frame 3 and the current starting frame are both greater than or equal to a similarity threshold, or it can be determined whether the similarity between frame A and frame 2, frame A and frame 3, and frame 2 and frame 3 are similar. The judgment rules in the judgment process can be set according to actual application requirements, and the embodiments of the present application do not impose specific restrictions on this.

[0146] If it is determined that the similarities between frame A, frame 2 and frame 3 in the judgment window 1 are greater than or equal to the similarity threshold, a first reference frame (such as the starting frame A, or a random sample) can be determined from the multiple reference frames of window 1 (i.e., frame A, frame 2 and frame 3), and the next reference frame outside the current sliding window is used as the new starting frame. Figure 5 , the next reference frame outside window 1: frame 4 is taken as the new starting frame. Correspondingly, the reference frames contained in the new sliding window (currently window 2) include: frame 4, frame 5 and frame 6.

[0147] A similarity judgment is performed on the reference frames in window 2. Assume that the judgment rule is: whether the similarities between other reference frames in the sliding window and the current starting frame are greater than or equal to the similarity threshold, and assume that it is judged that the similarity between frame 5 and frame 4 (the current starting frame) is greater than the similarity threshold, and the similarity between frame 6 and frame 4 is less than the similarity threshold, at this time, at least one first video frame can be determined according to the current sliding window.

[0148] As an example, when it is determined that the similarities between the reference frames in the sliding window are not all greater than or equal to the similarity threshold, a first video frame can be determined based on the current starting frame, and a first video frame can be determined based on the first reference frame (i.e., the above-mentioned frame 6, assumed to be called the target reference frame) whose similarity to the current starting frame is less than the similarity threshold, and then the next reference frame of the target reference frame (i.e., Figure 57) as a new starting frame, and based on the sliding window corresponding to the new starting frame, continue to determine whether the similarity thresholds between the reference frames in the sliding window are greater than or equal to the similarity threshold, and then determine the first video frame and the new starting frame in a corresponding manner according to the determination result until all reference frames are determined. Figure 5 As shown, through the above determination steps, the following six first video frames can be obtained: frame 1, frame 4, frame 6, frame 7, frame 8 and frame 9.

[0149] As another example, when it is determined that the similarities between the reference frames in the sliding window are not all greater than or equal to the similarity threshold, it can be determined whether the similarity between the current starting frame and the most recently determined first video frame is greater than or equal to the similarity threshold; then if the similarity is greater than or equal to the threshold, the current starting frame can be used as the first video frame, and if the similarity is less than the similarity threshold, a first video frame will not be determined based on the current starting frame. At the same time, a first video frame is determined based on the first reference frame whose similarity to the current starting frame is less than or equal to the similarity threshold.

[0150] In an embodiment of the present application, a sliding window method is used to determine multiple first video frames from a reference frame sequence, and at least one first video frame is determined in each sliding window. When the number of consecutive reference frames with high similarity in the reference frame sequence is greater than the sliding window length, two or more first video frames can be determined therefrom, rather than using only one first video frame to fully represent multiple consecutive reference frames with high similarity. This allows the first video frame sequence to better represent the continuous flow of human body movements in the reference action video, which is beneficial to improving the accuracy of video retrieval.

[0151] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0152] Embodiment 2:

[0153] Corresponding to the video retrieval method described in the above embodiment, Figure 6 A structural block diagram of a video retrieval device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0154] Reference Figure 6 The device comprises: a first video frame sequence acquisition module 61 and a retrieval module 62.

[0155] A first video frame sequence acquisition module 61 is used to determine a first video frame sequence according to a reference action video, wherein the reference action video is a video showing human body actions, and the first video frame sequence includes a plurality of first video frames containing human body actions, and all the first video frames contain a plurality of human body actions;

[0156] The retrieval module 62 is used to retrieve the video to be retrieved based on the first video frame sequence to obtain a video retrieval result, wherein the video retrieval result is used to indicate whether there is a video frame sequence matching the first video frame sequence in the video to be retrieved.

[0157] In the embodiment of the present application, multiple first video frames are determined from the reference action video to obtain a first video frame sequence, and then the video to be retrieved is retrieved based on the first video frame sequence to retrieve whether there is a video frame sequence matching the first video frame sequence in the video to be retrieved. There is no need to perform video retrieval based on the entire reference action video, which can effectively reduce the amount of data processing for video retrieval, thereby reducing the computing resources required in the retrieval process and reducing the computing power requirements for the device, improving the efficiency of video retrieval, and facilitating practical applications. At the same time, the first video frame contains video frames of human body movements, and all first video frames contain multiple human body movements, that is, the first video frame sequence can reflect different human body movements in the reference action video. Retrieving the video to be retrieved based on each first video frame in the first video frame sequence can reduce the degree of influence of the situation of missing some repeated human body movements in the video to be retrieved on the video retrieval of the action stream, thereby ensuring the video retrieval effect.

[0158] In some embodiments, the video retrieval device further includes:

[0159] The second video frame sequence acquisition module is used to determine a second video frame sequence according to the video to be retrieved, wherein the second video frames included in the second video frame sequence are video frames containing human body movements in the video to be retrieved.

[0160] Correspondingly, the above-mentioned retrieval module 62 is used for:

[0161] The second video frame sequence is retrieved based on the first video frame sequence to obtain the video retrieval result.

[0162] In some embodiments, the retrieval module 62 includes:

[0163] The frame retrieval unit is used to retrieve the second video frame sequence for each of the first video frames based on the human body movements in the first video frames to obtain a frame retrieval result.

[0164] The video retrieval result acquisition unit is used to determine the video retrieval result according to each of the frame retrieval results.

[0165] In some embodiments, the video retrieval device further includes:

[0166] The first feature extraction module is used to extract human body movements in each of the first video frames based on preset skeleton key points to obtain a first feature.

[0167] The second feature extraction module is used to extract human body movements in each of the second video frames based on the skeleton key points to obtain a second feature.

[0168] Correspondingly, the above-mentioned retrieval module 62 includes:

[0169] The frame retrieval result determining unit is used to determine the frame retrieval result of the first video frame based on the similarity between the first feature and each of the second features for each of the first video frames.

[0170] In some embodiments, the retrieval module 62 further includes:

[0171] The posture similarity calculation unit is used to determine the posture similarity between the first video frame and each second video frame according to the first feature and the second feature for each of the first video frames.

[0172] A directed weight graph construction unit is used to construct a directed weight graph based on each of the above-mentioned posture similarities corresponding to each of the above-mentioned first video frames, and the weights in the above-mentioned directed weight graph are determined according to the above-mentioned posture similarities, and the above-mentioned posture similarities are negatively correlated with the above-mentioned weights.

[0173] The shortest path solving unit is used to solve the directed weighted graph using the shortest path algorithm to obtain the shortest path.

[0174] The result determination unit is used to determine the frame retrieval result of each of the first video frames based on the shortest path.

[0175] In some embodiments, the retrieval module 62 includes:

[0176] A frame sequence acquisition unit is used to determine a candidate video frame sequence from the above-mentioned video to be retrieved based on the first and last two target frames, and to determine a reference video frame sequence from the above-mentioned reference action video based on the first and last two first video frames, wherein the above-mentioned target frame is a second video frame that matches the above-mentioned first video frame as indicated by the above-mentioned frame retrieval result.

[0177] The alignment unit is used to align the candidate video frame sequence based on the reference video frame sequence to obtain the aligned candidate video frame sequence, wherein the number of video frames in the aligned candidate video frame sequence is the same as the number of video frames in the reference video frame sequence.

[0178] The video retrieval unit is used to determine the video retrieval result according to the aligned candidate video frame sequence.

[0179] In some embodiments, the first video frame sequence acquisition module 61 includes:

[0180] The reference frame sequence acquisition unit is used to determine a plurality of reference frames containing a human body according to the reference action video to obtain a reference frame sequence.

[0181] The start frame determination unit is used to determine the start frame of the sliding window, and the length of the sliding window is greater than 1.

[0182] The judging unit is used to judge whether the similarities between the reference frames in the sliding window are greater than or equal to a similarity threshold.

[0183] The first video frame determination unit is used to determine the first video frame based on the reference frame in the sliding window when it is determined that each of the above similarities is greater than or equal to the above similarity threshold, and to determine a new starting frame based on the next reference frame outside the sliding window.

[0184] The loop unit is used to execute the steps of judging whether the similarities between the reference frames in the sliding window are greater than or equal to the similarity threshold based on the new starting frame to determine the new starting frame until the judgment of all the reference frames is completed.

[0185] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0186] Embodiment three:

[0187] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 7 As shown, the electronic device 7 of this embodiment includes: at least one processor 70 ( Figure 7 Only one processor is shown in the figure), a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, wherein the processor 70 implements the steps of any of the above-mentioned method embodiments when executing the computer program 72.

[0188] The electronic device 7 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Figure 7 It is only an example of the electronic device 7 and does not constitute a limitation on the electronic device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0189] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0190] In some embodiments, the memory 71 may be an internal storage unit of the electronic device 7, such as a hard disk or memory of the electronic device 7. In other embodiments, the memory 71 may also be an external storage device of the electronic device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7. Further, the memory 71 may also include both an internal storage unit and an external storage device of the electronic device 7. The memory 71 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0191] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0192] An embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above-mentioned method embodiments when executing the computer program.

[0193] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0194] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0195] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0196] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0197] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0198] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0199] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A video retrieval method, characterized in that: include: Determine a first video frame sequence according to a reference action video, wherein the reference action video is a video showing human body actions, and the first video frame sequence includes a plurality of first video frames containing human body actions, and all the first video frames contain a plurality of human body actions; The video to be retrieved is retrieved based on the first video frame sequence to obtain a video retrieval result, where the video retrieval result is used to indicate whether there is a video frame sequence matching the first video frame sequence in the video to be retrieved.

2. The video retrieval method according to claim 1, characterized in that: Before retrieving the video to be retrieved based on the first video frame sequence to obtain the video retrieval result, the method further includes: Determine a second video frame sequence according to the video to be retrieved, wherein the second video frame included in the second video frame sequence is a video frame containing a human body action in the video to be retrieved; Correspondingly, searching the video to be searched based on the first video frame sequence to obtain a video search result includes: The second video frame sequence is retrieved based on the first video frame sequence to obtain the video retrieval result.

3. The video retrieval method according to claim 2, characterized in that: The retrieving the second video frame sequence based on the first video frame sequence to obtain the video retrieval result includes: For each of the first video frames, searching the second video frame sequence based on the human body motion in the first video frame to obtain a frame search result; The video retrieval result is determined according to each of the frame retrieval results.

4. The video retrieval method according to claim 3, characterized in that: Before searching the second video frame sequence based on the human body motion in each of the first video frames to obtain the frame search result, the method further includes: For each of the first video frames, extracting a human body motion in the first video frame based on preset skeleton key points to obtain a first feature; For each of the second video frames, extracting a human body motion in the second video frame based on the skeleton key points to obtain a second feature; Correspondingly, for each of the first video frames, searching the second video frame sequence based on the human body motion in the first video frame to obtain a frame search result includes: For each of the first video frames, the frame retrieval result of the first video frame is determined based on the similarity between the first feature and each of the second features.

5. The video retrieval method according to claim 4, characterized in that: The step of determining, for each of the first video frames, the frame retrieval result of the first video frame based on the similarity between the first feature and each of the second features comprises: For each of the first video frames, determining a posture similarity between the first video frame and each of the second video frames according to the first feature and the second feature; Constructing a directed weight graph based on each of the posture similarities corresponding to each of the first video frames, wherein the weights in the directed weight graph are determined according to the posture similarities, and the posture similarities are negatively correlated with the weights; Using the shortest path algorithm to solve the directed weighted graph to obtain the shortest path; The frame retrieval result of each of the first video frames is determined based on the shortest path.

6. The video retrieval method according to claim 3, characterized in that: Determining the video retrieval result according to each of the frame retrieval results includes: Determine a candidate video frame sequence from the video to be retrieved according to the first and last two target frames, and determine a reference video frame sequence from the reference action video according to the first and last two first video frames, wherein the target frame is a second video frame that matches the first video frame as indicated by the frame retrieval result; Performing an alignment process on the candidate video frame sequence based on the reference video frame sequence to obtain the aligned candidate video frame sequence, wherein the number of video frames in the aligned candidate video frame sequence is the same as the number of video frames in the reference video frame sequence; The video retrieval result is determined according to the aligned candidate video frame sequence.

7. The video retrieval method according to any one of claims 1 to 6, characterized in that: The determining of the first video frame sequence according to the reference action video comprises: Determining a plurality of reference frames containing a human body according to the reference action video to obtain a reference frame sequence; Determine a starting frame of a sliding window, wherein the length of the sliding window is greater than 1; Determine whether the similarities between the reference frames in the sliding window are greater than or equal to a similarity threshold; In the case where it is determined that each of the similarities is greater than or equal to the similarity threshold, determining a first video frame based on the reference frame within the sliding window, and determining a new starting frame based on the next reference frame outside the sliding window; Based on the new starting frame, the steps of judging whether the similarities between the reference frames in the sliding window are greater than or equal to a similarity threshold and determining the new starting frame are performed until the judgment of all the reference frames is completed.

8. A video retrieval device, characterized in that: include: A first video frame sequence acquisition module is used to determine a first video frame sequence according to a reference action video, wherein the reference action video is a video showing human body actions, and the first video frame sequence includes a plurality of first video frames containing human body actions, and all the first video frames contain a plurality of human body actions; The retrieval module is used to retrieve the video to be retrieved based on the first video frame sequence to obtain a video retrieval result, wherein the video retrieval result is used to indicate whether there is a video frame sequence matching the first video frame sequence in the video to be retrieved.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images