Video feature extraction method, video satisfaction evaluation method and related device

Abstract

The present disclosure provides a video feature extraction method, a video satisfaction evaluation method and related devices, which are related to the technical fields of artificial intelligence such as deep learning, object recognition, natural language processing and big data. The method comprises: encapsulating continuous image frames constituting a target video into continuous image frame groups by using a sliding window technology; regarding each image frame in the continuous image frame groups as a different node, and establishing hyper-edges between multiple nodes with node similarity to obtain an initial sub-hypergraph; performing a hypergraph convolution operation on the multiple nodes connected by the hyper-edges to obtain updated nodes; performing weighted average processing on image features corresponding to each updated node to obtain an updated sub-hypergraph; and performing weighted average processing on image features of the updated sub-hypergraphs corresponding to different continuous image frame groups to obtain a global image feature of the target video. The method can extract more comprehensive and representative video features.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing, in particular to the field of artificial intelligence technologies such as deep learning, object recognition, natural language processing, big data, and more particularly to a video feature extraction and video satisfaction evaluation method, and a corresponding device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] The waterfall video stream continuously pushes a large number of videos, mainly short videos, to the user in a streaming manner, so that the user can perform operations such as attention, likes, comments, forwarding, and attention in the process of browsing the video.

[0003] Although the user experience evaluation of the video information stream application using the questionnaire survey method can obtain the overall satisfaction of the user with the application, the conventional scheme based only on the video image features cannot be used to more finely reflect the satisfaction of the user with the video stream because the image features provided by the conventional scheme are relatively simple.

[0004] Therefore, how to extract more representative features from the video is a problem to be solved by those skilled in the art. SUMMARY

[0005] The present disclosure provides a video feature extraction and video satisfaction evaluation method, device, electronic equipment, computer readable storage medium and computer program product.

[0006] In a first aspect, the present disclosure provides a video feature extraction method, comprising: encapsulating continuous image frames constituting a target video into continuous image frame groups by using a sliding window technology; regarding each image frame in the continuous image frame groups as a different node, and establishing hyper-edges between multiple nodes having node similarity to obtain an initial sub-hypergraph; performing a hypergraph convolution operation on the multiple nodes connected by the hyper-edges to obtain updated nodes; performing a weighted average on image features corresponding to each updated node to obtain an updated sub-hypergraph; and performing a weighted average processing on image features of the updated sub-hypergraphs corresponding to different continuous image frame groups to obtain global image features of the target video.

[0007] In a second aspect, the embodiments of the present disclosure provide a video feature extraction device, comprising: a continuous image frame group packaging unit configured to package continuous image frames constituting a target video into continuous image frame groups by using a sliding window technology; an initial sub-hypergraph construction unit configured to take each image frame in the continuous image frame groups as a different node, and establish a hyperedge between multiple nodes with node similarity, to obtain an initial sub-hypergraph; a hypergraph convolution operation unit configured to perform a hypergraph convolution operation on multiple nodes connected by a hyperedge, to obtain updated nodes; a sub-hypergraph updating unit configured to perform a weighted average on image features corresponding to each updated node, to obtain an updated sub-hypergraph; and a continuous image frame group updating unit configured to perform a weighted average processing on image features of the updated sub-hypergraph corresponding to different continuous image frame groups, to obtain a global image feature of the target video.

[0008] In a third aspect, the embodiments of the present disclosure provide a video satisfaction evaluation method, comprising: obtaining video features of a target video, the video features comprising a global image feature, the global image feature being obtained according to the video feature extraction method described in the first aspect; determining a video satisfaction corresponding to the video features by using a preset video satisfaction evaluation model; wherein the video satisfaction evaluation model is used to represent a corresponding relationship between different video features and different satisfactions, and the video satisfaction evaluation model is trained based on training samples constituted by satisfaction annotations on video elements including images of sample videos.

[0009] In a fourth aspect, the embodiments of the present disclosure provide a video satisfaction evaluation device, comprising: a video feature obtaining unit configured to obtain video features of a target video, the video features comprising a global image feature; wherein the global image feature is obtained according to the video feature extraction device described in the second aspect; and a video satisfaction determining unit configured to determine a video satisfaction corresponding to the video features by using a preset video satisfaction evaluation model; wherein the video satisfaction evaluation model is used to represent a corresponding relationship between different video features and different satisfactions, and the video satisfaction evaluation model is trained based on training samples constituted by satisfaction annotations on video elements including images of sample videos.

[0010] In a fifth aspect, the embodiments of the present disclosure provide an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can implement the video feature extraction method described in the first aspect or the video satisfaction evaluation method described in the third aspect when executed.

[0011] In a sixth aspect, the embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer instructions, which, when executed by a computer, enable the computer to implement the video feature extraction method described in the first aspect or the video satisfaction evaluation method described in the third aspect.

[0012] In a seventh aspect, the embodiments of the present disclosure provide a computer program product comprising a computer program, which, when executed by a processor, enables the video feature extraction method described in the first aspect or the video satisfaction evaluation method described in the third aspect.

[0013] The video feature extraction scheme provided by the embodiments of the present disclosure first acquires different continuous image frame groups composed of different continuous image frames constituting a target video through a sliding window, then for each continuous image frame group, each image frame is regarded as a node, and a hyperedge connection is established between similar nodes, then the image features of the nodes covered by the hyperedge are processed through hyperedge framework and hypergraph convolution processing, so that the processed image features can better reflect the high-order time sequence relationship between adjacent video frames, and the finally obtained global image features can better represent the target video.

[0014] The video satisfaction evaluation scheme provided by the embodiments of the present disclosure, for a target video to be evaluated for satisfaction, by acquiring the video features of the target video including global image features, can obtain more comprehensive and representative video features of the video, and with the help of a preset video satisfaction evaluation model, can output more accurate video satisfaction evaluation.

[0015] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0016] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the following drawings:

[0017] Figure 1 is an exemplary system architecture to which the present disclosure can be applied;

[0018] Figure 2 is a flowchart of a video feature extraction method provided by the embodiments of the present disclosure;

[0019] Figure 3 is a flowchart of a method for acquiring continuous image frame groups using a sliding window provided by the embodiments of the present disclosure;

[0020] Figure 4A flowchart of a method for establishing a hyperedge between similar nodes according to an embodiment of the present disclosure;

[0021] Figure 5 A flowchart of a method for hypergraph convolution operation according to an embodiment of the present disclosure;

[0022] Figure 6 A flowchart of a method for hypergraph convolution operation on a group of continuous image frames according to an embodiment of the present disclosure;

[0023] Figure 7 A flowchart of a method for splicing global image features and text features to obtain comprehensive video features according to an embodiment of the present disclosure;

[0024] Figure 8 A schematic diagram of a method for extracting text features based on text information according to an embodiment of the present disclosure;

[0025] Figure 9 A flowchart of a method for video satisfaction evaluation according to an embodiment of the present disclosure;

[0026] Figure 10 A flowchart of a method for video satisfaction evaluation based on comprehensive video features according to an embodiment of the present disclosure;

[0027] Figure 11 A structural block diagram of a video feature extraction device according to an embodiment of the present disclosure;

[0028] Figure 12 A structural block diagram of a video satisfaction evaluation device according to an embodiment of the present disclosure;

[0029] Figure 13 A structural schematic diagram of an electronic device suitable for executing a video feature extraction method and / or a video satisfaction evaluation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help in understanding. These should be considered in their context only as illustrative. Therefore, those of ordinary skill in the art should recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and structures are omitted in the following description for the sake of clarity and conciseness. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0031] The collection, storage, use, processing, transmission, provision and disclosure of the user personal information in the technical solutions of the present disclosure comply with relevant laws and regulations and do not violate public order and good customs.

[0032] Figure 1 An exemplary system architecture 100 is shown that can apply embodiments of the video feature extraction method, video satisfaction evaluation method, device, electronic device and computer readable storage medium of the present application.

[0033] As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0034] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, 103 and the server 105 can be installed with various applications for realizing information communication between them, such as video feature extraction applications, satisfaction labeling applications, model training applications, satisfaction evaluation applications, etc.

[0035] The terminal devices 101, 102, 103 and the server 105 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned electronic devices, which can be implemented as multiple software or software modules, or as a single software or software module, which is not specifically limited here. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, which is not specifically limited here.

[0036] The server 105 can provide various services through various built-in applications. Taking a video feature extraction application as an example, the server 105 can achieve the following effects when running the video feature extraction application: first, receiving a target video specified by the terminal 101, 102 or 103 through the network 104; then, encapsulating continuous image frames constituting the target video into a continuous image frame group by using a sliding window technology; next, regarding each image frame in the continuous image frame group as a different node, and establishing a hyperedge between multiple nodes with node similarity to obtain an initial sub-hypergraph; next, performing a hypergraph convolution operation on multiple nodes connected by the hyperedge to obtain updated nodes; continuing, performing a weighted average on image features corresponding to each updated node to obtain an updated sub-hypergraph; finally, performing a weighted average on image features of the updated sub-hypergraph corresponding to different continuous image frame groups to obtain global image features of the target video.

[0037] Further, after the server 105 extracts the video features of the target video through the above application, the server 105 can also evaluate the user satisfaction of the target video based on the extracted video features. For example, the server 105 can achieve the following effects when running a satisfaction evaluation application: first, obtaining video features of the target video, including global image features; then, determining a video satisfaction corresponding to the video features by using a preset video satisfaction evaluation model, the video satisfaction evaluation model being used to represent the corresponding relationship between different video features and different satisfactions, and the video satisfaction evaluation model being trained based on training samples constituted by satisfaction labels of video elements including images of sample videos.

[0038] Since a lot of computing resources and strong computing capability are required for training the video satisfaction evaluation model and performing video feature extraction, the video feature extraction method and the video satisfaction evaluation method provided in the subsequent embodiments of the present application are generally executed by the server 105 with strong computing capability and more computing resources, and accordingly, the video feature extraction device and the video satisfaction evaluation device are generally arranged in the server 105. However, it should also be noted that when the terminal device 101, 102 or 103 also has computing capability and computing resources that meet the requirements, the terminal device 101, 102 or 103 can also complete the above operations by using the video feature extraction application or the video satisfaction evaluation application installed thereon, and then output the same result as the server 105. Accordingly, the video feature extraction device and the video satisfaction evaluation device can also be arranged in the terminal device 101, 102 or 103. In this case, the example system architecture 100 can also not include the server 105 and the network 104.

[0039] Of course, the server used to train the video satisfaction evaluation model can be different from the server used to call the trained video satisfaction evaluation model. In particular, the video satisfaction evaluation model trained via the server 105 can also be adapted to a lightweight video satisfaction evaluation model suitable for being placed in the terminal device 101, 102, 103 through model distillation, that is, according to the actual requirement of recognition accuracy, the lightweight video satisfaction evaluation model in the terminal device 101, 102, 103 can be flexibly selected for use, or the more complex video satisfaction evaluation model in the server 105 can be selected for use.

[0040] It should be understood that Figure 1 The number of terminal devices, networks and servers in the above embodiment is only illustrative. Any number of terminal devices, networks and servers can be provided according to the implementation needs.

[0041] Reference can be made to Figure 2 , Figure 2 A flowchart of a video feature extraction method provided by the embodiment of the present disclosure is shown in FIG. 2, wherein the flow 200 includes the following steps:

[0042] Step 201: encapsulating the continuous image frames constituting the target video into continuous image frame groups by using the sliding window technology.

[0043] This step aims to encapsulate the continuous image frames within the window coverage into continuous image frame groups by using the sliding window created by the execution subject of the video feature extraction method (for example, the server 105 shown in FIG. 1). Figure 1 The sliding window created by using the sliding window technology is used to encapsulate the continuous image frames within the window coverage into continuous image frame groups in a sliding manner.

[0044] In the above embodiment, the continuous image frames are all image frames constituting the target video, and it is assumed that the window coverage of the sliding window is 10 frames, which means that every 10 continuous image frames will be encapsulated into 1 continuous image frame group, and the sliding of the sliding window is usually started from the first frame. Taking a target video with a total of 100 frames as an example, if the sliding distance is 1 frame each time, 90 different continuous image frame groups can be obtained, wherein the first continuous image frame group is from the first frame to the tenth frame, and the 90th continuous image frame group is from the 91st frame to the 100th frame.

[0045] It should be noted that, in order not to be omitted, the sliding distance of the sliding window is usually 1 frame, but it is not excluded that the sliding window uses other frame numbers as the sliding distance in some scenarios.

[0046] Step 202: regarding each image frame in the continuous image frame group as a different node, and establishing hyper-edges between multiple nodes with node similarity to obtain an initial sub-hypergraph.

[0047] On the basis of step 201, this step aims to take each image frame in the continuous image frame group as a different node, and establish a hyperedge between multiple nodes with node similarity, to construct an initial sub-hypergraph connected by multiple nodes with similarity through hyperedges.

[0048] A hypergraph is a generalized graph structure, containing a set of nodes and hyperedges. Unlike simple Figure 1 A simple edge contains two nodes, while a hyperedge can contain any number of nodes. Compared with a graph structure that can only model the connection relationship between two levels, a hypergraph has a significant advantage in modeling complex relationships. For example, in a hypergraph, nodes represent researchers, and hyperedges represent articles co-signed by multiple authors. In this way, the closer the cooperation between researchers, the more hyperedges there are between them.

[0049] Hypergraph learning is closely related to graph learning, because a hypergraph is a generalized graph structure. Similar to graph learning, learning on a hypergraph can also be seen as information propagation on a hypergraph structure. In this sense, graph learning is a special case of hypergraph learning. Unlike graph learning, hypergraph learning models explore high-order relationships in data, so it has better results.

[0050] Hypergraph learning is derived from the popular graph neural network in recent years, which uses edges between nodes to transfer and share information, achieving the effect of feature enhancement. However, unlike traditional graph neural networks, each edge (i.e. hyperedge) in a hypergraph can connect any number of nodes, and the model can make nodes share more information through hyperedges.

[0051] This disclosure formally takes advantage of this feature of hypergraph and applies it to a group of continuous image frames with temporal association, in order to determine the commonality of image features corresponding to the nodes played by image frames, so as to better extract video image features containing high-order temporal features.

[0052] Step 203: performing a hypergraph convolution operation on the multiple nodes connected by the hyperedge to obtain updated nodes;

[0053] On the basis of step 202, this step aims to perform a hypergraph convolution operation on the multiple nodes connected by the hyperedge by the above execution subject to obtain updated nodes.

[0054] Among them, the hypergraph convolution operation is to aggregate the image features of each node connected by the same hyperedge to the hyperedge to form a hyperedge feature, and the hyperedge feature will be dispersed to each node again to update the original image feature of the node, to obtain an updated node with updated node features.

[0055] Step 204: performing a weighted average on the image features corresponding to each updated node to obtain an updated sub-hypergraph;

[0056] Based on step 203, this step aims to have the aforementioned executing entity perform a weighted average of the image features corresponding to each updated node, so that the image features in a continuous group of image frames are aggregated to obtain the updated sub-hypergraph.

[0057] Step 205: Perform weighted averaging on the image features of the updated sub-hypergraphs corresponding to different consecutive image frame groups to obtain the global image features of the target video.

[0058] Based on step 204, this step aims to have the aforementioned executing entity perform weighted averaging on the image features of the updated sub-hypergraphs corresponding to different consecutive image frame groups, so that the image features in different consecutive image frame groups are also aggregated, thereby finally obtaining the global image features of the target video.

[0059] The video feature extraction method provided in this embodiment first obtains different continuous image frame groups composed of different continuous image frames constituting the target video through a sliding window. Then, for each continuous image frame group, each image frame is regarded as a node, and a hyperedge connection is established between similar nodes. Next, the image features of the hyperedge-covered nodes are processed through hyperedge framework and hypergraph convolution processing, so that the processed image features can better reflect the high-order temporal relationship between adjacent video frames, and the final global image features can better represent the target video.

[0060] Please refer to Figure 3 , Figure 3 A flowchart illustrating a method for acquiring consecutive image frame groups using a sliding window, as provided in this disclosure embodiment, is shown below. Figure 2 Step 201 in process 200 provided a specific implementation. Other steps in process 200 are not adjusted; a new complete embodiment is obtained by replacing step 201 with the specific implementation provided in this embodiment. Process 300 includes the following steps:

[0061] Step 301: Determine the set of consecutive image frames that constitute the target video;

[0062] Step 302: Establish a target sliding window covering a preset number of consecutive image frames;

[0063] Building upon step 301, this step aims to establish a target sliding window covering a preset number of consecutive image frames by the aforementioned executing entity. This preset number can be a fixed value such as 5 frames or 10 frames, or it can be a value calculated proportionally based on the total number of frames in the target video; no specific limitation is made here.

[0064] Step 303: Pack the consecutive image frames covered by the target sliding window each time into different consecutive image frame groups.

[0065] On the basis of step 302, the present step aims to encapsulate, by the above-mentioned execution subject, the continuous image frames respectively covered by each sliding target sliding window into different continuous image frame groups. That is, all the continuous image frames between the first frame and the last frame covered after each sliding are encapsulated into a continuous image frame group, and since the first frame and the last frame after each sliding are different, the continuous image frame group obtained after each encapsulation is also necessarily different.

[0066] The present embodiment specifically provides an implementation scheme of how to split the target video into multiple continuous image frame groups through the sliding window by means of steps 301-303, which has high feasibility and execution efficiency.

[0067] Please refer to Figure 4 , Figure 4 The flowchart of the method for establishing a hyperedge between similar nodes provided by the present embodiment, that is, the specific implementation provided by the hyperedge establishment part in step 202 of the flowchart 200 shown in Figure 2 does not make any adjustment to other steps in the flowchart 200, and the specific implementation provided by the present embodiment is obtained in the form of replacing step 202 to obtain a new complete embodiment. The flowchart 400 includes the following steps:

[0068] Step 401: regarding each image frame in the continuous image frame group as a different node;

[0069] Step 402: for each node, determining K similar nodes closest to the node by using a K-Nearest Neighbor algorithm, K being a positive integer greater than 1;

[0070] Step 403: establishing a hyperedge between the node and the K similar nodes.

[0071] The present embodiment provides an implementation scheme for establishing a hyperedge connection based on the K-Nearest Neighbor algorithm to determine similar nodes, that is, by respectively calculating the K nodes closest to each node by using the K-Nearest Neighbor algorithm, the K nodes closest to the node are regarded as similar nodes, and a hyperedge connection is established between the nodes with similarity. Different from other implementation schemes for determining node similarity, the K-Nearest Neighbor algorithm has a lower computational complexity and a better use effect.

[0072] Please refer to Figure 5 , Figure 5 The flowchart of the method for establishing a hyperedge between similar nodes provided by the present embodiment, that is, the specific implementation provided by the hyperedge establishment part in step 202 of the flowchart 200 shown in Figure 2The step 203 in the flow 200 shown provides a specific implementation, and other steps in the flow 200 are not adjusted, and the specific implementation provided by the embodiment is used to replace the step 203 to obtain a new complete embodiment. The flow 500 includes the following steps:

[0073] Step 501: aggregate image features of nodes connected by a hyperedge to the hyperedge to obtain hyperedge features;

[0074] Step 502: distribute the hyperedge features to the nodes connected by the hyperedge;

[0075] Step 503: control each node to update its image features according to the hyperedge features distributed to itself to obtain an updated node after image feature update.

[0076] The embodiment of the disclosure specifically provides a process of how to perform a hypergraph convolution operation through steps 501-503, that is, first aggregate image features of nodes connected by the same hyperedge to the hyperedge to form hyperedge features, and then distribute the hyperedge features to each node again to update the original image features of the node, and finally obtain an updated node with updated node features.

[0077] To further deepen the understanding of how to extract global image features containing high-order temporal features by means of a hypergraph framework, a hyperedge, and a hypergraph convolution operation, the embodiment further provides a scheme as shown in Figure 6 The structural schematic diagram is shown in

[0078] First, the image frames of a complete target video are separated and the image features of each frame are extracted, and then a sliding window with a length of 10 frames is set, and a hypergraph is constructed by the sliding window for every 10 frames of images. The entire video can be modeled as multiple sub-hypergraphs. Next, each sub-hypergraph uses a hypergraph convolution operation to update features, the purpose being to aggregate video frame features connected by the same hyperedge to the hyperedge to form hyperedge features, and the hyperedge features will be distributed to each node again to update node features. Finally, the features between each group of video frames are weighted and averaged to generate the final global features.

[0079] In this embodiment, the initial image features of each image frame in the continuous image frame group consisting of 10 continuous image frames are taken as nodes in the hypergraph, and three nodes close to each node in the high-dimensional space are found by the K-neighbor algorithm, and a hyperedge is established among the three nodes. Finally, in this way, the hypergraph corresponding to the entire data set is constructed. After the construction of the hypergraph is completed, the hypergraph convolution layer in the model is used to update the node features (i.e. the high-order features of the nodes connected by the same hyperedge are first aggregated to the corresponding hyperedge, and then the hyperedge distributes the obtained high-order features to each connected node to complete the feature update). Finally, the updated node features are aggregated to obtain the global image features corresponding to the target video.

[0080] On the basis of any of the above embodiments, in addition to the global image features extracted from the image frames by using the above hypergraph framework, the remaining video features including text features can be additionally added to further enrich the comprehensiveness and representativeness of the comprehensive video features.

[0081] Figure 7 A flowchart of a method for splicing global image features and text features to obtain comprehensive video features provided by the embodiments of the present disclosure, the flow 700 includes the following steps:

[0082] Step 701: In response to the target video being cut from the waterfall video stream, text information is identified from each image frame;

[0083] The waterfall video stream has the presentation characteristic of being able to continuously present diversified video information, that is, even if the user constantly switches, other videos that the user can watch can still be continuously provided to the user, also known as streaming video or short video stream.

[0084] When the target video is cut from the waterfall video stream, this step aims to identify text information from each image frame by the above execution subject.

[0085] Among them, the identified text information includes first text information embedded in the image frame and second text information in the video evaluation parameter located on the upper layer of the image frame; wherein the video evaluation parameter includes: the number of likes, the number of collections, the number of comments, the number of forwards, whether to follow and the number of followers. It should be understood that the second text information is set or provided by the video application based on the characteristics of the waterfall video stream, and is dedicated to presenting the viewing and feedback information of the video.

[0086] For example, see Figure 8 The left side is the original presentation picture and the right side is the identified text information.

[0087] For the recognition of text information, the target detection can be first performed on the video frame, the trained yolov5 model (a model for target detection) accurately recognizes the image area where the text field is located and records the label corresponding to each area, and then all the recognized areas are cropped and input into the text recognition module. The text recognition result and the label of the image area are output in the form of key-value pair and saved as a JSON (JavaScript Object Notation, JS object notation) file. The types of text fields that need to be recognized are as follows: author name, mounting, resource type, video description, POI (Point Of Interest, interest point), whether to follow, like amount, collection amount, forwarding amount, and comment amount.

[0088] Step 702: generating text features of the target video according to the text information and the characteristics of the waterfall video stream;

[0089] On the basis of step 701, this step aims to generate text features of the target video according to the text information and the characteristics of the waterfall video stream by the above-mentioned execution subject.

[0090] Step 703: splicing the text features and the global image features to obtain the comprehensive video features of the target video.

[0091] On the basis of step 702, this step aims to splice the text features and the global image features by the above-mentioned execution subject to obtain the comprehensive video features of the target video.

[0092] Specifically, before splicing, the text features and the global image features also need to be normalized respectively, so that they have equal measurement standards after splicing. In the case where there is no explicit requirement for the splicing order, the splicing order can be selected by itself. At the same time, if there is a requirement for the length of the spliced comprehensive video features, length compression or lengthening processing can also be performed after splicing.

[0093] The above-mentioned embodiments illustrate from various aspects how to extract representative video features from the target video. In the case where the representative video features can be extracted from the target video, in combination with the actual application scenario of the video satisfaction evaluation, the present disclosure also provides a scheme for evaluating the video satisfaction of the target video based on the video features of the video. Figure 9 Specifically, a scheme for evaluating the video satisfaction of a target video based on the video features of the video is provided, and the flow 900 includes the following steps:

[0094] Step 901: obtaining the video features of the target video, including global image features;

[0095] Step 902: determining the video satisfaction corresponding to the video features through a preset video satisfaction evaluation model.

[0096] The video satisfaction evaluation model is used to represent the corresponding relationship between different video features and different satisfactions, and the video satisfaction evaluation model is trained based on a training sample constituted by satisfaction labels of video elements including images of sample videos. Further, in order to increase the interpretability of the satisfaction labels as much as possible, the satisfaction labels in the training sample can also include satisfaction reasons, so as to learn more implicit knowledge from the satisfaction reasons.

[0097] That is, the embodiment provides a scheme for determining video satisfaction based on video features through steps 901-902. When the video features only include global image features, it is equivalent to judging the video satisfaction based on only the global image features. When the video features are comprehensive video features in which the global image features are spliced with text features, it is equivalent to jointly judging the video satisfaction based on the global image features and the text features.

[0098] For a target video to be evaluated for satisfaction, the video features including global image features of the target video are obtained, so that more comprehensive and representative video features of the video can be obtained, and a more accurate video satisfaction evaluation can be output by means of the preset video satisfaction evaluation model.

[0099] To further understand the scheme provided by the embodiment, the disclosure also provides a full-process scheme diagram as shown in Figure 10

[0100] On the basis of obtaining the global image features through the hypergraph framework, the video text features extracted through target detection and text recognition are also obtained. As shown in Figure 8 Among all the recognized text features, the mounting, resource type, POI, attention, author name, content / description are integrated and used for feature extraction by using a BERT (a model for processing natural language) pre-training model, while the numerical features such as the number of likes, the number of comments, the number of forwards, and the number of collections are normalized and spliced with other feature vectors generated by BERT, and then spliced with the video global features in the early stage to form the final video fusion features. Finally, the fusion features of all videos in the data set are output by a multi-layer perception to obtain a video evaluation score.

[0101] Further reference is made to Figure 11 and Figure 12 As an implementation of the method shown in the above figures, the disclosure provides a video feature extraction device embodiment and a video satisfaction evaluation device embodiment, respectively. The video feature extraction device embodiment corresponds to the video feature extraction method embodiment shown in Figure 2 , and the video satisfaction evaluation device embodiment corresponds to the video satisfaction evaluation method embodiment shown in Figure 9 ​The illustrated video satisfaction evaluation method corresponds to the given embodiment. The aforementioned device can be specifically applied to various electronic devices.

[0102] 0 as Figure 11 As shown, the video feature extraction device 1100 of this embodiment may include: a connecting...

[0103] The system includes a continuous image frame group encapsulation unit 1101, an initial sub-hypergraph construction unit 1102, a hypergraph convolution operation unit 1103, a sub-hypergraph update unit 1104, and a continuous image frame group update unit 1105. The continuous image frame group encapsulation unit 1101 is configured to use a sliding window technique to encapsulate the target video...

[0104] The continuous image frames are encapsulated into a continuous image frame group; the initial sub-hypergraph construction unit 1102 is configured to treat each image frame in the continuous image frame group as a different node, and establish hyperedges between multiple nodes with node similarity to obtain an initial sub-hypergraph; the hypergraph convolution operation unit 1103 is configured to perform hypergraph convolution operations on multiple nodes connected by hyperedges to obtain updated nodes; the sub-hypergraph update unit 1104 is configured to process the image features corresponding to each updated node.

[0105] The weighted average is used to obtain the updated sub-hypergraph; the continuous image frame group update unit 1105 is configured to 0 to perform weighted average processing on the image features of the updated sub-hypergraphs corresponding to different continuous image frame groups to obtain the global image features of the target video.

[0106] In this embodiment, the video feature extraction device 1100 includes: a continuous image frame group encapsulation unit 1101, an initial sub-hypergraph construction unit 1102, a hypergraph convolution operation unit 1103, and a sub-hypergraph re-encapsulation unit 1104.

[0107] The specific processing of the new unit 1104 and the continuous image frame group update unit 1105, and the resulting technical effects, can be found in [reference 5]. Figure 2 The relevant descriptions of steps 201-205 in the corresponding embodiments will not be repeated here.

[0108] In some optional implementations of this embodiment, the continuous image frame group encapsulation unit 1101 can be further configured as follows:

[0109] Determine the set of consecutive image frames that constitute the target video;

[0110] 0. Create a target sliding window that covers a preset number of consecutive image frames;

[0111] Each time the target sliding window is slid, the consecutive image frames covered by it are grouped into different consecutive image frame groups.

[0112] In some optional implementations of the embodiment, the initial sub-hypergraph construction unit 1102 comprises a hyperedge construction sub-unit configured to establish a hyperedge between a plurality of nodes with node similarity, and the hyperedge construction sub-unit can be further configured to:

[0113] For each node, a K-Nearest Neighbor algorithm is used to determine K similar nodes closest to the node, K being a positive integer greater than 1;

[0114] A hyperedge is established between the node and the K similar nodes.

[0115] In some optional implementations of the embodiment, the hypergraph convolution operation unit 1103 can be further configured to:

[0116] The image features of the nodes connected by the hyperedge are aggregated to the hyperedge to obtain a hyperedge feature;

[0117] The hyperedge feature is dispersed to the nodes connected by the hyperedge;

[0118] The nodes are controlled to update their image features according to the hyperedge feature dispersed to themselves, to obtain updated nodes after the image feature update.

[0119] In some optional implementations of the embodiment, the video feature extraction apparatus 1100 can further comprise:

[0120] A text information recognition unit configured to recognize text information from each image frame in response to the target video being captured from a waterfall video stream;

[0121] A text feature generation unit configured to generate a text feature of the target video according to the text information and the characteristics of the waterfall video stream;

[0122] A feature splicing unit configured to splice the text feature and the global image feature to obtain a comprehensive video feature of the target video.

[0123] In some optional implementations of the embodiment, the text information comprises first text information embedded in the image frame and second text information in a video evaluation parameter located on an upper layer of the image frame; and the video evaluation parameter comprises a number of likes, a number of collections, a number of comments, a number of forwards, whether to follow, and a number of followers.

[0124] As Figure 12As shown, the video satisfaction evaluation device 1200 in this embodiment can include a video feature acquisition unit 1201 and a video satisfaction determination unit 1202. The video feature acquisition unit 1201 is configured to acquire video features of a target video, including global image features. The global image features are obtained by the video feature extraction device 1100. The video satisfaction determination unit 1202 is configured to determine the video satisfaction corresponding to the video features by a preset video satisfaction evaluation model. The video satisfaction evaluation model is used to represent the corresponding relationship between different video features and different satisfaction degrees. The video satisfaction evaluation model is trained based on training samples constituted by satisfaction labels of video elements, including images, of sample videos.

[0125] In this embodiment, the video feature acquisition unit 1201 and the video satisfaction determination unit 1202 in the video satisfaction evaluation device 1200 have specific processing and technical effects, which can correspond to the related description in the method embodiments, and will not be repeated here.

[0126] In some optional implementation manners of this embodiment, the satisfaction label in the training sample further includes a satisfaction reason.

[0127] This embodiment exists as a device embodiment corresponding to the above-mentioned method embodiment. The video feature extraction device provided in this embodiment first acquires different continuous image frame groups composed of different continuous image frames constituting a target video by using a sliding window. Then, for each continuous image frame group, each image frame is regarded as a node, and a hyperedge connection is established between similar nodes. Then, the image features of the nodes covered by the hyperedge are processed by using a hyperedge framework and a hypergraph convolution processing, so that the processed image features can better reflect the high-order time sequence relationship between adjacent video frames, and the finally obtained global image features can better represent the target video.

[0128] The video satisfaction evaluation device provided in this embodiment can acquire more comprehensive and representative video features of a target video by acquiring the video features of the target video, including global image features, and output more accurate video satisfaction evaluation by using a preset video satisfaction evaluation model.

[0129] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, which includes at least one processor and a memory connected with the at least one processor in communication. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to implement the video feature extraction method and / or the video satisfaction evaluation method described in any of the above embodiments.

[0130] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium storing computer instructions for enabling a computer to implement the video feature extraction method and / or the video satisfaction evaluation method described in any of the above embodiments when the computer instructions are executed.

[0131] The present disclosure provides a computer program product, which, when executed by a processor, can implement the video feature extraction method and / or the video satisfaction evaluation method described in any of the above embodiments.

[0132] Figure 13 A schematic block diagram of an example electronic device 1300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0133] As shown in Figure 13 The device 1300 includes a computing unit 1301 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. Various programs and data required for the operation of the device 1300 can also be stored in the RAM 1303. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other through a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0134] Various components in the device 1300 are connected to the I / O interface 1305, including an input unit 1306, such as a keyboard, a mouse, etc., an output unit 1307, such as various types of displays, speakers, etc., a storage unit 1308, such as a magnetic disk, an optical disk, etc., and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0135] The computing unit 1301 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit 1301 performs various methods and processes described above, such as the video feature extraction method and / or the video satisfaction evaluation method. For example, in some embodiments, the video feature extraction method and / or the video satisfaction evaluation method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1308. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded onto the RAM 1303 and executed by the computing unit 1301, one or more steps of the video feature extraction method and / or the video satisfaction evaluation method described above can be performed. Alternatively, in other embodiments, the computing unit 1301 can be configured to perform the video feature extraction method and / or the video satisfaction evaluation method by any other appropriate means, such as by means of firmware.

[0136] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0137] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0138] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0139] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0140] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0141] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) services.

[0142] The video feature extraction scheme of the embodiments of the present disclosure first acquires different continuous image frame groups composed of different continuous image frames constituting a target video through a sliding window, then for each continuous image frame group, each image frame is regarded as a node, and a hyperedge connection is established between similar nodes, then the image features of the hyperedge covering nodes are processed through a hyperedge framework and a hypergraph convolution processing, so that the processed image features can better reflect the high-order time sequence relationship between adjacent video frames, and the finally obtained global image features can better represent the target video.

[0143] The video satisfaction evaluation scheme provided by the embodiments of the present disclosure can obtain more comprehensive and representative video features of the target video by acquiring the video features of the target video including global image features, and output more accurate video satisfaction evaluation by means of a preset video satisfaction evaluation model.

[0144] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

[0145] The above detailed description does not limit the scope of the disclosure. Various modifications, combinations, sub-combinations and alternatives can be made to the detailed description. Any modification, equivalent replacement and improvement etc. made within the spirit and principle of the disclosure shall be included in the scope of the disclosure.

Claims

1. A method for evaluating video satisfaction, comprising: The sliding window technique creates a sliding window that encapsulates consecutive image frames that constitute the target video within the window's coverage into a group of consecutive image frames in a sliding manner. The sliding distance of the sliding window is 1 frame, and the groups of image frames have a temporal relationship. Each image frame in the continuous image frame group is treated as a different node, and a hyperedge is established between multiple nodes with node similarity to obtain an initial sub-hypergraph. Performing a hypergraph convolution operation on multiple nodes connected by the hyperedge to obtain updated nodes includes: aggregating the image features of each node connected by the hyperedge to the hyperedge to obtain hyperedge features; distributing the hyperedge features to each node connected by the hyperedge; and controlling each node to update its own image features according to the hyperedge features distributed to it, thereby obtaining updated nodes with completed image feature updates. The updated sub-hypergraph is obtained by weighted averaging of the image features corresponding to each updated node. The image features of the updated sub-hypergraph corresponding to different consecutive image frame groups are weighted and averaged to obtain the global image features of the target video; Based on the video features of the target video, including the global image features, a preset video satisfaction evaluation model is used to determine the video satisfaction level. The video satisfaction evaluation model is used to characterize the correspondence between different video features and different satisfaction levels. The video satisfaction evaluation model is trained on training samples consisting of satisfaction annotations of video elements, including images, in the sample video.

2. The method according to claim 1, wherein, The sliding window created using the sliding window technique encapsulates consecutive image frames constituting the target video within the window's coverage into a group of consecutive image frames in a sliding manner, including: Determine the set of consecutive image frames that constitute the target video; Establish a target sliding window that covers a preset number of consecutive image frames; Each time the target sliding window is slid, the consecutive image frames covered by it are grouped into different consecutive image frame groups.

3. The method according to claim 1, wherein, The process of establishing a hyperedge between multiple nodes with node similarity includes: For each node, the K nearest neighbor algorithm is used to determine the K most similar nodes that are closest to the node, where K is a positive integer greater than 1; Establish the hyperedge between the node and the K similar nodes.

4. The method according to any one of claims 1-3, further comprising: In response to the target video being extracted from a waterfall-style video stream, text information is identified from each of the image frames; Based on the text information and the characteristics of the waterfall-style video stream, the text features of the target video are generated; By concatenating the text features and the global image features, the comprehensive video features of the target video are obtained.

5. The method according to claim 4, wherein, The text information includes: first text information embedded in the image frame and second text information located in the video evaluation parameters above the image frame; wherein, the video evaluation parameters include: number of likes, number of favorites, number of comments, number of reposts, whether followed, and number of followers.

6. The method according to claim 1, wherein, The satisfaction labels in the training samples also include the reasons for the satisfaction.

7. A video satisfaction evaluation device, comprising: A video feature acquisition unit is configured to acquire video features of a target video, including global image features; wherein the global image features are obtained through a video feature extraction device, the video feature extraction device comprising: The continuous image frame group encapsulation unit is configured to use a sliding window created by sliding window technology to encapsulate the continuous image frames constituting the target video within the window coverage into a continuous image frame group in a sliding manner. The sliding distance of the sliding window is 1 frame, and the image frame groups have temporal correlation. The initial sub-hypergraph construction unit is configured to treat each image frame in the continuous image frame group as a different node, and to establish hyperedges between multiple nodes with node similarity to obtain the initial sub-hypergraph. The hypergraph convolution operation unit is configured to perform a hypergraph convolution operation on multiple nodes connected by the hyperedge to obtain updated nodes, including: aggregating the image features of each node connected by the hyperedge to the hyperedge to obtain hyperedge features; distributing the hyperedge features to each node connected by the hyperedge; and controlling each node to update its own image features according to the hyperedge features distributed to it to obtain updated nodes with completed image feature updates. The sub-hypergraph update unit is configured to perform a weighted average of the image features corresponding to each of the updated nodes to obtain the updated sub-hypergraph; The continuous image frame group update unit is configured to perform weighted averaging of the image features of the updated sub-hypergraphs corresponding to different continuous image frame groups to obtain the global image features of the target video. The video satisfaction determination unit is configured to determine the video satisfaction corresponding to the video features using a preset video satisfaction evaluation model; wherein, the video satisfaction evaluation model is used to characterize the correspondence between different video features and different satisfaction levels, and the video satisfaction evaluation model is trained based on training samples consisting of satisfaction annotations of video elements, including images, in the sample videos.

8. The apparatus according to claim 7, wherein, The continuous image frame group encapsulation unit is further configured to: Determine the set of consecutive image frames that constitute the target video; Establish a target sliding window that covers a preset number of consecutive image frames; Each time the target sliding window is slid, the consecutive image frames covered by it are grouped into different consecutive image frame groups.

9. The apparatus according to claim 7, wherein, The initial subhypergraph construction unit includes a hyperedge construction subunit configured to establish hyperedges between multiple nodes with node similarity, and the hyperedge construction subunit is further configured to: For each node, the K nearest neighbor algorithm is used to determine the K most similar nodes that are closest to the node, where K is a positive integer greater than 1; Establish the hyperedge between the node and the K similar nodes.

10. The apparatus according to any one of claims 7-9, further comprising: The text information recognition unit is configured to recognize text information from each of the image frames in response to the target video being extracted from a waterfall video stream; The text feature generation unit is configured to generate text features of the target video based on the text information and the characteristics of the waterfall video stream; The feature splicing unit is configured to splice the text features and the global image features to obtain the comprehensive video features of the target video.

11. The apparatus according to claim 10, wherein, The text information includes: first text information embedded in the image frame and second text information located in the video evaluation parameters above the image frame; wherein, the video evaluation parameters include: number of likes, number of favorites, number of comments, number of reposts, whether followed, and number of followers.

12. The apparatus according to claim 7, wherein, The satisfaction labels in the training samples also include the reasons for the satisfaction.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the video satisfaction evaluation method according to any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the video satisfaction evaluation method according to any one of claims 1-6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the video satisfaction evaluation method according to any one of claims 1-6.

Images