Method, device, equipment and medium for video optimization

Abstract

According to an embodiment of the present disclosure, a method, apparatus, device and medium for video optimization are provided. The method includes: dividing a target video into multiple video clips; extracting multiple material features of multiple video materials from the multiple video clips respectively; for each of the multiple video clips, based on the multiple material features of the multiple video materials, determining multiple importance levels of the multiple video materials in the video clip to the target group; and based on the importance levels determined for the multiple video clips, providing optimization information for the target video, the optimization information at least indicating optimization tips for at least one video material of at least one video clip in the multiple video clips. In this way, automatic attribution analysis of the video is achieved, and fine-grained optimization information is provided to guide the content provider to complete the video optimization in a targeted manner.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for video optimization. Background Art

[0002] Many content publishing platforms allow entities to publish videos for users to watch. In some embodiments, such videos may have advertising attributes. The entity providing the video may be paid or have other benefits based on the user's interaction with the video. Such interactions include, for example, video clicks, sharing, comments, complete playback, etc. Improvements to the video may increase the number of users who expect to interact with the video, and thus increase the benefits of the entity controlling the video. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for video optimization is provided. The method includes: dividing a target video into multiple video segments; extracting multiple material features of multiple video materials from the multiple video segments respectively; for each of the multiple video segments, determining multiple importance levels of multiple video materials in the video segment to a target group based on the multiple material features of the multiple video materials; and providing optimization information for the target video based on the importance levels determined for the multiple video segments, the optimization information at least indicating an optimization prompt for at least one video material of at least one video segment in the multiple video segments.

[0004] In a second aspect of the present disclosure, a device for video optimization is provided. The device includes: a division module configured to divide a target video into multiple video segments; a feature extraction module configured to extract multiple material features of multiple video materials from the multiple video segments respectively; an importance determination module configured to determine, for each of the multiple video segments, multiple importances of multiple video materials in the video segment to a target group based on multiple material features of the multiple video materials; and an optimization indication module configured to provide optimization information for the target video based on the importance determined for the multiple video segments, the optimization information at least indicating an optimization prompt for at least one video material of at least one video segment in the multiple video segments.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0010] Figure 2 A flowchart of a process for video optimization according to some embodiments of the present disclosure is shown;

[0011] Figure 3 An example for extracting feature attributes according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A schematic diagram showing an example architecture of a content evaluation model according to some embodiments of the present disclosure;

[0013] Figure 5 A schematic diagram showing an example architecture for model training and application according to some embodiments of the present disclosure;

[0014] Figure 6 A block diagram showing an apparatus for video optimization according to some embodiments of the present disclosure; and

[0015] Figure 7 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0018] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0019] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0020] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0023] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0024] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0025] Generally, machine learning can be roughly divided into three stages, namely the training stage, the verification stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association from input to output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the verification stage, the verification input is applied to the trained model to verify whether the model can provide the correct output, thereby determining the performance of the model. In the application stage, the model can be used to process the actual input based on the parameter values ​​obtained from the training to determine the corresponding output.

[0026] Figure 1 A schematic diagram of an example environment 100 in which an embodiment of the present disclosure can be implemented is shown. One or more content providing entities 120-1, 120-2, 130-3 (for ease of discussion, collectively or individually referred to as content providing entity 120) can distribute the content provided by them via a content distribution platform 110. In an embodiment of the present disclosure, video-type content is of primary concern. For example, videos 122-1, 122-2, ..., 122-M (for ease of discussion, collectively or individually referred to as video 122) provided by content providing entity 120 can be stored in a content library 125 and distributed by content distribution platform 110. Content providing entity 120 can be an individual entity or an organizational entity, which is not limited here.

[0027] One or more terminal devices 130-1, 130-2, 130-3, etc. (for ease of discussion, collectively or individually referred to as terminal devices 130) are associated with the content distribution platform 110, and can, for example, access various types of content provided on the content distribution platform 110 based on requests from corresponding users 132-1, 132-2, 132-3, etc. (for ease of discussion, collectively or individually referred to as users 132). The content distribution platform 110 can also provide corresponding content to the target group based on corresponding strategies, including one or more videos 122. As an example, the content distribution platform 110 can be an application, a website, a webpage, and other accessible platforms. The terminal device 130 can be installed with an application for accessing the content distribution platform 110, or the content distribution platform 110 can be accessed in an appropriate manner.

[0028] exist Figure 1 In the content distribution platform 110, the content distribution platform 110 can be any system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. The terminal device can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The server includes but is not limited to a mainframe, an edge computing node, a computing device in a cloud environment, and the like. The content providing entity 120 can be a device, or can be a user, and the user can access the content distribution platform via the corresponding device.

[0029] It should be understood that Figure 1 The components and arrangements in the illustrated environment are merely examples, and a computing system suitable for implementing the example embodiments described in this disclosure may include one or more different components, other components, and / or different arrangements.

[0030] Some videos may have advertising attributes. Such videos are sometimes referred to as video advertising creatives. The entity that provides the video (sometimes referred to as an advertiser) may be paid or have other benefits based on the user's interaction with the video. Such interactions include, for example, video clicks, sharing, comments, full playback, and other types of behaviors. Improvements to the video will increase the number of users interested in the video, and thereby increase the benefits of the entity that controls the video.

[0031] However, it may be difficult for the content provider to determine how the video should be changed to improve it. Usually, the screening of high-quality videos is carried out after the video is delivered based on the user's interaction results with the video, so as to determine whether certain videos are to be optimized. However, evaluation by delivery will incur a large cost, including system cost and time cost. In addition, the evaluation results based on delivery are greatly influenced by the delivery environment, model and user environment, and the conclusions of multiple deliveries are not necessarily consistent, so it is impossible to provide effective guidance to the video producer. On the other hand, even if the evaluation results of the delivery show that the video needs to be improved, the content provider entity cannot clearly know which materials of the video need to be improved, but can only remake it from the creative point of view and deliver it again to verify the effect. In some implementations, it is also provided to use a model to predict whether the user is interested in a specific video. However, from the prediction result, it can only be determined that the overall video needs to be optimized, but the specific optimization direction cannot be determined.

[0032] According to an embodiment of the present disclosure, an improved video optimization scheme is provided. In this scheme, a video is finely disassembled, including dividing the video into multiple segments and extracting the corresponding material features of the video material contained in each segment. Further, multiple importance levels of the video material in each video segment to the target group are determined. Based on the analysis of the importance of the fine-grained material in the segment, optimization information for the video is provided to indicate optimization tips for at least one video segment in the multiple video segments. In this way, automatic attribution analysis of the video is achieved, and fine-grained optimization information is provided to guide the content provider to complete the video optimization in a targeted manner. In this way, optimization can be performed before the video is released, and efficient, reliable and explainable video content evaluation and analysis can be achieved. This not only improves the efficiency and reliability of screening high-quality videos, but also reduces the cost of content delivery and improves content production efficiency.

[0033] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0034] Figure 2 A flow chart of a process 200 for video optimization according to some embodiments of the present disclosure is shown. The process 200 can be implemented, for example, in the content distribution platform 110 or any other suitable platform or system to implement optimization guidance for a specific video. For the convenience of discussion only, the process 200 is implemented in the content distribution platform 110 as an example for description below.

[0035] In block 210, the content distribution platform 110 divides the target video into a plurality of video segments. The target video may be, for example, a video provided by a content provider, and the content provider entity wishes to determine whether the video needs to be optimized. In order to provide fine-grained optimization suggestions, the overall target video is first divided into a plurality of video segments.

[0036] The division of video segments can be performed according to any appropriate standard. In some embodiments, the target video can be split according to the granularity of the shot. The shot information of the target video can be obtained, and the shot information can indicate the switching time point between two consecutive shots in the target video. The shot information can be obtained by automatically analyzing the target video or from an existing data source. Based on the shot information, the target video can be divided into multiple shots. Each video segment to be analyzed can correspond to one or more shots in the target video. In some embodiments, in addition to dividing by shot, the video segments can also be divided according to a predetermined time interval so that the length of each video segment is substantially the same. In some embodiments, in order to avoid the expansion of feature data extracted from the video segments later, an upper limit on the number of video segments divided for each video can be set.

[0037] At block 220 , the content distribution platform 110 determines a plurality of material features of a plurality of video materials from the plurality of video clips, respectively.

[0038] The video material here refers to the unit attribute that can be adjusted in video creation. In some embodiments, the video material may relate to items, characters, text, audio, etc. that appear in the video. For each type, one or more aspects of the material can be concerned. For example, the video material related to the product may include the name of the product, the category of the product, the appearance of the product (e.g., color, size, etc.), the position of the product in the picture, etc. The video material related to the character may include the appearance of the character, the position of the character in the picture, etc. The video material related to the text may include the text information corresponding to the subtitles or dubbing in the picture, the appearance of the text (e.g., font, size, etc.), the position of the text in the picture, etc. The video material related to the audio may include the type of audio (e.g., songs, light music, etc.), the volume of the audio, etc. In addition to the video material mentioned here, other video materials available in the video may also be considered additionally or alternatively.

[0039] For each video material, the features or attributes displayed in different video clips may be different, or may not appear in every video clip. In the embodiment of the present disclosure, the feature analysis of the material is performed on each video clip with the video clip as the granularity.

[0040] Figure 3 An example of extracting characteristic attributes according to some embodiments of the present disclosure is shown. Figure 3 As shown, it is assumed that a target video is divided into N video segments, including video segments 310-1, 310-2, 310-3, ..., 310-N (for the convenience of discussion, sometimes collectively or individually referred to as video segments 310). Figure 3As shown, for each video clip 310, features of text material, object material 1, object material 2, person material 1, person material 2, audio material, etc. can be extracted.

[0041] For each video clip, the material features of each video material in the video clip can be represented by a structured representation. For example, for video clip 310-1, feature ID 100 can be used to represent text clip 1 of the video clip, feature ID 1001 can be used to represent the object features corresponding to object material 1, feature ID 1002 can be used to represent the object features corresponding to object material 2, feature ID 1003 can be used to represent the character features corresponding to character material 1, and so on. For other video clips, the structured representation of the corresponding features can also be constructed in a similar manner.

[0042] For each video clip, the type of video material to be considered may be fixed, but the features of the corresponding video material may not be extracted from different videos. For example, for object materials 1 and 2, the corresponding features may not be extracted from video clips 310-1 and 310-N, and for person materials 1 and 2, the corresponding features may not be extracted from video clips 310-1 and 310-N. In this case, in the structured representation corresponding to the video clip, the features that cannot be extracted can be filled with 0 values, null values, or any other appropriate values.

[0043] In some embodiments, one or more video features may be extracted from the entire target video to facilitate subsequent analysis. The extracted video features may be, for example, the title features of the entire target video, the screen appearance features (e.g., whether the overall screen color is bright or dim), the industry features involved in the video (e.g., involving daily life, office environment, specific industrial industry or other), etc. For example, feature ID000 may be used to represent the screen appearance features of the target video, feature ID0001 may be used to represent the industry features of the target video, and so on.

[0044] In some embodiments, the material features or the video material of the target video can be represented as a multi-dimensional feature vector or feature embedding to facilitate subsequent processing. For different similar video materials, corresponding features can be extracted in different ways.

[0045] In some embodiments, for text-like features, optical character recognition (OCR) can be used to identify subtitles in the video clip, and speech recognition (ASR) can be used to identify speech in the video clip, and so on. Of course, if the text corresponding to the target video is directly available, there is no need to perform an automatic process. After obtaining the text segment corresponding to each video segment, the text segment can be represented in the form of a text feature vector by establishing a vocabulary. For example, the vector representation of each word in the text segment can be determined by establishing a vocabulary. For another example, the text features corresponding to the text segment can be determined by a pre-trained language model, such as a word2vec model.

[0046] In some embodiments, for image features, such as the name of a product, the category of a product, the appearance of a product, the appearance of a person, the appearance of text, etc., the corresponding features can be extracted through an image processing model. For example, an image classification model can be used to classify objects in a picture as products, people, or texts, and the name of the product, the category of the product, etc. can be specifically identified. In addition, the appearance of the product, the appearance of the person, the appearance of the text, etc. can be extracted for the picture area belonging to the product, person, or text.

[0047] In some embodiments, for location features, such as the location of titles, products, characters, text, etc., the intersection over union (IoU) can be calculated between the bounding box position and the predefined anchor box position, and the anchor box matching the maximum IoU is determined as the location feature. Each location feature can represent the coordinates of the corresponding object in the two-dimensional space of the video screen.

[0048] In some embodiments, for color-related features, such as the overall color of a screen or the color of a specific object (such as text, merchandise, etc.), a predetermined color space (such as an RGB space) may be divided into multiple intervals (such as 10x10x10 intervals), each of which corresponds to a feature value. In this way, the color falling within the corresponding interval may be represented by the corresponding feature value.

[0049] Some types of feature extraction methods are given above. However, it can be understood that the embodiments of the present disclosure do not limit the specific means of extracting features.

[0050] In block 230 , the content distribution platform 110 determines, for each of the plurality of video clips, a plurality of importance levels of the plurality of video materials in the video clip to the target group based on a plurality of material features of the plurality of video materials.

[0051] The target group can be the user group that the target video expects to be put, and can relate to one or more users.In certain embodiments, the importance of each video material in each video clip can be analyzed about the interest level of the target video by the target group.The importance of each video material to the target group can indicate that this video material can influence the degree that the target group is interested in the target video.That is to say, if the importance of a certain video material in a certain video clip is high, it means that this video material can make the target group interested in the target video to a large extent.

[0052] By performing importance analysis based on the video material in the video clip as the granularity, fine-grained optimization suggestions can be better provided. In block 240, the content distribution platform 110 provides optimization information for the target video based on the importance determined for the multiple video clips, the optimization information at least indicating optimization tips for at least one video material of at least one video clip in the multiple video clips.

[0053] The optimization hint is a finer-grained distinction of the target video, which can achieve optimization guidance for specific materials in a specific segment of the video. In addition, based on the optimization hint, feature materials in a specific segment that are more important to the target group can also be screened out. The optimization hint can be transmitted to the content provider, or presented to be available to the content provider.

[0054] In some embodiments, the importance of each video material in each video clip to the target group can be determined simultaneously in the process of determining the interest level of the target group to the target video. The determined importance level is used to finally determine the interest level of the target group to the target video as an intermediate result. In addition, these importance levels are also provided for optimization information.

[0055] In some embodiments, the determination of importance can be implemented based on a machine learning model. This will be discussed in more detail below.

[0056] Specifically, the attention module included in the trained content evaluation model can be used to determine the importance of multiple video materials in the video clip to the target group. The input of the content evaluation model at least includes multiple material features of multiple video materials determined for multiple video clips and group features determined for the target group. The content evaluation model is configured to determine the interest level of the target group in the target video, that is, the model output is the interest level.

[0057] Figure 4A schematic diagram of an example architecture of a content assessment model 400 according to some embodiments of the present disclosure is shown. As shown, the input of the content assessment model 400 includes group features 412 determined by the target group. The group features 412 can be various types of features suitable for characterizing the target group. The input of the content assessment model 400 also includes material features 410-1, 410-2, ... 410-XXX (collectively or individually referred to as material features) of each video material of each video clip of the target video. In some embodiments, additionally, the input of the content assessment model 400 can also include the overall features of the target video, such as the video title, screen appearance features, industry features, etc. mentioned above. The features input to the content assessment model 400 can be in vector form.

[0058] The content evaluation model 400 includes a feature extraction module 422 for processing group features 412, and also includes a feature extraction part 420, which is configured to process relevant features of the target video, including material features 410. The extracted features are output to the attention module 430 respectively. The attention module 430 is configured to determine the importance of each video material of each video clip to the target group based on the attention mechanism. The attention module 430 can determine the importance of the video material to the target group based on the group features for the target group (features extracted by the feature extraction part 422) and the material features of each video material of each video clip. In this way, for material features 410-1, 410-2, ... 410-XXX, importances from 1 to XXX can be determined. In some embodiments, in addition to the video material, the attention module 430 can also determine the importance of other features (features at the overall video level) inputted and related to the target video.

[0059] The importance determined by the attention module 430 is output for analyzing the optimization information for the target video. In addition, the importance determined by the attention module 430 is also provided to the weighting module 432, which weights the determined importance to the features respectively extracted by the feature extraction module 420, and provides the weighted results to the subsequent feature extraction part 440 to perform further feature processing. In addition, the features extracted by the feature extraction part 422 with respect to the target group are also provided to the subsequent feature extraction part 442 for further processing. The feature extraction results of the feature extraction parts 440 and 442 can be provided to the summing module 450 to provide the summing results to the output layer 460. The output layer 460 determines the model output, that is, the degree of interest of the target group in the target video.

[0060] Figure 4Only a schematic architecture of the content evaluation model is given. The feature extraction part in the content evaluation model 400 may include various learning network structures, such as a fully connected (FC) layer, a factorization machine (FM), a cascade layer, etc. The embodiments of the present disclosure do not limit the composition of each feature extraction part and the connection relationship between them. The content evaluation model 400 may also have Figure 4 Different architectures are acceptable as long as the attention module can determine the importance of the video material in each video clip to the target group.

[0061] From the above discussion, we can see that the content evaluation model can not only evaluate the interest of the target group in the target video, but also the importance of the intermediate output can be attributed to such results, and it is a fine-grained attribution analysis. Based on the results of the attribution analysis, specific optimization suggestions for the target video can be provided.

[0062] In some embodiments, in order to reduce the error as much as possible, the importance can be determined by the mean of multiple results. For example, the target group may include multiple users. For each user, the multiple importances of multiple video materials in each video clip to the user can be determined. When applying the content evaluation model, the group feature 412 input to the model may include the relevant features of a single user. For each video material in the multiple video materials, the importance of the video material to the target group is determined by aggregating the multiple importances of the video material determined for multiple users. For example, if the importance of each video material of each video clip is determined for 200 users, the average value of the 200 importances can be calculated as the importance of the video material.

[0063] In some embodiments, when providing optimization information, it is expected to be able to analyze which materials need to be adjusted and which materials are more important from the importance of each video material in each video clip, so as to guide the content providing entity to further optimize.

[0064] Specifically, for a given video segment of multiple video segments, the segment importance of the given video segment to the target group can be determined based on the multiple importances of the multiple video materials determined for the given video segment. In this way, an importance measurement index of the video segment granularity can be obtained. If the segment importance of the video segment is lower than the segment threshold, optimization information can be provided to at least indicate an optimization prompt for the given video segment. The optimization prompt is an optimization prompt at the segment granularity, which can prompt the content provider to conduct a comprehensive analysis of the video materials of the segment.

[0065] In certain embodiments, multiple importances of multiple video materials of a given video clip can also be compared with multiple material thresholds for each of the multiple video materials. For example, for different types of video materials, the same or different material thresholds can be set. If the importance of at least one video material in the multiple video materials is lower than the material threshold for at least one video material, it can be determined that at least one video material in the given video clip is to be optimized. Accordingly, optimization information can be provided to at least indicate optimization prompts for at least one video material of the given video clip. In this way, optimization prompts for material granularity in the video clip can be implemented, so that the content providing entity can complete the optimization more specifically. For example, if the importance of a text segment in a certain video segment is found to be lower through the analysis of the importance, the content providing entity can be prompted to optimize the text segment in the video segment.

[0066] In some embodiments, if the segment importance of a given video segment is lower than the segment threshold, multiple importances of multiple video materials of the given video segment may also be presented, for example, to a content provider entity. In this way, the content provider entity may autonomously analyze which video material in the video segment has a lower score and should be improved.

[0067] In some embodiments, it is desirable to determine the target video with a low estimated interest level of the target group for optimization. Therefore, when it is determined that the interest level of the target group in the target video is lower than the interest threshold, optimization information can be provided to guide the optimization of the target video.

[0068] In the case of considering multiple users, the interest level of multiple users in the target group in the target video can also be determined. By aggregating the interest levels of multiple users, the interest level of the target group in the target video can be determined for threshold comparison. This can avoid statistical errors.

[0069] In some embodiments of the above importance determination, a content evaluation model is applied, which can be obtained through training. To facilitate understanding of the life cycle management of machine learning models, Figure 5 A schematic diagram of an example architecture for model training and application according to some embodiments of the present disclosure is shown. Figure 5 In the architecture 500, it is desirable to train and use such a content evaluation model for evaluating quality metrics of input recommended content items.

[0070] like Figure 5 As shown, the architecture 500 includes a model training system 510 and a model application system 520. Figure 5In an example embodiment, the model training system 510 is configured to train the content assessment model 400 using training data. The training data may include a sample video 520 and a corresponding sample population 524. For the sample video 521, it can be divided into multiple video clips 522 based on the embodiment discussed above about the target video. Subsequently, multiple material features of multiple video materials of concern to the content assessment model 400 can be extracted for each video clip. In addition, depending on the requirements of the input of the content assessment model 400, other features of other target videos can also be used. For the sample population 524, the corresponding population features can also be extracted as part of the model input.

[0071] The training data also includes group feedback information 526, which is used to indicate the degree of interest of the sample group 524 (or each user therein) in the sample video 521. For example, the sample video 521 can be a video that is presented to the sample group for a period of time. The degree of interest of the sample group or user in the video can be measured from one or more appropriate aspects. For example, if a user clicks on a video, shares a video, likes a video, or watches any of the entire video, it can be considered that the user is interested in the video. The group feedback information 526 can use 1 to indicate that the sample group 524 is interested in the sample video 521 (for example, because the sample video 521 is clicked or shared by most users, etc.), and use 0 to indicate that the sample group 524 is not interested in the sample video 521 (for example, because the sample video 521 is rarely clicked or shared by users, etc.).

[0072] The group feedback information 526 can be used as annotation information of the current sample video 521 and the sample group 524 to guide the training of the content evaluation model 400 .

[0073] Before training, the parameter values ​​of the content evaluation model 400 can be initialized, or the pre-trained parameter values ​​can be obtained through a pre-training process. After the training process, the parameter values ​​of the content evaluation model 400 are updated and adjusted. After the training is completed, the content evaluation model 400 has the trained parameter values. Based on such parameter values, the content evaluation model 400 can correctly or substantially correctly predict the interest level of the sample group 524 in the sample video 521.

[0074] In the model application stage, the model application system 520 can be configured to process the target video 520 and the target group 534 using the trained content assessment model 400 to determine the optimization information for the target video. The model application system 520 can determine the optimization information based on the importance output by the attention module 430 included in the content assessment model 400, and can also be combined with the interest level output by the content assessment model 400.

[0075] exist Figure 5In the embodiment, the model training system 510 and the model application system 520 can be any system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. The terminal device can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Servers include but are not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0076] It should be understood that Figure 5 The components and arrangements in the illustrated environment are merely examples, and a computing system suitable for implementing the example embodiments described in the present disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as being separate, the model training system 510 and the model application system 520 may be integrated into the same system or device. The embodiments of the present disclosure are not limited in this respect. In some embodiments, the content delivery platform 110 may include the model application system 520.

[0077] Figure 6 A schematic structural block diagram of an apparatus 600 for video optimization according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in the model application system 120. Each module / component in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0078] As shown in the figure, the device 600 includes a division module 610, which is configured to divide the target video into multiple video segments. The device 600 includes a feature extraction module 620, which is configured to extract multiple material features of multiple video materials from the multiple video segments respectively. The device 600 includes an importance determination module 630, which is configured to determine, for each of the multiple video segments, multiple importances of the multiple video materials in the video segment to the target group based on the multiple material features of the multiple video materials. The device 600 includes an optimization indication module 640, which is configured to provide optimization information for the target video based on the importance determined for the multiple video segments, and the optimization information at least indicates an optimization prompt for at least one video material of at least one video segment in the multiple video segments.

[0079] In some embodiments, the optimization indication module 640 includes: a segment importance determination module, configured to determine, for a given video segment of multiple video segments, the segment importance of a given video segment to a target group based on multiple importances of multiple video materials determined for the given video segment; and a segment-based optimization prompt module, configured to provide optimization information to at least indicate an optimization prompt for the given video segment if the segment importance of the given video segment is lower than a segment threshold.

[0080] In some embodiments, the optimization indication module 640 includes: a threshold comparison module, configured to compare multiple importance levels of multiple video materials of a given video clip with multiple material thresholds for each of the multiple video materials; a material optimization determination module, configured to determine that at least one video material in a given video clip is to be optimized if the importance level of at least one video material among the multiple video materials is lower than the material threshold for at least one video material; and a material optimization prompt module, configured to provide optimization information to at least indicate an optimization prompt for at least one video material of a given video clip.

[0081] In some embodiments, the apparatus 600 further includes: a presentation module configured to present multiple importance levels of multiple video materials of the given video segment if the segment importance level of the given video segment is lower than the segment threshold.

[0082] In some embodiments, the optimization indication module 640 includes: an interest determination module configured to determine the interest level of the target group in the target video; and an interest-based optimization prompt module configured to provide optimization information if the interest level is below an interest threshold.

[0083] In some embodiments, the target group includes multiple users. In some embodiments, the segment importance determination module includes: a user granularity importance determination module configured to determine, for each of the multiple users, multiple importances of multiple video materials in the video segment to the user; and an aggregation module configured to determine, for each of the multiple video materials, the importance of the video material to the target group by aggregating the multiple importances of the video material determined for the multiple users.

[0084] In some embodiments, the plurality of video materials include at least one of the following items appearing in the video clip: at least one video material related to a person, at least one video material related to an object, at least one video material related to text, or at least one video material related to audio.

[0085] In some embodiments, the importance determination module 630 includes: a model-based determination module configured to use an attention module included in a trained content evaluation model to determine multiple importances of multiple video materials in the video clip to the target group. In some embodiments, the input of the content evaluation model includes at least multiple material features of multiple video materials determined for multiple video clips and group features determined for the target group, and the content evaluation model is configured to determine the interest of the target group in the target video.

[0086] In some embodiments, the division module 610 is configured to: divide the target video into a plurality of video segments based on the shot information of the target video, each video segment corresponding to at least one shot in the target video.

[0087] Figure 7 1 shows a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 7 The electronic device 700 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown may be used to implement the content distribution platform 110, the model training system 510 and / or the model application system 520. The electronic device 700 may include or be implemented as Figure 6 Device 600.

[0088] like Figure 7 As shown, the electronic device 700 is in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of the electronic device 700.

[0089] The electronic device 700 typically includes a plurality of computer storage media. Such media may be any available media accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the electronic device 700.

[0090] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0091] The communication unit 740 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0092] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 700, or communicate with any device that allows the electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0093] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0094] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0095] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0096] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0097] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0098] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for video optimization, comprising: Divide the target video into multiple video segments; Extracting a plurality of material features of a plurality of video materials from the plurality of video clips respectively, the plurality of video materials appearing in the plurality of video clips and having a plurality of different information types, the material features of the video materials of different information types corresponding to different extraction methods, and each video clip including at least part of the plurality of video materials; For each of the plurality of video clips, determining, based on the plurality of material features of the plurality of video materials, a plurality of importance levels of the plurality of video materials in the video clip to a target group; as well as Based on the importance of the multiple video materials determined for the multiple video clips, optimization information for the target video is provided, wherein the optimization information at least indicates an optimization hint for at least one video material of at least one video clip among the multiple video clips.

2. The method of claim 1, wherein providing the optimization information comprises: For a given video segment of the plurality of video segments, Determining the importance of the given video segment to the target group based on the plurality of importances of the plurality of video materials determined for the given video segment; as well as If the segment importance of the given video segment is lower than a segment threshold, the optimization information is provided to at least indicate an optimization hint for the given video segment.

3. The method according to claim 2, wherein providing the optimization information further comprises: comparing a plurality of importance levels of the plurality of video materials of the given video clip with a plurality of material thresholds for each of the plurality of video materials; If the importance of at least one video material among the plurality of video materials is lower than a material threshold for the at least one video material, determining that the at least one video material in the given video segment is to be optimized; as well as The optimization information is provided to at least indicate optimization hints for the at least one video material of the given video segment.

4. The method according to claim 3, further comprising: If the segment importance of the given video segment is lower than the segment threshold, multiple importance levels of the multiple video materials of the given video segment are presented.

5. The method of claim 1, wherein providing the optimization information comprises: Determining the interest level of the target group in the target video; as well as If the interest level is below an interest threshold, providing the optimization information.

6. The method according to claim 1, wherein the target group includes a plurality of users, and determining a plurality of importance levels of the plurality of video materials in the video clip to the target group comprises: For each of the multiple users, determining multiple importance levels of the multiple video materials in the video clip to the user; as well as For each of the multiple video materials, the importance of the video material to the target group is determined by aggregating multiple importance levels of the video material determined for the multiple users.

7. The method according to claim 1, wherein the plurality of video materials comprises at least one of the following items appearing in the video clip: At least one video clip related to the character, At least one video clip related to the item, At least one video clip related to the text, or At least one video material associated with the audio.

8. The method according to claim 1, wherein determining the importance of the plurality of video materials in the video clip to the target group comprises: Determine multiple importance levels of the multiple video materials in the video clip to the target group using the attention module included in the trained content assessment model, and The input of the content evaluation model includes at least a plurality of material features of the plurality of video materials respectively determined for the plurality of video clips and a group feature determined for the target group, and the content evaluation model is configured to determine the interest level of the target group in the target video.

9. The method according to claim 1, wherein dividing the target video into the plurality of video segments comprises: Based on the shot information of the target video, the target video is divided into the multiple video segments, each video segment corresponds to at least one shot in the target video.

10. A device for video optimization, comprising: A division module, configured to divide the target video into a plurality of video segments; a feature extraction module configured to extract a plurality of material features of a plurality of video materials from the plurality of video clips, respectively, the plurality of video materials appearing in the plurality of video clips and having a plurality of different information types, the material features of the video materials of different information types corresponding to different extraction methods, and each video clip including at least part of the plurality of video materials; an importance determination module configured to determine, for each of the plurality of video clips, a plurality of importances of the plurality of video materials in the video clip to a target group based on the plurality of material features of the plurality of video materials; as well as The optimization indication module is configured to provide optimization information for the target video based on the importance of the multiple video materials determined for the multiple video clips, and the optimization information at least indicates optimization tips for at least one video material of at least one video clip among the multiple video clips.

11. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit.

12. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 9 when executed by a processor.

Images