Video processing method, device, computer equipment and storage medium

By dividing the video into video clips and units, and calculating the core degree using feature and position information, the problem of inaccurate extraction of key information in the video is solved, and higher accuracy and completeness are achieved.

CN113569093BActive Publication Date: 2024-12-06TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110062919.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-18
Publication Date
2024-12-06
Estimated Expiration
2041-01-18

AI Technical Summary

Technical Problem

The prior art is difficult to accurately extract core content from videos, resulting in inaccurate acquisition of key information in videos.

Method used

By dividing the video into multiple video clips and video units, the core content degree is calculated using video features and position information, and then the core video is determined.

Benefits of technology

Improve the accuracy of extracting core parts from the video, ensuring the completeness and accuracy of key information in the video.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113569093B_ABST
    Figure CN113569093B_ABST
Patent Text Reader

Abstract

The present application relates to a video processing method, device, computer equipment and storage medium. The method relates to machine learning technology, and the video processing method comprises: obtaining a video to be processed, the video to be processed is used to describe a target object; after dividing the video to be processed into a plurality of video segments, obtaining the position information of each video segment in the video to be processed, each video segment is used to describe different contents of the target object; dividing each video segment into a plurality of video units; based on the video features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located, obtaining the content coreness of each video unit; according to the content coreness, determining the core video describing the target object from the video to be processed. The use of this method can improve the accuracy of extracting the core part from the video.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of computer technology and Internet technology, video has become one of the important channels for people to obtain information. The forms of video have also become more diverse, such as live video, e-commerce video, self-media video, etc. Various video forms promote each other and continue to develop.

[0003] In some scenarios, users need to obtain short video content from a video, such as the essence of the video or the key video content in the video. These short videos can allow users to quickly understand the general content of the video. However, currently, some video content is generally selected roughly from the original video manually, and the video content selected in this way is often not accurate enough. Summary of the invention

[0004] Based on this, it is necessary to provide a video processing method, apparatus, computer equipment and storage medium that can improve the accuracy of extracting core content from a video in response to the above technical problems.

[0005] A video processing method, the method comprising:

[0006] Obtain a video to be processed, where the video to be processed is used to describe a target object;

[0007] After dividing the video to be processed into multiple video segments, obtaining the position information of each video segment in the video to be processed, each video segment is used to describe different contents of the target object;

[0008] Divide each video clip into a plurality of video units;

[0009] Based on the video features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located, the content coreness of each video unit is obtained;

[0010] According to the core degree of the content, a core video describing the target object is determined from the video to be processed.

[0011] In one embodiment, the video to be processed is divided into a plurality of video segments, including:

[0012] Obtaining feature vectors of each video frame in the video to be processed;

[0013] Clustering is performed on each video frame according to the feature vector to obtain multiple clusters;

[0014] According to the video frames classified into the same cluster, a plurality of video clips are obtained.

[0015] In one embodiment, each video frame is clustered according to the feature vector to obtain multiple clusters, including:

[0016] Selecting a target number of video frames from each video frame of the video to be processed as clustering centers;

[0017] According to the similarity between the feature vector of each video frame and the feature vector of the cluster center, each video frame is clustered to obtain the target number of clusters.

[0018] In one embodiment, the method further comprises:

[0019] After performing clustering processing according to the target number and obtaining video clips, traverse each video frame of the video to be processed;

[0020] Determine the intra-cluster similarity between the traversed current video frame and other video frames in the same video segment;

[0021] Determine out-of-cluster similarity between the traversed current video frame and other video clips;

[0022] According to the difference between the similarity within the cluster and the similarity outside the cluster, the clustering coefficient corresponding to the current video frame is determined;

[0023] According to the clustering coefficient corresponding to each video frame in the to-be-processed video obtained after the traversal is completed, the clustering score corresponding to the clustering processing according to the target number is determined;

[0024] According to the cluster clusters corresponding to the target number when the clustering score is the highest, multiple video clips corresponding to the video to be processed are determined.

[0025] In one embodiment, each video segment is divided into a plurality of video units, including:

[0026] Obtaining feature vectors of each video frame in the video clip;

[0027] Clustering each video frame in the video clip according to the feature vector to obtain multiple clusters;

[0028] According to the video frames classified into the same cluster, a plurality of video units are obtained.

[0029] In one embodiment, clustering is performed on each video frame in the video clip according to the feature vector to obtain multiple clusters, including:

[0030] Selecting a target number of video frames from each video frame of the video clip as clustering centers;

[0031] According to the similarity between the feature vector of each video frame and the feature vector of the cluster center, each video frame in the video clip is clustered to obtain the target number of clusters.

[0032] In one embodiment, the method further comprises:

[0033] After performing clustering processing according to the target number and obtaining video units, traversing each video frame of the video clip;

[0034] Determine the intra-cluster similarity between the traversed current video frame and other video frames in the same video unit;

[0035] Determine out-of-cluster similarity between the traversed current video frame and other video units;

[0036] According to the difference between the similarity within the cluster and the similarity outside the cluster, the clustering coefficient corresponding to the current video frame is determined;

[0037] According to the clustering coefficient corresponding to each video frame in the video clip obtained after the traversal is completed, the clustering score corresponding to the clustering processing according to the target number is determined;

[0038] According to the cluster clusters corresponding to the target number when the clustering score is the highest, a plurality of video units corresponding to the video clip are determined.

[0039] In one embodiment, the method further comprises:

[0040] Input the video to be processed into the trained core video generation model;

[0041] Through the feature vector generation network in the core video generation model, feature vectors are extracted from each video frame in the processed video to obtain the feature vectors corresponding to each video frame.

[0042] In one embodiment, obtaining the position information of each video segment in the video to be processed includes:

[0043] Obtain the sequence of each video frame in the video clip in the video to be processed;

[0044] The position information of the video segment where the video frame is located in the video to be processed is determined according to the sequence.

[0045] In one embodiment, based on the video features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located, the content coreness of each video unit is obtained, including:

[0046] Determine the image core score of each video unit according to the image features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located;

[0047] Determine the audio core score of each video unit according to the audio features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located;

[0048] The image core score and audio core score of each video unit are integrated to obtain the content core degree of each video unit.

[0049] In one embodiment, determining the image core score of each video unit according to the image features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located includes:

[0050] Obtaining a feature vector corresponding to the position information of each video unit;

[0051] The image features and feature vectors corresponding to each video unit are input into the image evaluation network in the trained core video generation model in turn to obtain the image core score of each video unit.

[0052] In one embodiment, determining the audio core score of each video unit according to the audio features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located includes:

[0053] Obtaining a feature vector corresponding to the position information of each video unit;

[0054] The audio features and feature vectors corresponding to each video unit are input into the audio evaluation network in the trained core video generation model in turn to obtain the audio core score of each video unit.

[0055] In one embodiment, the image core score and the audio core score of each video unit are integrated to obtain the content core degree of each video unit, including:

[0056] Obtain the weights corresponding to the image core score and the audio core score respectively;

[0057] The image core score and the audio core score of each video unit are weighted and summed according to the weight to obtain the content core degree of each video unit.

[0058] In one embodiment, determining a core video describing a target object from the video to be processed according to the coreness of the content includes:

[0059] Determine the core degree distribution curve corresponding to the video to be processed according to each video unit and the corresponding content core degree;

[0060] According to the core degree distribution curve, video units with content core degrees higher than a threshold are selected from the divided video units;

[0061] A core video describing the target object is generated based on the filtered video units.

[0062] In one embodiment, the method further comprises:

[0063] After dividing each video segment into a plurality of video units, obtaining position information of each video unit in the video to be processed;

[0064] Based on the video features corresponding to each video unit and the position information of each video unit in the video to be processed, the content coreness of each video unit is obtained.

[0065] In one embodiment, obtaining a video to be processed includes:

[0066] Get the pending video describing the product;

[0067] According to the core content, determine the core video describing the target object from the video to be processed, including:

[0068] Determine the video unit that describes the key information of the product based on the core content;

[0069] A core video describing key information of the product is generated according to the determined video units.

[0070] In one embodiment, the core video is obtained by a core video generation model, and the training steps of the core video generation model include:

[0071] Obtain a sample video and an annotated core video corresponding to the sample video;

[0072] Input the sample video into the core video generation model;

[0073] After dividing the sample video into multiple video segments through the core video generation model, obtaining the position information of each video segment in the sample video, dividing each video segment into multiple video units, and obtaining the content coreness of each video unit based on the video features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located. According to the content coreness, determine the predicted core video describing the target object from the sample video;

[0074] Construct a loss function based on the labeled core video and the predicted core video;

[0075] After updating the model parameters of the core video generation model according to the loss function, return to the step of obtaining the sample video and the labeled core video corresponding to the sample video to continue training until the training stop condition is met, thereby obtaining the trained core video generation model.

[0076] A video processing device, comprising:

[0077] An acquisition module is used to acquire a video to be processed, where the video to be processed is used to describe a target object;

[0078] A division module is used to divide the video to be processed into multiple video segments, and obtain the position information of each video segment in the video to be processed, each video segment is used to describe different contents of the target object;

[0079] The division module is further used to divide each video segment into a plurality of video units;

[0080] The acquisition module is further used to obtain the content coreness of each video unit based on the video features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located;

[0081] The determination module is used to determine the core video describing the target object from the video to be processed according to the core degree of the content.

[0082] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0083] Obtain a video to be processed, where the video to be processed is used to describe a target object;

[0084] After dividing the video to be processed into multiple video segments, obtaining the position information of each video segment in the video to be processed, each video segment is used to describe different contents of the target object;

[0085] Divide each video clip into a plurality of video units;

[0086] Based on the video features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located, the content coreness of each video unit is obtained;

[0087] According to the core degree of the content, a core video describing the target object is determined from the video to be processed.

[0088] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0089] Obtain a video to be processed, where the video to be processed is used to describe a target object;

[0090] After dividing the video to be processed into multiple video segments, obtaining the position information of each video segment in the video to be processed, each video segment is used to describe different contents of the target object;

[0091] Divide each video clip into a plurality of video units;

[0092] Based on the video features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located, the content coreness of each video unit is obtained;

[0093] According to the core degree of the content, a core video describing the target object is determined from the video to be processed.

[0094] A computer program includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps of the above-mentioned video processing method.

[0095] The above-mentioned video processing method, device, computer equipment and storage medium, when extracting the core part from the video, use the characteristic that the relative position of the core part in the video has a common point. First, the video is divided into multiple video clips, so that each video clip describes different contents of the target object. Compared with the method of equally dividing the video to be processed according to the specified duration in the traditional technology, the integrity of the contents describing the target object in the video can be guaranteed; then, the position information of each video clip in the video to be processed is obtained. The position information obtained according to this division method can reflect the possibility that the video clip contains the core part; then, each video clip is divided into multiple video units respectively, and the content core degree of each video unit is obtained based on the video features corresponding to each video unit and the position information corresponding to the video clip where each video unit is located. The content core degree obtained in combination with the position information can more accurately reflect the possibility that the video unit is the core part. Therefore, determining the core video from the video to be processed according to the content core degree can improve the accuracy of extracting the core part from the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 A diagram showing an application environment of a video processing method in an embodiment;

[0097] Figure 2 is a schematic flow chart of a video processing method in one embodiment;

[0098] Figure 3 The following is a schematic diagram of an interface of a video processing page in one embodiment;

[0099] Figure 4 is a flowchart of a video processing method in one embodiment;

[0100] Figure 5 A schematic diagram of extracting a core part from an e-commerce live video in one embodiment;

[0101] Figure 6 A schematic diagram of a detailed process of dividing a video to be processed into video segments in one embodiment;

[0102] Figure 7A schematic diagram of a detailed process of dividing video clips into video units in one embodiment;

[0103] Figure 8 A schematic diagram of a detailed process of obtaining the content coreness of a video unit in one embodiment;

[0104] FIG9( a ) is a schematic diagram of the structure of a core video generation model in one embodiment;

[0105] FIG9( b ) is a schematic diagram of a partial structure of a core video generation model in one embodiment;

[0106] Fig.10 A flowchart of determining a core degree distribution curve in one embodiment;

[0107] Fig.11 is a flow chart of a video processing method in another embodiment;

[0108] Fig.12 is a structural block diagram of a video processing device in one embodiment;

[0109] Fig.13 is an internal structure diagram of a computer device in one embodiment;

[0110] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0111] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0112] The video processing method provided in this application involves machine learning (ML) technology. Machine learning is a multi-disciplinary cross-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0113] The video processing method provided in this application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The terminal 102 obtains the video to be processed that describes the target object, and sends the video to be processed to the server 104. After the server 104 divides the video to be processed into multiple video clips, it obtains the position information of each video clip in the video to be processed, and each video clip is used to describe different contents of the target object; the server 104 divides each video clip into multiple video units; the server 104 obtains the content core degree of each video unit based on the video features corresponding to each video unit and the position information corresponding to the video clip where each video unit is located; the server 104 determines the core video describing the target object from the video to be processed according to the content core degree. The server 104 can also return the core video to the terminal 102, and the terminal 102 displays the core video.

[0114] The terminal 102 may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0115] The video processing method provided in the embodiment of the present application may be executed by the video processing device provided in the embodiment of the present application, or by a computer device integrated with the video processing device, wherein the video processing device may be implemented in hardware or software. The computer device may be Figure 1 The terminal 102 or server 104 shown in .

[0116] The video processing method provided in the embodiment of the present application can be used to extract highlight moments from various videos. The highlight moment of a video refers to the core part of the video, that is, the part that best reflects the key content, highlights and purpose of the video production. For example, the duration of a video is 60 seconds, of which the 20th to 30th seconds and the 35th to 45th seconds are the core parts of the video. Then, through the embodiment of the present application, the 20th to 30th seconds and the 35th to 45th seconds can be automatically extracted from the video. Among them, various types of videos include but are not limited to TV video ads, online video ads, e-commerce videos, video blogs, self-media videos, and the like.

[0117] Take e-commerce live streaming as an example. It is used to recommend products to live viewers to arouse their desire to buy, thus achieving the purpose of product promotion. The highlight of e-commerce live streaming is the part that introduces the key information of the product. The key information of the product is the information that describes the characteristics of the product. For example, for a sweeping robot, its key information is the function of the sweeping robot, how to use it, the sweeping effect, how to buy it, etc.

[0118] The video processing method provided in the embodiments of the present application can also be applied to advertising production scenarios, short video production scenarios, video editing scenarios, and the like.

[0119] For example, in an advertising production scenario, the total length of a video is 60 seconds, but the advertising space is only 30 seconds. Through the method provided in the embodiment of the present application, the core part of 30 seconds, that is, the part that best reflects the key information of the product, can be extracted from the 60-second video and presented to the public as advertising content. The advertising space can be a TV advertising space or an online advertising space.

[0120] For another example, in the short video production scenario, referring to the above-mentioned e-commerce live broadcast example, it can be understood that in order to liven up the live broadcast atmosphere, the anchor often chats and interacts with the live broadcast viewers, and the content involved is mostly unrelated to the product. After the live broadcast ends, the method provided in the embodiment of the present application can be used to extract the core part from the recorded live broadcast as short video content and put it on the short video platform to achieve the purpose of promoting the product.

[0121] For another example, in a video editing scenario, a video creator recorded a beauty video. In order to increase fan stickiness, the video creator interspersed the beauty tutorial with sharing of daily life, making the total length of the video too long. Through the method provided in the embodiment of the present application, the beauty tutorial part can be extracted from the recorded video, reducing the total length of the video while retaining the core part of the video.

[0122] In one embodiment, Figure 2 As shown, a video processing method is provided. This embodiment mainly applies the method to a computer device (the above Figure 1 The terminal 102 or server 104 in the example is used as an example, and the steps include:

[0123] Step S202, obtaining a video to be processed, where the video to be processed is used to describe the target object.

[0124] The video to be processed is a video whose highlight moments are to be extracted through the embodiments of the present application. The videos to be processed include but are not limited to TV video ads, online video ads, e-commerce videos, video blogs, self-media videos, and the like. The target object is the object that the video to be processed focuses on describing, which can be a commodity, an item, a tutorial, a summary of experience, and the like. For example, a user who shoots a video introduces a commodity through a video, and a user who watches the video can understand the performance of the commodity, how to buy it, and the like based on the video.

[0125] In one embodiment, the computer device can obtain the videos to be processed stored locally. For example, the terminal can start the video processing application installed thereon according to the user operation and enter the video processing page, which includes a video upload entry. In response to the triggering operation of the video upload entry, the video to be processed selected from the locally stored videos is obtained. For example, refer to Figure 3 , Figure 3 FIG. 1 is a schematic diagram of the interface of a video processing page in an embodiment. As can be seen, Figure 3 The video upload entry 302 is included, and the user clicks the video upload entry 302 to select a video to be processed from the locally stored videos.

[0126] In other embodiments, the computer device may also obtain the video to be processed downloaded from the network. For example, the video processing page may include a video address input area. In response to an input operation in the video address input area, the video to be processed corresponding to the video address is obtained. The computer device may also obtain the video to be processed transmitted by other computer devices, such as the above Figure 1 The server 104 obtains the video to be processed transmitted by the terminal 102.

[0127] In one embodiment, the terminal obtains the video to be processed, and processes the video to be processed to extract the core part from the video to be processed. In another embodiment, the terminal obtains the video to be processed, and sends the video to be processed to the server, and the server processes the video to be processed to extract the core part from the video to be processed.

[0128] Step S204, after dividing the to-be-processed video into a plurality of video segments, the position information of each video segment in the to-be-processed video is obtained, and each video segment is used to describe different contents of the target object.

[0129] The computer device divides the video frames of the video to be processed into multiple parts, and the video frames contained in each part constitute a video segment. Figure 4 , Figure 4 : is a flowchart of a video processing method in an embodiment. It can be seen that the computer device first divides the video to be processed into multiple video segments such as video segment A, video segment B, and video segment C. The position information is used to describe the sequence of the video segments in the video to be processed. The position information can be a character or string composed of at least one of numbers and letters, such as sequence numbers 1, 2, 3, etc. The position information of the video segment can be determined by the sequence of the video frames contained therein in the video to be processed.

[0130] In one embodiment, the computer device divides the video to be processed into multiple video clips, and each video clip describes the target object with different content. The different content describing the target object can be specifically reflected in whether it involves the description of the target object, or in the different target objects being described, or in the different subject contents describing the same target object. For example, an e-commerce live video is divided into multiple video clips, some of which do not involve the description of the product, some of which describe the content of product A, and some of which describe the content of product B, and so on. For another example, an e-commerce live video is divided into multiple video clips, some of which do not involve the description of the product, some of which describe the function of product A, some of which describe the method of using product A, and some of which describe the effect of using product A, and so on.

[0131] The following is an introduction to the inventor's idea of ​​dividing the video to be processed into multiple video clips:

[0132] First, after studying a large number of videos, the inventors found that the relative positions of the core parts in the videos have something in common. Taking e-commerce live broadcast videos as an example, the host will usually chat and interact with the users watching the live broadcast at the beginning of the live broadcast, involving content such as weather, catering, etc. Then the host will introduce the products recommended in this live broadcast. Taking clothing as an example, the host will try them on and introduce the material, version, fitting effect, matching suggestions, etc. of the clothing in turn, and provide a purchase link. Among them, the core part of this live broadcast appears in the part where the host introduces the products recommended in this live broadcast. It can be seen that the relative position of the core part in the e-commerce live broadcast video is regular.

[0133] Secondly, in order to take advantage of the fact that the relative positions of the core parts in the video have something in common, the video needs to be divided. The video can be divided into multiple parts according to the different contents describing the target object. For example, according to whether the target object is involved, the video can be initially divided into a part that does not involve the target object and a part that involves the target object; and according to the different target objects being described, the part involving the target object can be further divided into parts that describe different target objects; and according to the different themes describing the same target object, the parts that describe different target objects can be divided into parts that describe different themes of the same target object. Taking the e-commerce live video as an example, refer to Figure 5 , Figure 5 FIG. 1 is a schematic diagram of extracting the core part from an e-commerce live video in an embodiment. The e-commerce live video can be divided into part ①, part ② and part ③. Part ① is the part where the host chats and interacts with the live viewers, that is, the part that does not involve the goods in this live broadcast; part ② and part ③ are the parts where the host introduces different goods, that is, the parts that describe different goods in this live broadcast, which include the core part of this live broadcast.

[0134] Next, due to the differences in the content presented in each part of the video, this difference is reflected in the video picture, video voice, video length, etc. Figure 5 , comparing part ①, part ② and part ③, judging only from the video screen, the proportion and position of the video subject (i.e. the anchor) on the screen, as well as factors of the video subject itself such as clothing, are different. Since the content presented in the same part of the video is similar, this similarity is mainly reflected in the video screen, video voice and other aspects. Continue to refer to Figure 5 , taking part ② as an example, the proportion of the video subject on the screen, the position, etc., as well as the factors of the video subject itself such as clothing, etc. are similar. Therefore, the video to be processed can be automatically divided by using the characteristics that the different parts of the video are different and the same part is similar.

[0135] In one embodiment, the computer device can divide the processed video by a video division strategy. The video division strategy can be a video segmentation network or a clustering algorithm. Among them, the video segmentation network is a model structure with video segmentation capability through sample learning. The input end of the video segmentation network is a video, and the output end is a plurality of video clips. The video segmentation network may include a multi-layer network structure, and different network layers perform different processing on the input data, and output the processing results to the next network layer. Regarding the specific implementation method of dividing the processed video by a clustering algorithm, reference can be made to the following embodiments, which will not be repeated here.

[0136] Since different contents describing the target object in the video have different durations, for example, the description of the function of product A is 2 minutes, and the description of the usage of product A is 5 minutes, dividing the video to be processed according to the different contents describing the target object can ensure the integrity of the contents describing the target object in the video compared to the traditional method of equally dividing the video to be processed according to the specified duration; and the position information of each video segment obtained according to this division method can reflect the possibility that the video segment contains the core part to a certain extent.

[0137] In one embodiment, obtaining position information of each video segment in the video to be processed includes: obtaining the sequence of each video frame in the video segment in the video to be processed; and determining the position information of the video segment where the video frame is located in the video to be processed according to the sequence.

[0138] Specifically, the computer device divides the video frames of the video to be processed, obtains the video segments to which each video frame belongs, and determines the position information of each video segment according to the sequence of the video frames in the video to be processed.

[0139] In a specific embodiment, the computer device determines the position information of each video segment according to the video frame that is ranked the most forward in each video segment in the video to be processed. For example, the video to be processed is divided into four video segments: video segment A, video segment B, video segment C, and video segment D, wherein video segment A includes frames 1-100 and frames 400-500, and video segment B includes frames 101-300, then the position information of video segment A can be 1, and the position information of video segment B can be 2.

[0140] Step S206: divide each video segment into a plurality of video units.

[0141] For each video segment, the computer device divides the video frames of the video segment into multiple parts, and the video frames included in each part constitute a video unit. Figure 4 It can be seen that after the computer device divides the video to be processed into multiple video segments such as video segment A, video segment B, and video segment C, for each video segment, taking video segment A as an example, it is further divided into multiple video units such as video unit A1 and video unit A2.

[0142] In one embodiment, the computer device divides each video clip into a plurality of video units, and each video unit describes a target object with different contents. The different contents describing the target object here can be specifically reflected in the different subject contents describing the same target object. For example, a video clip describes the content of commodity A, and the video clip is divided into a plurality of video units, some of which describe the function of commodity A, some of which describe the method of using commodity A, and some of which describe the effect of using commodity A, and so on. For another example, a video clip describes the function of commodity A, and the video clip is divided into a plurality of video units, some of which describe function 1 of commodity A, some of which describe function 2 of commodity A, and some of which describe function 3 of commodity A, and so on.

[0143] This application first divides the video to be processed into multiple video segments, obtains the position information of each video segment in the video to be processed, and then divides the video segment into multiple video units. In other words, this application includes two segmentation operations. The reason why this application uses two segmentation operations instead of one segmentation operation is to obtain the position information of each video segment through the first segmentation operation, and use the position information as a position feature to participate in the generation process of the core video.

[0144] For the first division operation, since the duration of different contents describing the target object in the video is different, the video to be processed is divided according to the different contents describing the target object, which can ensure the integrity of the contents describing the target object in the video, and the position information of each video segment obtained by this division method can reflect the possibility that the video segment contains the core part to a certain extent. For the second division operation, the video segment is divided into multiple video units in order to find highlights from each video segment. For example, a video segment describes the material of product A, and its duration is 20 seconds, of which 5 seconds can best highlight the advantages of the material of product A. These 5 seconds are the highlights of the video segment.

[0145] In this way, by combining the position information of each video segment obtained by the first division operation and the video features of each video unit obtained by the second division operation, the video units belonging to the core part can be accurately identified from the video to be processed.

[0146] In one embodiment, the computer device may divide the video clips through a video division strategy. The video division strategy may be a video clip segmentation network or a clustering algorithm, etc. The video clip segmentation network is a model structure that has the ability to divide video clips through sample learning. The input end of the video clip segmentation network is a video clip, and the output end is a plurality of video units. The video clip segmentation network may include a multi-layer network structure, and different network layers perform different processing on the input data, and output the processing results to the next network layer. Regarding the specific implementation method of dividing the video clips through the clustering algorithm, please refer to the following embodiments, which will not be repeated here.

[0147] Step S208: obtaining the content coreness of each video unit based on the video features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located.

[0148] The video feature is data that reflects the inherent characteristics of the video. The content coreness is used to describe the possibility that the video unit is the core part, and the content coreness is positively correlated with the possibility that the video unit is the core part. Figure 4 , it can be seen that the computer device obtains the content coreness of each video unit, and filters out the video units belonging to the core part from each video unit according to the content coreness.

[0149] In one embodiment, the video features include at least one of audio features and image features. Audio features are data reflecting the inherent characteristics of video audio. Image features are data reflecting the inherent characteristics of video images. The computer device extracts audio elements and image elements corresponding to each video frame from the video to be processed, determines the audio features corresponding to the video unit based on the audio elements corresponding to the video frames contained in the video unit, and determines the image features corresponding to the video unit based on the image elements corresponding to the video frames contained in the video unit. It can be understood that general audio features and image features can meet the requirements of the present application for video features, so general audio features and image features can be extracted from the video to be processed as video features in the present application.

[0150] In one embodiment, the computer device can extract video features from the video to be processed through a feature extraction strategy. The feature extraction strategy can be a feature extraction network or a general feature extraction algorithm, etc. Among them, the feature extraction network is a model structure with feature extraction capabilities through sample learning. The input end of the feature extraction network is a video unit, and the output end is a video feature. The feature extraction network may also include at least one of an audio feature extraction network and an image feature extraction network, the input end of the audio feature extraction network is an audio element, and the output end is an audio feature, and the input end of the image feature extraction network is an image element, and the output end is an image feature.

[0151] In one embodiment, the computer device inputs the video features corresponding to each video unit and the position information corresponding to the video clip where each video unit is located into the trained content coreness evaluation network to obtain the content coreness of each video unit. The content coreness evaluation network is a model structure that has the ability to evaluate content coreness through sample learning.

[0152] When obtaining the content coreness of the video unit, since the position information can reflect the possibility that the video segment contains the core part to a certain extent, the content coreness obtained by combining the position information corresponding to the video segment where each video unit is located is more accurate.

[0153] Step S210, determining a core video describing a target object from the videos to be processed according to the coreness of the content.

[0154] The core video is a video that only contains the core part of the video to be processed. Specifically, the computer device can select video units from high to low according to the core content of each video unit, and generate the core video according to the selected video units.

[0155] It is understood that the duration of the core video can be preset. The computer device selects the video units from high to low according to the preset duration and the core content of each video unit. If the duration of the selected video unit is greater than the preset duration, the computer device can reduce the number of video frames of the last selected video unit.

[0156] In one embodiment, the computer device generates a core video according to the sequence of the video frames in the selected video unit in the video to be processed.

[0157] The method provided in this application is compared with the traditional method, and the recall rate and precision rate are used to evaluate the extraction effect of the core part, as shown in Table 1 below:

[0158] Recall Accuracy The method provided by this application 71.5% 69.2% Traditional method 1 68.8% 62.7% Traditional method 2 67.4% 59.6%

[0159] Table 1

[0160] It can be seen that the method provided by this application can greatly improve the accuracy of extracting the core part from the video.

[0161] In the above-mentioned video processing method, when extracting the core part from the video, the characteristic that the relative position of the core part in the video has a common point is utilized. First, the video is divided into multiple video clips, so that each video clip describes different contents of the target object. Compared with the method of equally dividing the video to be processed according to the specified duration in the traditional technology, the integrity of the contents describing the target object in the video can be guaranteed; then, the position information of each video clip in the video to be processed is obtained. The position information obtained according to this division method can reflect the possibility that the video clip contains the core part; then, each video clip is divided into multiple video units respectively, and the content core degree of each video unit is obtained based on the video features corresponding to each video unit and the position information corresponding to the video clip where each video unit is located. The content core degree obtained in combination with the position information can more accurately reflect the possibility that the video unit is the core part. Therefore, determining the core video from the video to be processed according to the content core degree can improve the accuracy of extracting the core part from the video.

[0162] In one embodiment, Figure 6 As shown, the video to be processed is divided into multiple video segments, including:

[0163] Step S602, obtaining a feature vector of each video frame in the video to be processed.

[0164] The feature vector refers to the result of expressing other forms of data in mathematical form. For example, the video frame is represented as the mathematical form "[0 0 0 1 0 0 0 0 0 0 0...]", and "[0 0 0 1 0 0 0 0 0 00...]" is the result of vectorizing the video frame, that is, the feature vector of the video frame. It can be understood that there is no limitation on what kind of vector the video frame is converted into, as long as the video frame can be mathematically represented. For example, the video frame can be converted into a high-dimensional sparse vector or a low-dimensional dense vector.

[0165] Specifically, the computer device converts each video frame of the video to be processed into a feature vector. For example, if the video to be processed has N video frames, N feature vectors can be obtained.

[0166] Step S604: clustering each video frame according to the feature vector to obtain a plurality of clusters.

[0167] Among them, clustering processing is to automatically classify similar data objects into one category. A cluster is a collection of data objects.

[0168] Specifically, the computer device performs clustering processing on each feature vector, classifies similar feature vectors into the same cluster, and classifies different feature vectors into different clusters, so that the video frames corresponding to the feature vectors classified into the same cluster have similarity.

[0169] In one embodiment, the computer device can cluster each feature vector using a general clustering algorithm, such as K-Means clustering algorithm, K-means++ clustering algorithm, AP (Affinity propagation Clustering Algorithm) clustering algorithm, density-based clustering algorithm, agglomerative hierarchical clustering algorithm, divisive hierarchical clustering algorithm, and the like.

[0170] In one embodiment, step S604 includes: selecting a target number of video frames from each video frame of the video to be processed as cluster centers; clustering each video frame according to the similarity between the feature vector of each video frame and the feature vector of the cluster center to obtain a target number of cluster clusters.

[0171] Among them, the similarity between feature vectors can be represented by the vector distance between the feature vectors.

[0172] Specifically, the computer device divides each video frame of the video to be processed into a target number of video segments, each of which describes different contents of the target object. The target number can be determined according to the total duration of the video to be processed. For example, the value range of the target number can be set to: R(T / 15)≤N≤R(T / 10), where R represents an integer, T represents the total duration of the video to be processed, and N represents the target number.

[0173] Specifically, the computer device selects a target number of video frames from each video frame of the video to be processed, uses the feature vectors of the selected video frames as cluster centers, and calculates the similarity between the feature vectors of the video frames and the feature vectors of each cluster center for each remaining video frame, and divides the feature vectors of the video frames into cluster clusters corresponding to the cluster center with the greatest similarity. For each cluster cluster, each time a feature vector of a video frame is added, the cluster center corresponding to the cluster cluster is recalculated until the feature vectors of all video frames are assigned.

[0174] In one embodiment, the computer device may randomly select a target number of video frames and use the feature vectors of the selected video frames as cluster centers. In another embodiment, the computer device may first randomly select one of the video frames, use the feature vector of the video frame as the cluster center, and calculate the similarity between the feature vectors of the remaining video frames and the feature vector of the cluster center. The greater the similarity, the greater the possibility that the feature vector of the video frame will be the next cluster center. The feature vector of the video frame with the greatest possibility is selected as the next cluster center based on the roulette mode until the target number of cluster centers are selected.

[0175] Specifically, after the computer device divides each video frame of the video to be processed into a target number of clusters, it evaluates the clustering result.

[0176] In one embodiment, the method further includes: after clustering processing is performed according to the target number and video segments are obtained, traversing each video frame of the video to be processed; determining the intra-cluster similarity between the traversed current video frame and other video frames in the same video segment; determining the extra-cluster similarity between the traversed current video frame and other video segments; determining the clustering coefficient corresponding to the current video frame according to the difference between the intra-cluster similarity and the extra-cluster similarity; determining the clustering score corresponding to the clustering processing according to the target number according to the clustering coefficient corresponding to each video frame in the video to be processed obtained after the traversal is completed; and determining multiple video segments corresponding to the video to be processed according to the cluster cluster corresponding to the target number when the clustering score is the highest.

[0177] Among them, the intra-cluster similarity is used to describe the similarity between the current video frame and other video frames in the same video clip. The intra-cluster similarity can be the similarity between the current video frame and one of the video frames in the same video clip, or it can be the average of the similarities between the current video frame and more than one video frames in the same video clip. The larger the intra-cluster similarity, the greater the possibility that the current video frame belongs to the current video clip. The out-cluster similarity is used to describe the similarity between the current video frame and video frames in other video clips. The out-cluster similarity can be the similarity between the current video frame and one of the video frames in other video clips, or it can be the average of the similarities between the current video frame and more than one video frames in other video clips, or it can be the extreme value or average of the similarity averages between the current video frame and more than one other video clips. The smaller the out-cluster similarity, the greater the possibility that the current video frame belongs to the current video clip, and the smaller the possibility that it belongs to other video clips.

[0178] In one embodiment, the computer device traverses each video frame of the video to be processed, and determines whether the current video frame i is related to the same video segment (such as F a ) is the mean similarity V between other video frames in ia , V ia As the intra-cluster similarity. Determine the current video frame i and other video clips (such as F b 、F c ) respectively correspond to the mean similarity V ib 、V ic , from V ib 、V ic Select the minimum value V i-min as the out-of-cluster similarity.

[0179] Specifically, the computer device determines the clustering coefficient corresponding to the current video frame according to the difference between the similarity within the cluster and the similarity outside the cluster. In one embodiment, the computer device determines the clustering coefficient corresponding to the current video frame according to the difference between the similarity within the cluster and the similarity outside the cluster. For example, the clustering coefficient S i It can be calculated according to the following formula:

[0180]

[0181] Specifically, the computer device determines the clustering score of this clustering according to the clustering coefficients corresponding to each video frame in the to-be-processed video obtained after the traversal. In one embodiment, the computer device uses the sum of the clustering coefficients corresponding to each video frame as the clustering score of this clustering.

[0182] Specifically, the computer device performs multiple clustering on each video to be processed, and selects the clustering result corresponding to the highest clustering score as the final clustering result.

[0183] In this embodiment, multiple clustering is performed on each video frame of the video to be processed, and according to the clustering score of each clustering, the clustering result with the best clustering effect is selected as the final clustering result, thereby improving the accuracy of the division of the video to be processed.

[0184] Step S606 , obtaining a plurality of video segments according to the video frames classified into the same cluster.

[0185] Specifically, the computer device uses the video frames corresponding to the feature vectors classified into the same cluster as video segments, thereby obtaining multiple video segments describing different contents of the target object.

[0186] In this embodiment, the video to be processed is divided into multiple video segments by using a clustering algorithm, so that the multiple video segments respectively describe different contents of the target object, thereby improving the accuracy of the position information determined after the division.

[0187] In one embodiment, Figure 7 As shown, each video clip is divided into a plurality of video units, including:

[0188] Step S702: Obtain a feature vector of each video frame in the video clip.

[0189] Specifically, the computer device converts each video frame of the video clip into a feature vector. For example, if the video clip includes N video frames, N feature vectors can be obtained.

[0190] Step S704: clustering each video frame in the video clip according to the feature vector to obtain a plurality of clusters.

[0191] Specifically, the computer device performs clustering processing on each feature vector, classifies similar feature vectors into the same cluster, and classifies different feature vectors into different clusters, so that the video frames corresponding to the feature vectors classified into the same cluster have similarity.

[0192] In one embodiment, the computer device can cluster each feature vector using a general clustering algorithm, such as K-Means clustering algorithm, K-means++ clustering algorithm, AP (Affinity propagation Clustering Algorithm) clustering algorithm, density-based clustering algorithm, agglomerative hierarchical clustering algorithm, divisive hierarchical clustering algorithm, and the like.

[0193] In one embodiment, each video frame in a video clip is clustered according to a feature vector to obtain a plurality of cluster clusters, including: selecting a target number of video frames from each video frame in the video clip as cluster centers; and clustering each video frame in the video clip according to a similarity between a feature vector of each video frame and a feature vector of a cluster center to obtain a target number of cluster clusters.

[0194] Among them, the similarity between feature vectors can be represented by the vector distance between the feature vectors.

[0195] Specifically, the computer device divides each video frame of the video clip into a target number of video units. The target number can be determined according to the duration of the video clip. For example, the value range of the target number can be set to: R(t / 15)≤n≤R(t / 10), where R represents an integer, t represents the duration of the video clip, and n represents the target number.

[0196] Specifically, the computer device selects a target number of video frames from each video frame of the video clip, uses the feature vectors of the selected video frames as cluster centers, and for each remaining video frame, calculates the similarity between the feature vectors of the video frame and the feature vectors of each cluster center, and divides the feature vectors of the video frame into cluster clusters corresponding to the cluster center with the greatest similarity. For each cluster cluster, each time a feature vector of a video frame is added, the cluster center corresponding to the cluster cluster is recalculated until the feature vectors of all video frames are assigned.

[0197] In one embodiment, the computer device may randomly select a target number of video frames and use the feature vectors of the selected video frames as cluster centers. In another embodiment, the computer device may first randomly select one of the video frames, use the feature vector of the video frame as the cluster center, and calculate the similarity between the feature vectors of the remaining video frames and the feature vector of the cluster center. The greater the similarity, the greater the possibility that the feature vector of the video frame will be the next cluster center. The feature vector of the video frame with the greatest possibility is selected as the next cluster center based on the roulette mode until the target number of cluster centers are selected.

[0198] Specifically, after the computer device divides each video frame of the video clip into a target number of clusters, it evaluates the clustering result.

[0199] In one embodiment, the method further includes: after clustering processing is performed according to the target number and video units are obtained, traversing each video frame of the video clip; determining the intra-cluster similarity between the traversed current video frame and other video frames in the same video unit; determining the extra-cluster similarity between the traversed current video frame and other video units; determining the clustering coefficient corresponding to the current video frame based on the difference between the intra-cluster similarity and the extra-cluster similarity; determining the clustering score corresponding to the clustering processing according to the target number based on the clustering coefficient corresponding to each video frame in the video clip obtained after the traversal is completed; and determining multiple video units corresponding to the video clip based on the cluster cluster corresponding to the target number when the clustering score is the highest.

[0200] Among them, the intra-cluster similarity is used to describe the similarity between the current video frame and other video frames in the same video unit. The intra-cluster similarity can be the similarity between the current video frame and one of the video frames in the same video unit, or it can be the average of the similarities between the current video frame and more than one video frames in the same video unit. The larger the intra-cluster similarity, the greater the possibility that the current video frame belongs to the current video unit. The out-cluster similarity is used to describe the similarity between the current video frame and video frames in other video units. The out-cluster similarity can be the similarity between the current video frame and one of the video frames in other video units, or it can be the average of the similarities between the current video frame and more than one video frames in other video units, or it can be the extreme value or average of the similarity averages between the current video frame and more than one other video units. The smaller the out-cluster similarity, the greater the possibility that the current video frame belongs to the current video unit, and the smaller the possibility that it belongs to other video units.

[0201] In one embodiment, the computer device traverses each video frame of the video segment and determines whether the current video frame i is related to the same video unit (such as U a ) is the mean similarity V between other video frames in ia , V ia As the intra-cluster similarity. Determine the current video frame i and other video units (such as U b , U c ) respectively correspond to the mean similarity V ib 、V ic , from V ib 、V ic Select the minimum value V i-min as the out-of-cluster similarity.

[0202] Specifically, the computer device determines the clustering coefficient corresponding to the current video frame according to the difference between the similarity within the cluster and the similarity outside the cluster. In one embodiment, the computer device determines the clustering coefficient corresponding to the current video frame according to the difference between the similarity within the cluster and the similarity outside the cluster. For example, the clustering coefficient S i It can be calculated according to the following formula:

[0203]

[0204] Specifically, the computer device determines the clustering score of this clustering according to the clustering coefficients corresponding to each video frame in the video segment obtained after the traversal. In one embodiment, the computer device uses the sum of the clustering coefficients corresponding to each video frame as the clustering score of this clustering.

[0205] Specifically, the computer device clusters the videos of the video clip multiple times, and selects the clustering result corresponding to the highest clustering score as the final clustering result.

[0206] In this embodiment, the video frames of the video clips are clustered multiple times, and according to the clustering scores of each clustering, the clustering result with the best clustering effect is selected as the final clustering result, thereby improving the accuracy of the video clip division.

[0207] Step S706 , obtaining a plurality of video units according to the video frames classified into the same cluster.

[0208] Specifically, the computer device uses the video frames corresponding to the feature vectors classified into the same cluster as video units, thereby obtaining multiple video units.

[0209] In this embodiment, the video segment is divided into a plurality of video units by using a clustering algorithm, thereby improving the accuracy of the video segment division.

[0210] In one embodiment, the method further includes: inputting the video to be processed into a trained core video generation model; extracting feature vectors of each video frame in the video to be processed through a feature vector generation network in the core video generation model to obtain feature vectors corresponding to each video frame.

[0211] The core video generation model is a model structure that has the ability to extract core videos through sample learning. The feature vector generation network is a part of the model structure in the core video generation model, which has the ability to generate feature vectors based on input video frames. The feature vector generation network may include a multi-layer network structure, where different network layers perform different processing on the input data and output the processing results to the next network layer.

[0212] Specifically, the computer device inputs the video to be processed into a trained core video generation model, and extracts feature vectors of each video frame in the video to be processed through the feature vector generation network of the core video generation model to obtain the feature vectors corresponding to each video frame, so as to perform clustering processing on the video frames through the feature vectors.

[0213] In this embodiment, each video frame is mapped to the same feature vector space through a feature vector generation network, so as to facilitate operations such as clustering processing on each video frame.

[0214] In one embodiment, Figure 8 As shown, based on the video features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located, the content coreness of each video unit is obtained, including:

[0215] Step S802 : determining the image core score of each video unit according to the image features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located.

[0216] Among them, image features may include texture features, color features, gradient features, spatial relationship features, etc. Texture features describe the surface properties of each target in the image. Color features describe the color of each target in the image. Gradient features describe the shape and structure of each target in the image. Spatial relationship features refer to the spatial position or relative direction relationship between multiple targets segmented from the image. These relationships can also be divided into connection / adjacent relationships, overlap / overlapping relationships, and inclusion / containment relationships.

[0217] In one embodiment, the computer device extracts the image elements corresponding to each video frame from the video to be processed, and determines the image features corresponding to the video unit based on the image elements corresponding to the video frames contained in the video unit. The computer device can extract image features through an image feature extraction strategy. The image feature extraction strategy can be an image feature extraction network or a general image feature extraction algorithm. For example, texture features can be extracted through grayscale co-occurrence matrix method, Voronio checkerboard feature method, wavelet transform method, etc., color features can be extracted through color histogram method, color set method, color moment method, etc., gradient features can be extracted through boundary feature method, Fourier shape descriptor method, geometric parameter method, shape invariant moment method, etc., and spatial relationship features can be extracted by segmenting the image, dividing the objects or color areas contained in the image, extracting image features based on these areas and establishing indexes.

[0218] Specifically, the computer device determines the image core score of each video unit according to the image features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located through an image evaluation strategy. The image evaluation strategy may be an image evaluation algorithm or an image evaluation network.

[0219] In one embodiment, step S802 includes: obtaining a feature vector corresponding to the position information of each video unit; inputting the image features and feature vectors corresponding to each video unit into the image evaluation network in the trained core video generation model in turn to obtain the image core score of each video unit.

[0220] The image core score reflects the content coreness of the image elements of the video unit. The image evaluation network is a part of the model structure in the core video generation model, which has the ability to generate an image core score based on the feature vector corresponding to the input image features and position information. The image evaluation network may include a multi-layer network structure, where different network layers perform different processing on the input data and output the processing results to the next network layer.

[0221] In one embodiment, the feature vector corresponding to the position information of the video unit can be obtained through the feature vector generation network in the core video generation model. The computer device inputs the position information of the video unit into the trained core video generation model, extracts the feature vector of the position information through the feature vector generation network of the core video generation model, and obtains the feature vector corresponding to the position information.

[0222] Specifically, the computer device inputs the image features corresponding to the video unit and the feature vector corresponding to the position information of the video unit into the image evaluation network in the trained core video generation model to obtain the image core score of the video unit.

[0223] Step S804: determining the audio core score of each video unit according to the audio features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located.

[0224] Among them, audio features may include zero-crossing rate features, short-time energy features, short-time autocorrelation function features, short-time average amplitude difference features, spectrogram features, short-time power spectral density features, spectral entropy features, fundamental frequency features, resonance peak features, and the like.

[0225] In one embodiment, the computer device extracts audio elements corresponding to each video frame from the video to be processed, and determines the audio features corresponding to the video unit based on the audio elements corresponding to the video frames contained in the video unit. The computer device can extract audio features through an audio feature extraction strategy. The audio feature extraction strategy can be an audio feature extraction network or a general audio feature extraction algorithm.

[0226] Specifically, the computer device determines the audio core score of each video unit according to the audio features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located through an audio evaluation strategy. The audio evaluation strategy may be an audio evaluation algorithm or an audio evaluation network.

[0227] In one embodiment, step S804 includes: obtaining a feature vector corresponding to the position information of each video unit; inputting the audio features and feature vectors corresponding to each video unit into the audio evaluation network in the trained core video generation model in turn to obtain the audio core score of each video unit.

[0228] The audio core score reflects the content coreness of the audio element of the video unit. The audio evaluation network is a part of the model structure in the core video generation model, which has the ability to generate an audio core score based on the feature vector corresponding to the input audio features and position information. The audio evaluation network may include a multi-layer network structure, where different network layers perform different processing on the input data and output the processing results to the next network layer.

[0229] Specifically, the computer device inputs the audio features corresponding to the video unit and the feature vector corresponding to the position information of the video unit into the audio evaluation network in the trained core video generation model to obtain the audio core score of the video unit.

[0230] Step S806: The image core score and the audio core score of each video unit are integrated to obtain the content core degree of each video unit.

[0231] Specifically, the computer device determines the content core degree of each video unit according to the image core score and the audio core score of each video unit, such as obtaining the sum of the image core score and the audio core score of each video unit, and so on.

[0232] In one embodiment, step S806 includes: obtaining weights corresponding to the image core score and the audio core score respectively; and performing weighted summation of the image core score and the audio core score of each video unit according to the weights to obtain the content core degree of each video unit.

[0233] Among them, the weights corresponding to the image core score and the audio core score can be set according to actual applications. For example, the weight corresponding to the image core score can be 0.7, and the weight corresponding to the audio core score can be 0.3.

[0234] Specifically, the computer device performs weighted summation of the image core score and the audio core score of each video unit according to the weights corresponding to the image core score and the audio core score, so as to obtain the content core degree of each video unit.

[0235] In this embodiment, based on the image features, audio features and position information corresponding to each video unit, the core content of the image elements and audio elements of each video unit is evaluated, so that the core part can be accurately screened out from each video unit.

[0236] In one embodiment, the core video may be obtained through a core video generation model. The steps of obtaining the core video through the core video generation model are introduced below.

[0237] Referring to Figure 9(a), Figure 9(a) is a schematic diagram of the structure of the core video generation model in one embodiment. It can be seen that the core video generation model may include a feature vector generation network 902, a feature vector generation network 910, an image evaluation network 912, and an audio evaluation network 914. As for the video division part 904 in the core video generation model, it can be executed by a clustering algorithm or a video segmentation network; as for the video segment division part 906 in the core video generation model, it can be executed by a clustering algorithm or a video segmentation network; as for the video feature extraction part 908 in the core video generation model, it can be executed by an image feature extraction algorithm and an audio feature extraction algorithm, or by an image feature extraction network and an audio feature extraction network.

[0238] In one embodiment, the image evaluation network 912 and the audio evaluation network 914 may use the same network structure, such as a deep neural network, DeepFM, or other model structures. Both the image evaluation network and the audio evaluation network may include a part for extracting low-order features and a part for extracting high-order features, thereby being able to extract rich layers of image features.

[0239] In one embodiment, taking the DeepFM model structure as an example, referring to FIG9(b), the image evaluation network 912 may include a factor decomposition machine part 9121 for extracting low-order features and a feedforward neural network part 9122 for extracting high-order features. The factor decomposition machine part 9121 and the feedforward neural network part 9122 share the features provided by the feature vector generation network 910. For the image evaluation network 912, the input is the input vector x consisting of the image features and the position information. o For the audio evaluation network 914, the input is the input vector x consisting of audio features and position information o Taking the image evaluation network 912 as an example, the vector factorization machine part processes the input vector x o Perform feature cross processing to obtain low-order features. The feedforward neural network uses different network layers to process the input vector x. o Different processing is performed to obtain high-order features. Specifically, the feedforward neural network part processes the input vector x o The processing can be expressed by the following formula:

[0240] h1=RELU(w h,o *x o +b h,o )

[0241] Among them, h1 is the processing result of the first network layer of the feedforward neural network part; w h,o is the model weight; b h,ois a constant coefficient. The deep feature crossover of image features and position information is performed through a neural network to obtain high-order features.

[0242] Specifically, the factorization machine part performs the operation on the input vector x o The processing can be expressed by the following formula:

[0243] x c =x oi *x oj

[0244] Among them, x c is the processing result of the first network layer of the factorization machine part; x oi is the input vector x o The eigenvalue in the i-th feature dimension; x oj is the input vector x o The eigenvalue at the jth feature dimension.

[0245] The prediction results of the factorization machine part can be expressed by the following formula:

[0246]

[0247] Among them, y FM is the prediction result of the factorization machine part; w FM is the model weight; x ci is the eigenvector x c The eigenvalue in the i-th feature dimension; x cj is the eigenvector x c The eigenvalue in the jth feature dimension; v i is the representation vector in the i-th feature dimension; v j is the representation vector in the jth feature dimension. The inner product unit in the factor decomposition machine performs a second-order feature crossover on the image features and position information in the input vector to obtain the second-order features, i.e., low-order features.

[0248] Specifically, the prediction effect of the image evaluation network, that is, the image core score, can be expressed by the following formula:

[0249] P = sigmoid(y FM +y DNN )

[0250] Among them, P is the prediction effect of the image evaluation network, y FM is the prediction result of the factorization machine part, y DNN It is the prediction result of the feedforward neural network part.

[0251] Specifically, when obtaining the core video from the video to be processed, first, the computer device inputs the video to be processed into the trained core video generation model, and extracts the feature vectors of each video frame in the video to be processed through the feature vector generation network 902 in the core video generation model to obtain the feature vectors corresponding to each video frame.

[0252] Next, the computer device divides the video to be processed into multiple video segments through the video dividing part 904, each video segment describes different content of the target object, and the computer device obtains the position information of each video segment in the video to be processed.

[0253] Furthermore, the computer device divides each video segment into a plurality of video units through the video segment dividing part 906 .

[0254] Next, the computer device extracts the image features and audio features corresponding to each video unit through the video feature extraction part 908.

[0255] Furthermore, the computer device extracts feature vectors from the position information of the video clip where each video unit is located through the feature vector generation network 910 to obtain feature vectors of the position information of each video unit.

[0256] Next, the computer device inputs the feature vectors of the image features and position information of each video unit into the image evaluation network 912 to obtain the image core score of each video unit, and inputs the feature vectors of the audio features and position information of each video unit into the audio evaluation network 914 to obtain the audio core score of each video unit.

[0257] Finally, the computer device fuses the image core score and the audio core score of each video unit to obtain the content core degree of each video unit, and determines the core video describing the target object from the video to be processed based on the content core degree.

[0258] In one embodiment, the training steps of the core video generation model include: obtaining a sample video and an annotated core video corresponding to the sample video; inputting the sample video into the core video generation model; after dividing the sample video into multiple video clips through the core video generation model, obtaining the position information of each video clip in the sample video, dividing each video clip into multiple video units, obtaining the content coreness of each video unit based on the video features corresponding to each video unit and the position information corresponding to the video clip where each video unit is located, and determining the predicted core video describing the target object from the sample video according to the content coreness; constructing a loss function based on the annotated core video and the predicted core video; after updating the model parameters of the core video generation model according to the loss function, returning to the step of obtaining the sample video and the annotated core video corresponding to the sample video to continue training until the training stop condition is met, thereby obtaining a trained core video generation model.

[0259] The sample videos are sample data used to train the core video generation model. The sample videos include but are not limited to TV video ads, online video ads, e-commerce videos, video blogs, self-media videos, etc.

[0260] Specifically, continuing to refer to Figure 9(a), first, the computer device inputs the sample video into the core video generation model to be trained, and through the feature vector generation network 902 in the core video generation model, the feature vector of each video frame in the sample video is extracted to obtain the feature vector corresponding to each video frame.

[0261] Next, the computer device divides the sample video into multiple video segments through the video segmentation part 904, each video segment describes different content of the target object, and the computer device obtains the position information of each video segment in the sample video.

[0262] Furthermore, the computer device divides each video segment into a plurality of video units through the video segment dividing part 906 .

[0263] Next, the computer device extracts the image features and audio features corresponding to each video unit through the video feature extraction part 908.

[0264] Furthermore, the computer device extracts feature vectors from the position information of the video clip where each video unit is located through the feature vector generation network 910 to obtain feature vectors of the position information of each video unit.

[0265] Next, the computer device inputs the feature vectors of the image features and position information of each video unit into the image evaluation network 912 to obtain the image core score of each video unit, and inputs the feature vectors of the audio features and position information of each video unit into the audio evaluation network 914 to obtain the audio core score of each video unit.

[0266] Furthermore, the computer device fuses the image core score and the audio core score of each video unit to obtain the content core degree of each video unit, and determines the predicted core video describing the target object from the sample videos based on the content core degree.

[0267] Finally, the computer device constructs a loss function based on the labeled core video and the predicted core video, updates the model parameters of the core video generation model in the direction of minimizing the loss function, and returns to the step of obtaining the sample video and the labeled core video corresponding to the sample video to continue training until the training stop condition is met.

[0268] In this embodiment, the core video generation model is trained through sample videos so that the core video generation model has the ability to accurately obtain the core video based on the video to be processed; and the core video generation model is trained in combination with the position information of each video segment obtained by the first division operation and the video features of each video unit obtained by the second division operation. In practical applications, this can improve the accuracy of the core video generation model in identifying the video units belonging to the core part from the video to be processed.

[0269] In one embodiment, a core video describing a target object is determined from a video to be processed based on the core degree of the content, including: determining a core degree distribution curve corresponding to the video to be processed based on each video unit and the corresponding core degree of the content; screening video units having a content core degree higher than a threshold from the divided video units based on the core degree distribution curve; and generating a core video describing the target object based on the screened video units.

[0270] The horizontal coordinate of the core degree distribution curve may be each video unit, and the vertical coordinate may be the core degree of the content of each video unit.

[0271] Specifically, refer to Fig.10 , Fig.10 The flowchart of determining the core degree distribution curve in one embodiment is shown. After obtaining the corresponding content core degree of each video unit, the computer device can generate a core degree distribution curve according to each video unit and its corresponding content core degree. The core degree distribution curve can clearly reflect the content core degree of each video unit. According to the core degree distribution curve, the computer device selects a video unit whose content core degree is higher than a threshold from each video unit, and generates a core video according to the selected video unit.

[0272] In this embodiment, based on the coreness distribution curve, video units with content coreness higher than a threshold can be quickly and accurately screened out.

[0273] In one embodiment, the method further includes: after dividing each video clip into a plurality of video units, obtaining position information of each video unit in the video to be processed; and obtaining the content coreness of each video unit based on the video features corresponding to each video unit and the position information of each video unit in the video to be processed.

[0274] In this embodiment, when extracting the core part from the video, the characteristic that the relative position of the core part in the video has a common point is utilized. First, the video is divided into multiple video clips, so that each video clip describes different content of the target object; then, each video clip is divided into multiple video units, and the position information of each video unit in the video to be processed is obtained. The position information obtained according to this division method can reflect the possibility that the video unit contains the core part; then, based on the video features corresponding to each video unit and the position information of each video unit in the video to be processed, the content core degree of each video unit is obtained. The content core degree obtained in combination with the position information can more accurately reflect the possibility that the video unit is the core part. Therefore, determining the core video from the video to be processed according to the content core degree can improve the accuracy of extracting the core part from the video.

[0275] In one embodiment, obtaining a video to be processed includes: obtaining a video to be processed that describes a product; determining a core video that describes a target object from the video to be processed based on the core degree of the content, including: determining a video unit that describes key information of the product based on the core degree of the content; and generating a core video that describes key information of the product based on the determined video unit.

[0276] Specifically, the embodiments of the present application can be applied to the e-commerce field to extract the core parts of e-commerce videos.

[0277] In one embodiment, referring to Fig.11 , provides a video processing method, this embodiment mainly applies the method to a computer device (the above Figure 1 The terminal 102 or server 104 in the example is used as an example, and the steps include:

[0278] Step S1102, obtaining a video to be processed that describes the product.

[0279] Step S1104, input the video to be processed into the trained core video generation model, extract feature vectors from each video frame in the video to be processed through the feature vector generation network in the core video generation model, and obtain feature vectors corresponding to each video frame.

[0280] Step S1106, selecting a target number of video frames from each video frame of the video to be processed as cluster centers; clustering each video frame according to the similarity between the feature vector of each video frame and the feature vector of the cluster center to obtain a target number of cluster clusters; and obtaining multiple video clips based on the video frames classified into the same cluster cluster, each video clip is used to describe different content of the product.

[0281] Step S1108, after clustering processing is performed according to the target number and video segments are obtained, each video frame of the video to be processed is traversed; the intra-cluster similarity between the traversed current video frame and other video frames in the same video segment is determined, and the extra-cluster similarity between the traversed current video frame and other video segments is determined; according to the difference between the intra-cluster similarity and the extra-cluster similarity, the clustering coefficient corresponding to the current video frame is determined; according to the clustering coefficient corresponding to each video frame in the video to be processed obtained after the traversal is completed, the clustering score corresponding to the clustering processing according to the target number is determined; according to the cluster cluster corresponding to the target number when the clustering score is the highest, multiple video segments corresponding to the video to be processed are determined.

[0282] Step S1110, after dividing the to-be-processed video into a plurality of video segments, obtaining the sequence of each video frame in the video segment in the to-be-processed video, and determining the position information of the video segment where the video frame is located in the to-be-processed video according to the sequence.

[0283] Step S1112, obtaining the feature vector of each video frame in the video clip, selecting a target number of video frames from each video frame in the video clip as clustering centers; clustering each video frame in the video clip according to the similarity between the feature vector of each video frame and the feature vector of the clustering center to obtain a target number of cluster clusters; and obtaining multiple video units according to the video frames divided into the same cluster cluster.

[0284] Step S1114, after clustering processing is performed according to the target number and video units are obtained, traverse each video frame of the video clip; determine the intra-cluster similarity between the traversed current video frame and other video frames in the same video unit; determine the extra-cluster similarity between the traversed current video frame and other video units; determine the clustering coefficient corresponding to the current video frame based on the difference between the intra-cluster similarity and the extra-cluster similarity; determine the clustering score corresponding to the clustering processing according to the target number based on the clustering coefficients corresponding to each video frame in the video clip obtained after the traversal is completed; determine multiple video units corresponding to the video clip based on the cluster clusters corresponding to the target number when the clustering score is the highest.

[0285] Step S1116, obtain the feature vector corresponding to the position information of each video unit, and input the image features and feature vectors corresponding to each video unit into the image evaluation network in the trained core video generation model in turn to obtain the image core score of each video unit; obtain the feature vector corresponding to the position information of each video unit, and input the audio features and feature vectors corresponding to each video unit into the audio evaluation network in the trained core video generation model in turn to obtain the audio core score of each video unit; obtain the weights corresponding to the image core score and the audio core score respectively; perform weighted summation of the image core score and the audio core score of each video unit according to the weights to obtain the content core degree of each video unit.

[0286] Step S1118, determining the core degree distribution curve corresponding to the video to be processed according to each video unit and the corresponding content core degree; filtering out video units whose content core degrees are higher than a threshold value from the divided video units according to the core degree distribution curve; generating a core video describing key information of the product according to the filtered video units.

[0287] In this embodiment, when extracting the core part from the e-commerce video, the common property of the e-commerce video is utilized, that is, the relative position of the core part in the e-commerce video has a common point. First, the video is divided into multiple video clips, so that each video clip describes different content of the target object. Compared with the method of equally dividing the video to be processed according to the specified duration in the traditional technology, the integrity of the contents describing the target object in the video can be guaranteed; then, the position information of each video clip in the video to be processed is obtained. The position information obtained according to this division method can reflect the possibility that the video clip contains the core part; then, each video clip is divided into multiple video units, and the content core degree of each video unit is obtained based on the video features corresponding to each video unit and the position information corresponding to the video clip where each video unit is located. The content core degree obtained in combination with the position information can more accurately reflect the possibility that the video unit is the core part. Therefore, determining the core video from the video to be processed according to the content core degree can improve the accuracy of extracting the core part of the e-commerce video.

[0288] It should be understood that although Figure 2 , 6 The steps in the flowcharts of -8 and 11 are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , 6-At least part of the steps in 8 and 11 may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0289] In one embodiment, Fig.12 As shown, a video processing device is provided, which can be a part of a computer device using a software module or a hardware module, or a combination of the two. The device specifically includes: an acquisition module 1202, a division module 1204 and a determination module 1206, wherein:

[0290] An acquisition module 1202 is used to acquire a video to be processed, where the video to be processed is used to describe a target object;

[0291] A division module 1204 is used to divide the video to be processed into multiple video segments, and obtain the position information of each video segment in the video to be processed, each video segment is used to describe different contents of the target object;

[0292] The division module 1204 is further used to divide each video segment into a plurality of video units;

[0293] The acquisition module 1202 is further used to obtain the content coreness of each video unit based on the video features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located;

[0294] The determination module 1206 is used to determine the core video describing the target object from the videos to be processed according to the coreness of the content.

[0295] In one embodiment, the division module 1204 is further used to: obtain a feature vector of each video frame in the video to be processed; cluster each video frame according to the feature vector to obtain multiple clusters; and obtain multiple video clips according to the video frames divided into the same cluster.

[0296] In one embodiment, the division module 1204 is also used to: select a target number of video frames from each video frame of the video to be processed as cluster centers; cluster each video frame according to the similarity between the feature vector of each video frame and the feature vector of the cluster center to obtain a target number of cluster clusters.

[0297] In one embodiment, the division module 1204 is also used to: after clustering processing is performed according to the target number and video segments are obtained, traverse each video frame of the video to be processed; determine the intra-cluster similarity between the traversed current video frame and other video frames in the same video segment; determine the extra-cluster similarity between the traversed current video frame and other video segments; determine the clustering coefficient corresponding to the current video frame according to the difference between the intra-cluster similarity and the extra-cluster similarity; determine the clustering score corresponding to the clustering processing according to the target number according to the clustering coefficient corresponding to each video frame in the video to be processed obtained after the traversal is completed; determine multiple video segments corresponding to the video to be processed according to the cluster cluster corresponding to the target number when the clustering score is the highest.

[0298] In one embodiment, the division module 1204 is further used to: obtain a feature vector of each video frame in the video clip; cluster each video frame in the video clip according to the feature vector to obtain multiple clustering clusters; and obtain multiple video units according to the video frames divided into the same clustering cluster.

[0299] In one embodiment, the division module 1204 is also used to: select a target number of video frames from each video frame in the video clip as clustering centers; cluster each video frame in the video clip according to the similarity between the feature vector of each video frame and the feature vector of the clustering center, to obtain a target number of cluster clusters.

[0300] In one embodiment, the division module 1204 is further used to: after clustering processing is performed according to the target number and video units are obtained, traverse each video frame of the video clip; determine the intra-cluster similarity between the traversed current video frame and other video frames in the same video unit; determine the extra-cluster similarity between the traversed current video frame and other video units; determine the clustering coefficient corresponding to the current video frame based on the difference between the intra-cluster similarity and the extra-cluster similarity; determine the clustering score corresponding to the clustering processing according to the target number based on the clustering coefficients corresponding to each video frame in the video clip obtained after the traversal is completed; and determine multiple video units corresponding to the video clip based on the cluster clusters corresponding to the target number when the clustering score is the highest.

[0301] In one embodiment, the video processing device also includes a model processing module, which is further used to: input the video to be processed into a trained core video generation model; extract feature vectors of each video frame in the video to be processed through a feature vector generation network in the core video generation model, and obtain feature vectors corresponding to each video frame.

[0302] In one embodiment, the division module 1204 is further used to: obtain the sequence of each video frame in the video segment in the video to be processed; and determine the position information of the video segment where the video frame is located in the video to be processed according to the sequence.

[0303] In one embodiment, the acquisition module 1202 is also used to: determine the image core score of each video unit based on the image features corresponding to each video unit and the position information corresponding to the video clip where each video unit is located; determine the audio core score of each video unit based on the audio features corresponding to each video unit and the position information corresponding to the video clip where each video unit is located; and fuse the image core score and the audio core score of each video unit to obtain the content core degree of each video unit.

[0304] In one embodiment, the acquisition module 1202 is also used to: obtain the feature vector corresponding to the position information of each video unit; input the image features and feature vectors corresponding to each video unit into the image evaluation network in the trained core video generation model in turn to obtain the image core score of each video unit.

[0305] In one embodiment, the acquisition module 1202 is also used to: obtain the feature vector corresponding to the position information of each video unit; input the audio features and feature vectors corresponding to each video unit into the audio evaluation network in the trained core video generation model in turn to obtain the audio core score of each video unit.

[0306] In one embodiment, the acquisition module 1202 is further used to: obtain the weights corresponding to the image core score and the audio core score respectively; and perform weighted summation of the image core score and the audio core score of each video unit according to the weights to obtain the content core degree of each video unit.

[0307] In one embodiment, the determination module 1206 is also used to: determine the core degree distribution curve corresponding to the video to be processed based on each video unit and the corresponding content core degree; filter out video units with content core degrees higher than a threshold from the divided video units based on the core degree distribution curve; and generate a core video describing the target object based on the filtered video units.

[0308] In one embodiment, the division module 1204 is also used to: obtain the position information of each video unit in the video to be processed after dividing each video clip into multiple video units; obtain the content coreness of each video unit based on the video features corresponding to each video unit and the position information of each video unit in the video to be processed.

[0309] In one embodiment, the acquisition module 1202 is further used to: acquire a video to be processed that describes a product; the determination module 1206 is further used to: determine a video unit that describes key information of the product based on the core degree of the content; and generate a core video that describes key information of the product based on the determined video unit.

[0310] In one embodiment, the core video is obtained through a core video generation model, and the video processing device also includes a model training module, which is used to: obtain a sample video and a labeled core video corresponding to the sample video; input the sample video into the core video generation model; after dividing the sample video into multiple video clips through the core video generation model, obtain the position information of each video clip in the sample video, divide each video clip into multiple video units, obtain the content core degree of each video unit based on the video features corresponding to each video unit and the position information corresponding to the video clip where each video unit is located, and determine the predicted core video describing the target object from the sample video according to the content core degree; construct a loss function based on the labeled core video and the predicted core video; after updating the model parameters of the core video generation model according to the loss function, return to the step of obtaining the sample video and the labeled core video corresponding to the sample video to continue training until the training stop condition is met, and a trained core video generation model is obtained.

[0311] For the specific definition of the video processing device, please refer to the definition of the video processing method above, which will not be repeated here. Each module in the above video processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0312] The above-mentioned video processing device, when extracting the core part from the video, utilizes the characteristic that the relative position of the core part in the video has a common point. First, the video is divided into multiple video clips, so that each video clip describes different content of the target object. Compared with the method of equally dividing the video to be processed according to the specified duration in the traditional technology, it can ensure the integrity of the various contents describing the target object in the video; then, the position information of each video clip in the video to be processed is obtained. The position information obtained according to this division method can reflect the possibility that the video clip contains the core part; then, each video clip is divided into multiple video units, and the content core degree of each video unit is obtained based on the video features corresponding to each video unit and the position information corresponding to the video clip where each video unit is located. The content core degree obtained in combination with the position information can more accurately reflect the possibility that the video unit is the core part. Therefore, determining the core video from the video to be processed according to the content core degree can improve the accuracy of extracting the core part from the video.

[0313] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.13As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video processing data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video processing method is implemented.

[0314] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.14 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a video processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0315] Those skilled in the art will understand that Fig.13 , Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0316] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0317] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0318] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0319] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0320] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0321] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A video processing method, characterized in that: The method comprises: Acquire a video to be processed, where the video to be processed is used to describe a target object; After dividing the video to be processed into a plurality of video segments, obtaining position information of each video segment in the video to be processed, each video segment is used to describe different contents of the target object; Dividing each of the video clips into a plurality of video units; Inputting the video features corresponding to each of the video units and the position information corresponding to the video clip where each of the video units is located into the trained content coreness evaluation network to obtain the content coreness of each of the video units; According to the core degree of the content, a core video describing the target object is determined from the videos to be processed.

2. The method according to claim 1, characterized in that The step of dividing the video to be processed into a plurality of video segments comprises: Obtaining a feature vector of each video frame in the video to be processed; Performing clustering processing on each of the video frames according to the feature vector to obtain a plurality of clusters; According to the video frames classified into the same cluster, a plurality of the video segments are obtained.

3. The method according to claim 2, characterized in that The clustering process is performed on each of the video frames according to the feature vector to obtain a plurality of clusters, including: Selecting a target number of video frames from each video frame of the video to be processed as cluster centers; According to the similarity between the feature vector of each video frame and the feature vector of the cluster center, clustering processing is performed on each video frame to obtain the target number of clusters.

4. The method according to claim 3, characterized in that The method further comprises: After clustering is performed according to the target number and the video segments are obtained, traversing each video frame of the video to be processed; Determine the intra-cluster similarity between the traversed current video frame and other video frames in the same video segment; Determine out-of-cluster similarity between the traversed current video frame and other video clips; Determining a clustering coefficient corresponding to the current video frame according to a difference between the intra-cluster similarity and the extra-cluster similarity; Determine the clustering score corresponding to clustering processing according to the target number according to the clustering coefficient corresponding to each video frame in the video to be processed obtained after the traversal is completed; According to the cluster clusters corresponding to the target number when the clustering score is the highest, a plurality of the video segments corresponding to the video to be processed are determined.

5. The method according to claim 1, characterized in that The step of dividing each of the video clips into a plurality of video units comprises: Obtaining a feature vector of each video frame in the video clip; Clustering each video frame in the video clip according to the feature vector to obtain a plurality of clusters; A plurality of the video units are obtained according to the video frames classified into the same cluster.

6. The method according to claim 5, characterized in that The clustering process is performed on each video frame in the video clip according to the feature vector to obtain a plurality of cluster clusters, including: Selecting a target number of video frames from each video frame of the video clip as cluster centers; According to the similarity between the feature vector of each video frame and the feature vector of the cluster center, clustering processing is performed on each video frame in the video clip to obtain the target number of cluster clusters.

7. The method according to claim 6, characterized in that The method further comprises: After performing clustering processing according to the target number and obtaining the video units, traversing each video frame of the video clip; Determine the intra-cluster similarity between the traversed current video frame and other video frames in the same video unit; Determine out-of-cluster similarity between the traversed current video frame and other video units; Determining a clustering coefficient corresponding to the current video frame according to a difference between the intra-cluster similarity and the extra-cluster similarity; Determining a clustering score corresponding to clustering processing according to the target number according to the clustering coefficient corresponding to each video frame in the video segment obtained after the traversal is completed; According to the cluster clusters corresponding to the target number when the clustering score is the highest, a plurality of the video units corresponding to the video clip are determined.

8. The method according to claim 1, characterized in that The method further comprises: Inputting the video to be processed into a trained core video generation model; The feature vector generation network in the core video generation model is used to extract feature vectors from each video frame in the video to be processed, and feature vectors corresponding to each video frame are obtained.

9. The method according to claim 1, characterized in that: The obtaining of the position information of each of the video clips in the video to be processed includes: Obtaining the sequence of each video frame in the video clip in the video to be processed; The position information of the video segment where the video frame is located in the video to be processed is determined according to the sequence.

10. The method according to claim 1, characterized in that The step of inputting the video features corresponding to each of the video units and the position information corresponding to the video clip where each of the video units is located into the trained content coreness evaluation network to obtain the content coreness of each of the video units includes: Inputting the video features corresponding to each of the video units and the position information corresponding to the video clip where each of the video units is located into the content coreness evaluation network in the trained core video generation model; Determining the image coreness score of each video unit through the content coreness evaluation network according to the image features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located; Determine the audio core score of each video unit according to the audio features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located through the content coreness evaluation network; The image core score and the audio core score of each of the video units are integrated to obtain the content core degree of each of the video units.

11. The method according to claim 10, characterized in that Determining the image core score of each video unit through the content coreness evaluation network according to the image features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located includes: Obtaining a feature vector corresponding to the position information of each of the video units; The image features and the feature vectors corresponding to each of the video units are sequentially input into the image evaluation network in the content coreness evaluation network to obtain the image coreness score of each of the video units.

12. The method according to claim 10, characterized in that Determining the audio core score of each video unit through the content coreness evaluation network according to the audio features corresponding to each video unit and the position information corresponding to the video segment where each video unit is located includes: Obtaining a feature vector corresponding to the position information of each of the video units; The audio features and the feature vectors corresponding to each of the video units are sequentially input into the audio evaluation network in the content coreness evaluation network to obtain the audio coreness score of each of the video units.

13. The method according to claim 10, characterized in that The fusing the image core score and the audio core score of each of the video units to obtain the content core degree of each of the video units includes: Obtaining weights corresponding to the image core score and the audio core score respectively; The image core score and the audio core score of each of the video units are weighted and summed according to the weights to obtain the content core degree of each of the video units.

14. The method according to claim 1, characterized in that The step of determining a core video describing the target object from the video to be processed according to the core content degree includes: Determining a core degree distribution curve corresponding to the video to be processed according to each of the video units and the corresponding content core degrees; According to the core degree distribution curve, selecting video units whose content core degrees are higher than a threshold value from the divided video units; A core video describing the target object is generated according to the filtered video units.

15. The method according to any one of claims 1 to 14, characterized in that: The method further comprises: After dividing each of the video clips into a plurality of video units, obtaining position information of each of the video units in the video to be processed; Based on the video features corresponding to each of the video units and the position information of each of the video units in the video to be processed, the content coreness of each of the video units is obtained.

16. A video processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a video to be processed, where the video to be processed is used for describing a target object; A division module, used for dividing the video to be processed into a plurality of video segments, obtaining position information of each video segment in the video to be processed, each video segment being used to describe different contents of the target object; and dividing each video segment into a plurality of video units; The acquisition module is also used to input the video features corresponding to each of the video units and the position information corresponding to the video clip where each of the video units is located into the trained content coreness evaluation network to obtain the content coreness of each of the video units; A determination module is used to determine a core video describing the target object from the videos to be processed according to the core degree of the content.

17. The device according to claim 16, characterized in that The division module is also used to obtain the feature vector of each video frame in the video to be processed; cluster each video frame according to the feature vector to obtain multiple clusters; and obtain multiple video clips according to the video frames divided into the same cluster.

18. The device according to claim 17, characterized in that The division module is also used to select a target number of video frames from each video frame of the video to be processed as cluster centers; cluster each video frame according to the similarity between the feature vector of each video frame and the feature vector of the cluster center to obtain the target number of cluster clusters.

19. The device according to claim 18, characterized in that The partitioning module is also used to traverse each video frame of the video to be processed after clustering processing is performed according to the target number and the video fragment is obtained; determine the intra-cluster similarity between the traversed current video frame and other video frames in the same video fragment; determine the extra-cluster similarity between the traversed current video frame and other video fragments; determine the clustering coefficient corresponding to the current video frame according to the difference between the intra-cluster similarity and the extra-cluster similarity; determine the clustering score corresponding to the clustering processing performed according to the target number according to the clustering coefficient corresponding to each video frame in the video to be processed obtained after the traversal is completed; According to the cluster clusters corresponding to the target number when the clustering score is the highest, a plurality of the video segments corresponding to the video to be processed are determined.

20. The device according to claim 16, characterized in that The division module is also used to obtain a feature vector of each video frame in the video clip; cluster each video frame in the video clip according to the feature vector to obtain multiple clusters; and obtain multiple video units according to the video frames divided into the same cluster.

21. The device according to claim 20, characterized in that The division module is also used to select a target number of video frames from each video frame in the video clip as clustering centers; clustering is performed on each video frame in the video clip according to the similarity between the feature vector of each video frame and the feature vector of the clustering center to obtain the target number of cluster clusters.

22. The device according to claim 21, characterized in that The partitioning module is also used to traverse each video frame of the video segment after clustering according to the target number and obtaining the video unit; determine the intra-cluster similarity between the traversed current video frame and other video frames in the same video unit; determine the extra-cluster similarity between the traversed current video frame and other video units; determine the clustering coefficient corresponding to the current video frame according to the difference between the intra-cluster similarity and the extra-cluster similarity; determine the clustering score corresponding to the clustering according to the target number according to the clustering coefficient corresponding to each video frame in the video segment obtained after the traversal is completed; According to the cluster clusters corresponding to the target number when the clustering score is the highest, a plurality of the video units corresponding to the video clip are determined.

23. The device according to claim 16, characterized in that The device also includes: The model processing module is used to input the video to be processed into a trained core video generation model; through the feature vector generation network in the core video generation model, feature vectors of each video frame in the video to be processed are extracted to obtain feature vectors corresponding to each video frame.

24. The device according to claim 16, characterized in that The division module is further used to obtain the sequence of each video frame in the video segment in the video to be processed; and determine the position information of the video segment where the video frame is located in the video to be processed according to the sequence.

25. The device according to claim 16, characterized in that The acquisition module is also used to input the video features corresponding to each of the video units and the position information corresponding to the video clip where each of the video units is located into the content core degree evaluation network in the trained core video generation model; determine the image core score of each of the video units through the content core degree evaluation network according to the image features corresponding to each of the video units and the position information corresponding to the video clip where each of the video units is located; determine the audio core score of each of the video units through the content core degree evaluation network according to the audio features corresponding to each of the video units and the position information corresponding to the video clip where each of the video units is located; and obtain the content core degree of each of the video units by fusing the image core score and the audio core score of each of the video units.

26. The device according to claim 25, characterized in that The acquisition module is also used to obtain the feature vector corresponding to the position information of each video unit; the image features and the feature vector corresponding to each video unit are input into the image evaluation network in the content core degree evaluation network in turn to obtain the image core score of each video unit.

27. The device according to claim 25, characterized in that The acquisition module is also used to obtain the feature vector corresponding to the position information of each video unit; the audio features and the feature vector corresponding to each video unit are input into the audio evaluation network in the content core degree evaluation network in turn to obtain the audio core score of each video unit.

28. The device according to claim 25, characterized in that The acquisition module is also used to obtain the weights corresponding to the image core score and the audio core score respectively; perform weighted summation of the image core score and the audio core score of each video unit according to the weights to obtain the content core degree of each video unit.

29. The device according to claim 16, characterized in that The determination module is also used to determine the core degree distribution curve corresponding to the video to be processed based on each of the video units and the corresponding content core degree; based on the core degree distribution curve, filter out the video units whose content core degree is higher than a threshold from the divided video units; and generate a core video describing the target object based on the filtered video units.

30. The device according to any one of claims 16 to 29, characterized in that The division module is also used to obtain the position information of each video unit in the video to be processed after dividing each video clip into multiple video units; based on the video features corresponding to each video unit and the position information of each video unit in the video to be processed, obtain the content coreness of each video unit.

31. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of claims 1 to 15 when executing the computer program.

32. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

33. A computer program product, comprising computer instructions, wherein the computer instructions are stored in a computer-readable storage medium, a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps of the method described in any one of claims 1 to 15.