Data processing method and device

Abstract

The present application discloses a data processing method and device, wherein the method includes the following steps: obtaining a target video and a historical highlight video; the video frames in the historical highlight video are highlight video frames; obtaining a first semantic feature of each video frame in the target video and a second semantic feature of each video frame in the historical highlight video through a target network model; obtaining a comprehensive preference of each video frame in the target video based on the first semantic feature and the second semantic feature; determining a highlight video frame in the target video based on the target network model and the comprehensive preference of each video frame in the target video; and generating a highlight segment of the target video based on the highlight video frame. Using this application, the accuracy of detecting wonderful segments in videos can be improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data processing method and device. Background Art

[0002] Short videos have become an indispensable medium for people to acquire knowledge and share their daily lives. Capturing the highlights of short videos is essential, but unedited videos can take minutes or even hours to glean meaningful moments by scrolling through the entire video. Current methods for extracting highlights assume that a user's history contains only one preference, treating that history as a single preference to predict their preferences. However, given the inherent diversity of user interests, such as those involving travel, swimming, and pets, these methods are not rational, resulting in errors in extracting highlights from videos. Summary of the Invention

[0003] The embodiments of the present application provide a data processing method and device, which can improve the accuracy of detecting highlight segments in a video.

[0004] On the one hand, an embodiment of the present application provides a data processing method, which may include:

[0005] Obtain a target video and a historical highlight video; the video frame in the historical highlight video is a highlight video frame;

[0006] Obtaining, through a target network model, a first semantic feature of each video frame in the target video and a second semantic feature of each video frame in the historical highlight video;

[0007] Obtaining a comprehensive preference of each video frame in the target video according to the first semantic feature and the second semantic feature;

[0008] Determining highlight video frames in the target video by using the target network model and the comprehensive preference of each video frame in the target video;

[0009] Generate a highlight segment of the target video according to the highlight video frame.

[0010] In a feasible implementation manner, the data processing method further includes:

[0011] Obtaining an initial target network model and training sample data of the initial target network model; the training sample data is a video carrying a sample label;

[0012] Inputting the training sample data into the initial target network model, extracting semantic features of the training sample data through the initial target network model, and obtaining the highlight degree and non-highlight degree of the training sample data according to the semantic features;

[0013] A target network model is generated based on the highlight degree and the non-highlight degree of the training sample data and the sample labels carried by the training sample data.

[0014] In a feasible implementation manner, obtaining the comprehensive preference of each video frame in the target video according to the first semantic feature and the second semantic feature includes:

[0015] Obtain a target video frame in the target video, where the target video frame is any video frame in the target video;

[0016] Obtaining a vector distance between the target video frame and the first video frame based on a first semantic feature of the target video frame and a second semantic feature of the first video frame in the historical highlight video, and determining the vector distance as a target semantic similarity between the target video frame and the first video frame; the first video frame is any video frame in the historical highlight video;

[0017] Performing a weighted summation on each video frame in the historical highlight video according to the target semantic similarity to obtain a user preference for the target video frame;

[0018] The general preference of the target video frame is obtained based on the target network model, and the comprehensive preference of the target video frame is generated by using the user preference and the general preference.

[0019] In a feasible implementation manner, performing weighted summation on each video frame in the historical segment according to the target semantic similarity to obtain the user preference of the target video frame includes:

[0020] The target semantic similarity between the target video frame and the first video frame is used as the first weight information of the first video frame; the first video frame is any video frame in the historical video;

[0021] The first weight information is used to perform weighted summation on each video frame in the historical highlight video to generate a user preference for the target video frame.

[0022] In a feasible implementation, the acquiring of the general preference of the target video frame based on the target network model and the generating of the comprehensive preference of the target video frame using the user preference and the general preference include:

[0023] Obtaining a general preference of a target video frame based on the target network model, obtaining a first semantic similarity between the target video frame and the user preference, and obtaining a second semantic similarity between the target video frame and the general preference;

[0024] using the first semantic similarity and the second semantic similarity as second weight information of the user preference and the general preference respectively;

[0025] The second weight information is used to perform a weighted summation on the user preference and the general preference to generate a comprehensive preference of the target video frame.

[0026] In a feasible implementation manner, determining the highlight video frame in the target video by using the target network model and the comprehensive preference of each video frame in the target video includes:

[0027] Performing full-connection processing on the comprehensive preference through a fully-connected layer in a target network model to generate a highlight value for each video frame in the target video;

[0028] If the highlight level value of the target video frame is greater than the highlight threshold, the target video frame is determined as a highlight video frame; the target video frame is any video frame in the target video;

[0029] If the highlight level value of the target video frame is less than or greater than a highlight threshold, the target video frame is determined as a non-highlight video frame.

[0030] In a feasible implementation manner, generating the highlight segment of the target video according to the highlight video frame includes:

[0031] Acquire all highlight video frames in the target video, and splice the highlight video frames in chronological order to generate continuous video frames;

[0032] The continuous video frames are determined as highlight segments of the target video.

[0033] On the one hand, an embodiment of the present application provides a data processing device, which may include:

[0034] A video acquisition unit, configured to acquire a target video and a historical highlight video; wherein the video frames in the historical highlight video are highlight video frames;

[0035] a feature extraction unit, configured to obtain, through a target network model, a first semantic feature of each video frame in the target video and a second semantic feature of each video frame in the historical highlight video;

[0036] a comprehensive preference obtaining unit, configured to obtain a comprehensive preference of each video frame in the target video according to the first semantic feature and the second semantic feature;

[0037] a highlight video determining unit, configured to determine a highlight video frame in a target video by using the target network model and a comprehensive preference of each video frame in the target video;

[0038] The highlight segment generating unit is configured to generate a highlight segment of the target video according to the highlight video frame.

[0039] In a feasible implementation manner, the data processing device further includes:

[0040] A model training unit, configured to obtain an initial target network model and training sample data of the initial target network model; the training sample data is a video carrying a sample label;

[0041] Inputting the training sample data into the initial target network model, extracting semantic features of the training sample data through the initial target network model, and obtaining the highlight degree and non-highlight degree of the training sample data according to the semantic features;

[0042] A target network model is generated based on the highlight degree and the non-highlight degree of the training sample data and the sample labels carried by the training sample data.

[0043] In a feasible implementation manner, the comprehensive preference acquisition unit includes:

[0044] A similarity acquisition subunit is configured to acquire a target video frame in the target video, where the target video frame is any one video frame in the target video;

[0045] Obtaining a vector distance between the target video frame and the first video frame based on a first semantic feature of the target video frame and a second semantic feature of the first video frame in the historical highlight video, and determining the vector distance as a target semantic similarity between the target video frame and the first video frame; the first video frame is any video frame in the historical highlight video;

[0046] A user preference acquisition subunit, configured to perform weighted summation on each video frame in the historical highlight video according to the target semantic similarity, to acquire the user preference of the target video frame;

[0047] The comprehensive preference acquisition subunit is used to acquire the general preference of the target video frame based on the target network model, and generate the comprehensive preference of the target video frame using the user preference and the general preference.

[0048] In a feasible implementation manner, the user preference acquisition subunit is specifically configured to:

[0049] The target semantic similarity between the target video frame and the first video frame is used as the first weight information of the first video frame; the first video frame is any video frame in the historical video;

[0050] The first weight information is used to perform weighted summation on each video frame in the historical highlight video to generate a user preference for the target video frame.

[0051] In a feasible implementation manner, the comprehensive preference acquisition subunit is specifically configured to:

[0052] Obtaining a general preference of a target video frame based on the target network model, obtaining a first semantic similarity between the target video frame and the user preference, and obtaining a second semantic similarity between the target video frame and the general preference;

[0053] using the first semantic similarity and the second semantic similarity as second weight information of the user preference and the general preference respectively;

[0054] The second weight information is used to perform a weighted summation on the user preference and the general preference to generate a comprehensive preference of the target video frame.

[0055] In a feasible implementation manner, the highlight video determination unit is specifically configured to:

[0056] Performing full-connection processing on the comprehensive preference through a fully-connected layer in a target network model to generate a highlight value for each video frame in the target video;

[0057] If the highlight level value of the target video frame is greater than the highlight threshold, the target video frame is determined as a highlight video frame; the target video frame is any video frame in the target video;

[0058] If the highlight level value of the target video frame is less than or greater than a highlight threshold, the target video frame is determined as a non-highlight video frame.

[0059] In a feasible implementation manner, the highlight segment generating unit is specifically configured to:

[0060] Acquire all highlight video frames in the target video, and splice the highlight video frames in chronological order to generate continuous video frames;

[0061] The continuous video frames are determined as highlight segments of the target video.

[0062] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded by a processor and executing the above-mentioned method steps.

[0063] On the one hand, an embodiment of the present application provides a computer device, including: a processor, a memory, and a network interface; the processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program code, and the processor is used to call the program code to execute the above-mentioned method steps.

[0064] In one aspect, an embodiment of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described method steps.

[0065] In an embodiment of the present application, a target video and a historical highlight video are obtained, wherein the video frames in the historical highlight video are highlight video frames, and further, through a target network model, a first semantic feature of each video frame in the target video and a second semantic feature of each video frame in the historical highlight video are obtained, and based on the first semantic feature and the second semantic feature, a comprehensive preference of each video frame in the target video is obtained, and further, through the target network model and the comprehensive preference of each video frame in the target video, the highlight video frame in the target video is determined, and finally, based on the highlight video frame, a highlight segment of the target video is generated. The above method can avoid using the user's history as overall information to predict the user's preferences, thereby improving the accuracy of detecting wonderful segments in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0067] Figure 1 This is a system architecture diagram of data processing provided by an embodiment of the present application;

[0068] Figure 2 This is a flow chart of a data processing method provided in an embodiment of the present application;

[0069] Figure 3 This is an example diagram of extracting semantic features provided by an embodiment of the present application;

[0070] Figure 4 This is a flow chart of a data processing method provided in an embodiment of the present application;

[0071] Figure 5a This is a schematic diagram illustrating an example of a data processing method provided in an embodiment of the present application;

[0072] Figure 5b This is an example schematic diagram of extracting highlight video frames provided by an embodiment of the present application;

[0073] Figure 6 This is a structural diagram of a data processing device provided in an embodiment of the present application;

[0074] Figure 7 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0076] like Figure 1 As shown, the network architecture diagram may include a business server 100 and a user terminal cluster, and the user terminal cluster may include user terminal 10a, user terminal 10b, ..., user terminal 10c, wherein there may be a communication connection between the user terminal clusters, for example, there is a communication connection between user terminal 10a and user terminal 10b, and there is a communication connection between user terminal 10b and user terminal 10c, and any user terminal in the user terminal cluster may have a communication connection with the business server 100, for example, there is a communication connection between user terminal 10a and business server 100, and there is a communication connection between user terminal 10b and business server 100.

[0077] Among them, the above-mentioned user terminal cluster (also including the above-mentioned user terminal 10a, user terminal 10b and user terminal 10c) can all be integrated with the target application. Optionally, the target application may include an application with the function of obtaining data information such as images and videos. The database 10d stores the target network model and the training data of the target network model, and the target network model is used to extract the semantic features of the video. Specifically, the user terminal obtains the target video and the historical highlight video, and the video frames in the historical highlight video are highlight video frames. Further, the user terminal obtains the first semantic feature of each video frame in the target video and the second semantic feature of each video frame in the historical highlight video through the target network model, and obtains the comprehensive preference of each video frame in the target video based on the first semantic feature and the second semantic feature. Further, the user terminal determines the highlight video frame in the target video through the target network model and the comprehensive preference of each video frame in the target video, and finally generates the highlight segment of the target video based on the highlight video frame.

[0078] Optionally, the user terminal may be Figure 1 Any user terminal selected from the user terminal cluster of the corresponding embodiment, for example, the user terminal can be the above-mentioned user terminal 10b, and the user terminal can download the target network model to the terminal locally and generate a highlight segment of the target video on the user terminal side.

[0079] It is understandable that the method provided in the embodiment of the present application can be executed by a computer device, and the computer device includes but is not limited to a terminal or a server. The server 100 in the embodiment of the present application can be a computer device, and the user terminal in the user terminal cluster can also be a computer device, which is not limited here. The above-mentioned business server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The above-mentioned terminals may include: smart phones, tablet computers, laptops, desktop computers, smart TVs, smart speakers, desktop computers, smart watches and other smart terminals with image recognition functions, but are not limited to these. Among them, the user terminal and the business server can be directly or indirectly connected via wired or wireless communication, which is not limited here.

[0080] For further understanding, please see Figure 2 , Figure 2 This is a flow chart of the data processing method provided by the embodiment of the present application. The method can be performed by a user terminal (for example, the above Figure 1The user terminal shown in FIG) can also be executed by the user terminal and the service server (as shown above Figure 1 The data processing method may include at least the following steps S101 to S105:

[0081] S101, obtaining the target video and historical highlight videos;

[0082] Specifically, the user terminal obtains the target video and the historical highlight video. It can be understood that the target video is the video to be detected, which can be a video file in various formats, including but not limited to mp4, dat, mkv, flv, vob, etc. The video frames in the historical highlight video are highlight video frames, that is, the historical highlight video can be a highlight video clip extracted by the user from other videos.

[0083] S102, obtaining, through a target network model, a first semantic feature of each video frame in the target video and a second semantic feature of each video frame in the historical highlight video;

[0084] Specifically, the user terminal obtains the first semantic feature of each video frame in the target video and the second semantic feature of each video frame in the historical highlight video through the target network model. It can be understood that the target network model is used to extract semantic features of the video and determine the highlight video frame in the target video based on the comprehensive preference of the video frame. The target network model includes a feature extraction module and a fully connected layer module. The feature extraction module is used to extract semantic features of the video. For example, the feature extraction module can specifically use a U-net neural network to extract short-term features and long-term features of the video frame, and use the short-term features and long-term features as the semantic features of each video frame, that is, extract the short-term features and long-term features of each video frame in the target video as the first semantic feature of each video frame, and extract the short-term features and long-term features of each video frame in the historical highlight video as the second semantic feature of each video frame. Please refer to Figure 3 , Figure 3 This is an example diagram of extracting semantic features provided by the embodiment of the present application, such as Figure 3 As shown in the figure, the video (target video or historical highlight video) is input into the U-net neural network, and the semantic features corresponding to each frame in the video are generated by the pooling layer, convolution layer and normalization layer in the U-net neural network.

[0085] S103, obtaining a comprehensive preference of each video frame in the target video according to the first semantic feature and the second semantic feature;

[0086] Specifically, the user terminal obtains a target video frame in the target video, where the target video frame is any video frame in the target video. The user terminal obtains a vector distance between the target video frame and the first video frame through a first semantic feature of the target video frame and a second semantic feature of the first video frame in the historical highlight video. The vector distance is determined as a target semantic similarity between the target video frame and the first video frame, where the first video frame is any video frame in the historical highlight video. Further, a weighted sum is performed on each video frame in the historical highlight video according to the target semantic similarity to obtain a user preference for the target video frame. The general preference for the target video frame is obtained based on the target network model, and the user preference and the general preference are used to generate a comprehensive preference for the target video frame.

[0087] S104, determining a highlight video frame in the target video based on the target network model and the comprehensive preference of each video frame in the target video;

[0088] Specifically, the user terminal determines the highlight video frame in the target video through the target network model and the comprehensive preference of each video frame in the target video. It can be understood that the user terminal performs full-connection processing on the comprehensive preference through the fully-connected layer in the target network model to generate a highlight level value for each video frame in the target video, and compares the highlight level value with a highlight threshold. The highlight threshold is pre-set by the user. If the highlight level value of the target video frame is greater than the highlight threshold, the target video frame is determined to be a highlight video frame. If the highlight level value of the target video frame is less than or greater than the highlight threshold, the target video frame is determined to be a non-highlight video frame.

[0089] S105 : Generate a highlight segment of the target video according to the highlight video frame.

[0090] Specifically, the user terminal generates a highlight segment of the target video based on the highlight video frames. It is understood that the user terminal obtains all highlight video frames in the target video, splices the highlight video frames in chronological order, generates continuous video frames, and determines the continuous video frames as the highlight segment of the target video. It should be noted that to ensure temporal continuity of the target video, several non-highlight video frames before and after the highlight video frame may be added to the highlight segment.

[0091] In an embodiment of the present application, a target video and a historical highlight video are obtained, wherein the video frames in the historical highlight video are highlight video frames, and further, through a target network model, a first semantic feature of each video frame in the target video and a second semantic feature of each video frame in the historical highlight video are obtained, and based on the first semantic feature and the second semantic feature, a comprehensive preference of each video frame in the target video is obtained, and further, through the target network model and the comprehensive preference of each video frame in the target video, the highlight video frame in the target video is determined, and finally, based on the highlight video frame, a highlight segment of the target video is generated. The above method can avoid using the user's history as overall information to predict the user's preferences, thereby improving the accuracy of detecting wonderful segments in the video.

[0092] See Figure 4 , Figure 4 This is a flow chart of the data processing method provided by the embodiment of the present application. The method can be performed by a user terminal (for example, the above Figure 1 The user terminal shown in FIG) can also be executed by the user terminal and the service server (as shown above Figure 1 The data processing method may include at least the following steps S201 to S208:

[0093] S201, obtaining an initial target network model and training sample data of the initial target network model; the training sample data is a video carrying a sample label;

[0094] Specifically, the user terminal obtains an initial target network model and training sample data of the initial target network model, wherein the training sample data includes positive sample data and negative sample data, and further inputs the training sample data into the initial target network model, extracts semantic features of the training sample data through the initial target network model, obtains the highlight degree and non-highlight degree of the training sample data according to the semantic features, and generates a target network model based on the highlight degree and non-highlight degree of the training sample data and the sample labels carried by the training sample data. Specifically, when the loss function of the initial target network model is less than the loss function threshold, the initial target network model after adjusting the parameters is used as the target network model. Specifically, the loss function It can be expressed as follows:

[0095]

[0096]

[0097] Among them, Ω is the set of positive sample data, is the set of negative sample data. i and b i They are the highlight level and non-highlight level of the video frame respectively.

[0098] S202, obtaining a target video and a historical highlight video; the video frames in the historical highlight video are highlight video frames;

[0099] Wherein, step S202 of the embodiment of the present invention refers to Figure 2 The detailed description of step S101 of the illustrated embodiment is omitted here.

[0100] S203, obtaining, through a target network model, a first semantic feature of each video frame in the target video and a second semantic feature of each video frame in the historical highlight video;

[0101] Wherein, step S203 of the embodiment of the present invention is referred to Figure 2 The detailed description of step S102 of the illustrated embodiment is omitted here.

[0102] S204, obtaining a target video frame in the target video, where the target video frame is any video frame in the target video;

[0103] Specifically, the user terminal obtains any one video frame in the target video as the target video frame. Further, through the first semantic feature of the target video frame and the second semantic feature of the first video frame in the historical highlight video, the vector distance between the target video frame and the first video frame is obtained, and the vector distance is determined as the target semantic similarity between the target video frame and the first video frame. The first video frame is any one video frame in the historical highlight video. The vector distance can be represented by cosine similarity. Specifically, the cosine similarity between the first semantic feature and the second semantic feature can be expressed by the following formula:

[0104]

[0105] Among them, A is the first semantic feature, B is the second semantic feature, A i is the value of the i-th dimension of the first semantic feature, B i is the value of the i-th dimension of the second semantic feature.

[0106] S205, performing weighted summation on each video frame in the historical highlight video according to the target semantic similarity to obtain a user preference for the target video frame;

[0107] Specifically, the user terminal uses the target semantic similarity between the target video frame and the first video frame as the first weight information of the first video frame, and uses the first weight information to perform a weighted summation on each video frame in the historical highlight video to generate the user preference of the target video frame. Specifically, the user preference of the target video frame can be expressed by the following formula:

[0108]

[0109] in, is the user preference of the i-th frame in the target video, h j is the second semantic feature of the j-th frame video in the historical highlight video, is the weight information of the j-th frame video in the historical highlight video.

[0110] S206 , obtaining a general preference of a target video frame based on the target network model, and generating a comprehensive preference of the target video frame using the user preference and the general preference.

[0111] Specifically, the user terminal obtains the general preference of the target video frame based on the target network model. The general preference is a fixed value in the target network model. The first semantic similarity between the target video frame and the user preference is further obtained. The first semantic similarity can be represented by the vector distance between the target video frame and the user preference. Specifically, the vector distance can be represented by cosine similarity. The second semantic similarity between the target video frame and the general preference is further obtained. The second semantic similarity is obtained in the same way as the first semantic similarity. Finally, the first semantic similarity and the second semantic similarity are respectively used as the second weight information of the user preference and the general preference. The second weight information is used to perform weighted summation on the user preference and the general preference to generate the comprehensive preference of the target video frame. Specifically, the comprehensive preference of the target video frame can be expressed by the formula:

[0112]

[0113] in, is the comprehensive preference of the target video frame, is the user preference of the target video frame, is the first semantic similarity, is the second semantic similarity.

[0114] It should be noted that, when there is no historical highlight video, only the general preference can be used as the comprehensive preference of the target video frame.

[0115] S207, determining a highlight video frame in the target video based on the target network model and the comprehensive preference of each video frame in the target video;

[0116] Wherein, step S207 of the embodiment of the present invention refers to Figure 2 The detailed description of step S104 of the illustrated embodiment is omitted here.

[0117] S208 : Generate a highlight segment of the target video according to the highlight video frame.

[0118] Wherein, step S208 of the embodiment of the present invention is shown in FIG. Figure 2 The detailed description of step S105 of the illustrated embodiment is omitted here.

[0119] See Figure 5a , Figure 5a This is an example diagram of a data processing method provided in an embodiment of the present application. Figure 5a As shown, the user terminal obtains a target video and historical highlight videos. Using the target network model, it extracts the first semantic feature of each video frame in the target video and the second semantic feature of each video frame in the historical highlight videos. The target semantic similarity between the target video frame and the video frames in the historical highlight videos is obtained using the first semantic feature of the target video frame and the second semantic feature of the video frames in the historical highlight videos. Furthermore, using an attention mechanism, a weighted sum is performed on each video frame in the historical highlight videos based on the target semantic similarity to generate a user preference for the target video frame. Based on the target network model, the general preference for the target video frame is obtained. Using the attention mechanism, the user preference and the general preference are weighted to generate a comprehensive preference for the target video frame. The user terminal fully connects the comprehensive preference using the fully connected layer in the target network model to generate a highlight value for each video frame in the target video. This highlight value is compared with a highlight threshold. If the highlight value of the target video frame is greater than the highlight threshold, the target video frame is determined to be a highlight frame. If the highlight value of the target video frame is less than or equal to the highlight threshold, the target video frame is determined to be a non-highlight frame. Furthermore, the user terminal obtains all highlight video frames in the target video, splices the highlight video frames in chronological order to generate continuous video frames, and determines the continuous video frames as the highlight segments of the target video.

[0120] The following method is used to verify the extraction of highlight video frames in the target video. Figure 5b , Figure 5b This is an example schematic diagram of extracting highlight video frames provided by an embodiment of the present application, such as Figure 5aAs shown in the figure, video frame a is a highlight frame manually marked by the user in the target video, and video frames bd are video frames from the historical highlight video. That is, video frames bd and d are both highlight frames. The historical highlight video includes highlight frames from multiple scenes. In other words, extracting highlight frames based on the comprehensive preferences generated by the attention mechanism takes into account the diversity of user interests. Using the method in this solution, the highlight frame extracted from the target video is video frame eh. A comparison shows that the extracted highlight frame eh using this solution has almost no deviation in accuracy from the manually marked video frame a. Therefore, this method can improve the accuracy of detecting highlight clips in videos.

[0121] In an embodiment of the present application, a target video and a historical highlight video are obtained, wherein the video frames in the historical highlight video are highlight video frames, and further, through a target network model, a first semantic feature of each video frame in the target video and a second semantic feature of each video frame in the historical highlight video are obtained, and based on the first semantic feature and the second semantic feature, a comprehensive preference of each video frame in the target video is obtained, and further, through the target network model and the comprehensive preference of each video frame in the target video, the highlight video frame in the target video is determined, and finally, based on the highlight video frame, a highlight segment of the target video is generated. The above method can avoid using the user's history as overall information to predict the user's preferences, thereby improving the accuracy of detecting wonderful segments in the video.

[0122] See Figure 6 , Figure 6 This is a structural diagram of a data processing device provided in an embodiment of the present application. The data processing device may be a computer program (including program code) running on a computer device, for example, the data processing device is an application software; the device may be used to execute the corresponding steps of the method provided in an embodiment of the present application. Figure 6 As shown, the data processing device 1 of the embodiment of the present application may include: a video acquisition unit 11, a feature extraction unit 12, a comprehensive preference acquisition unit 13, a highlight video determination unit 14, and a highlight segment generation unit 15.

[0123] The video acquisition unit 11 is used to acquire the target video and the historical highlight video; the video frames in the historical highlight video are highlight video frames;

[0124] A feature extraction unit 12 is configured to obtain, through a target network model, a first semantic feature of each video frame in the target video and a second semantic feature of each video frame in the historical highlight video;

[0125] a comprehensive preference obtaining unit 13, configured to obtain a comprehensive preference of each video frame in the target video according to the first semantic feature and the second semantic feature;

[0126] a highlight video determining unit 14, configured to determine a highlight video frame in a target video by using the target network model and a comprehensive preference of each video frame in the target video;

[0127] The highlight segment generating unit 15 is configured to generate a highlight segment of the target video according to the highlight video frame.

[0128] See Figure 6 , the data processing device 1 of the embodiment of the present application may also include: a model training unit 16.

[0129] The model training unit 16 is used to obtain an initial target network model and training sample data of the initial target network model; the training sample data is a video with a sample label;

[0130] Inputting the training sample data into the initial target network model, extracting semantic features of the training sample data through the initial target network model, and obtaining the highlight degree and non-highlight degree of the training sample data according to the semantic features;

[0131] A target network model is generated based on the highlight degree and the non-highlight degree of the training sample data and the sample labels carried by the training sample data.

[0132] See Figure 6 The comprehensive preference acquisition unit 13 of the embodiment of the present application may include: a similarity acquisition subunit 131, a user preference acquisition subunit 132, and a comprehensive preference acquisition subunit 133.

[0133] A similarity acquisition subunit 131 is configured to acquire a target video frame in the target video, where the target video frame is any video frame in the target video;

[0134] Obtaining a vector distance between the target video frame and the first video frame based on a first semantic feature of the target video frame and a second semantic feature of the first video frame in the historical highlight video, and determining the vector distance as a target semantic similarity between the target video frame and the first video frame; the first video frame is any video frame in the historical highlight video;

[0135] A user preference acquisition subunit 132 is configured to perform a weighted summation on each video frame in the historical highlight video according to the target semantic similarity to acquire a user preference for the target video frame;

[0136] The comprehensive preference acquisition subunit 133 is configured to acquire the general preference of the target video frame based on the target network model, and generate the comprehensive preference of the target video frame using the user preference and the general preference.

[0137] In a feasible implementation manner, the user preference acquisition subunit 132 is specifically configured to:

[0138] The target semantic similarity between the target video frame and the first video frame is used as the first weight information of the first video frame; the first video frame is any video frame in the historical video;

[0139] The first weight information is used to perform weighted summation on each video frame in the historical highlight video to generate a user preference for the target video frame.

[0140] In a feasible implementation manner, the comprehensive preference acquisition subunit 133 is specifically configured to:

[0141] Obtaining a general preference of a target video frame based on the target network model, obtaining a first semantic similarity between the target video frame and the user preference, and obtaining a second semantic similarity between the target video frame and the general preference;

[0142] using the first semantic similarity and the second semantic similarity as second weight information of the user preference and the general preference respectively;

[0143] The second weight information is used to perform a weighted summation on the user preference and the general preference to generate a comprehensive preference of the target video frame.

[0144] In a feasible implementation manner, the highlight video determination unit 14 is specifically configured to:

[0145] Performing full-connection processing on the comprehensive preference through a fully-connected layer in a target network model to generate a highlight value for each video frame in the target video;

[0146] If the highlight level value of the target video frame is greater than the highlight threshold, the target video frame is determined as a highlight video frame; the target video frame is any video frame in the target video;

[0147] If the highlight level value of the target video frame is less than or greater than a highlight threshold, the target video frame is determined as a non-highlight video frame.

[0148] In a feasible implementation manner, the highlight segment generating unit 15 is specifically configured to:

[0149] Acquire all highlight video frames in the target video, and splice the highlight video frames in chronological order to generate continuous video frames;

[0150] The continuous video frames are determined as highlight segments of the target video.

[0151] In an embodiment of the present application, a target video and a historical highlight video are obtained, wherein the video frames in the historical highlight video are highlight video frames, and further, through a target network model, a first semantic feature of each video frame in the target video and a second semantic feature of each video frame in the historical highlight video are obtained, and based on the first semantic feature and the second semantic feature, a comprehensive preference of each video frame in the target video is obtained, and further, through the target network model and the comprehensive preference of each video frame in the target video, the highlight video frame in the target video is determined, and finally, based on the highlight video frame, a highlight segment of the target video is generated. The above method can avoid using the user's history as overall information to predict the user's preferences, thereby improving the accuracy of detecting wonderful segments in the video.

[0152] See Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 6 As shown, the computer device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Figure 7 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a data processing application.

[0153] exist Figure 7 In the computer device 1000 shown, the network interface 1004 can provide network communication functions, the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the data processing application stored in the memory 1005 to implement the above Figure 2-5b The description of the data processing method in any corresponding embodiment will not be repeated here.

[0154] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 2-5b The description of the data processing method in any corresponding embodiment can also be performed as described above. Figure 6 The description of the data processing device in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0155] In addition, it should be noted that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the data processing device mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 2-5b The description of the data processing method in any corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed to be executed on one computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed in multiple locations and interconnected by a communication network. Multiple computing devices distributed in multiple locations and interconnected by a communication network can constitute a blockchain system.

[0156] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The above-mentioned computer-readable storage medium can be a data processing device provided in any of the above-mentioned embodiments or an internal storage unit of the above-mentioned device, such as a hard disk or memory of an electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. The above-mentioned computer-readable storage medium can also include a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory, etc. Furthermore, the computer-readable storage medium can also include both the internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and quantities required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or is about to be output.

[0157] The terms "first," "second," and the like in the claims, specification, and drawings of the present invention are used to distinguish between different objects, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or device. Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The presence of such a phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments. The term "and / or," as used in the specification and appended claims of the present invention, refers to any and all possible combinations of one or more of the associated listed items, including, but not limited to, those combinations.

[0158] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the above description generally describes the components and steps of each example according to their functions. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the present invention.

[0159] The functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0160] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: include: Obtain a target video and a historical highlight video; the video frame in the historical highlight video is a highlight video frame; Obtaining, through a target network model, a first semantic feature of each video frame in the target video and a second semantic feature of each video frame in the historical highlight video; Obtaining a comprehensive preference of each video frame in the target video according to the first semantic feature and the second semantic feature; The comprehensive preference of the target video frame in the target video is generated by the general preference and user preference corresponding to the target video frame, and the user preference corresponding to the target video frame is obtained by weighted summing the second semantic feature of each highlight video frame in the historical highlight video and the weight information of each highlight video frame in the historical highlight video, where one weight information refers to the semantic similarity between a highlight video frame in the historical highlight video and the target video frame; Determining highlight video frames in the target video by using the target network model and the comprehensive preference of each video frame in the target video; Generate a highlight segment of the target video according to the highlight video frame in the target video.

2. The method according to claim 1, characterized in that Also includes: Acquire an initial target network model and training sample data of the initial target network model; The training sample data is a video with sample labels; Inputting the training sample data into the initial target network model, extracting semantic features of the training sample data through the initial target network model, and obtaining the highlight degree and non-highlight degree of the training sample data according to the semantic features; A target network model is generated based on the highlight degree and the non-highlight degree of the training sample data and the sample labels carried by the training sample data.

3. The method according to claim 1, characterized in that The obtaining, based on the first semantic feature and the second semantic feature, a comprehensive preference of each video frame in the target video includes: Obtain a target video frame in the target video, where the target video frame is any video frame in the target video; Obtaining a vector distance between the target video frame and the first video frame using a first semantic feature of the target video frame and a second semantic feature of the first video frame in the historical highlight video, and determining the vector distance as a target semantic similarity between the target video frame and the first video frame; the first video frame is any one video frame in the historical highlight video; Performing a weighted summation on each video frame in the historical highlight video according to the target semantic similarity to obtain a user preference for the target video frame; The general preference of the target video frame is obtained based on the target network model, and the comprehensive preference of the target video frame is generated by using the user preference and the general preference.

4. The method according to claim 3, characterized in that The acquiring of the general preference of the target video frame based on the target network model, and generating the comprehensive preference of the target video frame using the user preference and the general preference, includes: Obtaining a general preference of a target video frame based on the target network model, obtaining a first semantic similarity between the target video frame and the user preference, and obtaining a second semantic similarity between the target video frame and the general preference; using the first semantic similarity and the second semantic similarity as second weight information of the user preference and the general preference respectively; The second weight information is used to perform a weighted summation on the user preference and the general preference to generate a comprehensive preference of the target video frame.

5. The method according to claim 1, wherein Determining highlight video frames in the target video by using the target network model and the comprehensive preference of each video frame in the target video includes: Performing full-connection processing on the comprehensive preference through a fully-connected layer in a target network model to generate a highlight value for each video frame in the target video; If the highlight level value of the target video frame is greater than the highlight threshold, the target video frame is determined as a highlight video frame; the target video frame is any video frame in the target video; If the highlight level value of the target video frame is less than or equal to the highlight threshold, the target video frame is determined as a non-highlight video frame.

6. The method according to claim 1, characterized in that Generating the highlight segment of the target video according to the highlight video frame includes: Acquire all highlight video frames in the target video, and splice the highlight video frames in chronological order to generate continuous video frames; The continuous video frames are determined as highlight segments of the target video.

7. A data processing device, characterized in that: include: A video acquisition unit, configured to acquire a target video and a historical highlight video; wherein the video frames in the historical highlight video are highlight video frames; a feature extraction unit, configured to obtain, through a target network model, a first semantic feature of each video frame in the target video and a second semantic feature of each video frame in the historical highlight video; a comprehensive preference obtaining unit, configured to obtain a comprehensive preference of each video frame in the target video based on the first semantic feature and the second semantic feature; The comprehensive preference of the target video frame in the target video is generated by the general preference and user preference corresponding to the target video frame, and the user preference corresponding to the target video frame is obtained by weighted summing the second semantic feature of each highlight video frame in the historical highlight video and the weight information of each highlight video frame in the historical highlight video, where one weight information refers to the semantic similarity between a highlight video frame in the historical highlight video and the target video frame; a highlight video determining unit, configured to determine a highlight video frame in a target video by using the target network model and a comprehensive preference of each video frame in the target video; The highlight segment generating unit is configured to generate a highlight segment of the target video according to a highlight video frame in the target video.

8. A computer device, characterized in that: include: processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store program code, and the processor is used to call the program code to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 6.

Images