A video processing method, device, computer device, and readable storage medium

Abstract

This application provides a video processing method, apparatus, computer device, and readable storage medium. The method relates to fields such as cloud technology, artificial intelligence, intelligent transportation, video, and search technology. The method includes: obtaining a target video segment from a target video; performing comment type detection on the target video segment based on target commentary media data and the content of the target segment, obtaining a type detection result; if the type detection result indicates that the association between the target commentary media data and the content of the target segment meets the hot comment condition, then determining the target video segment as a video segment with a hot comment type; obtaining video segments from a video segment library that meet the segment similarity hot comment condition, and determining the video segments that meet the segment similarity hot comment condition as a set of similar comment segments. Using this application can enrich the video resources associated with the target video segment and improve the accuracy of the mined video segments.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video processing method, apparatus, computer device, and readable storage medium. Background Technology

[0002] With the development of multimedia technology, video has become a primary medium for obtaining information and enjoying entertainment in daily life. Current similar video mining algorithms mainly rely on the tag information of videos (e.g., target videos) for similarity mining. This tag information primarily refers to manually labeled video tags; for example, the target video's tag could be "humor" or "sports," etc. This allows for the discovery of videos with the same tag information from a video library, which are then considered similar to the target video. However, relying solely on tag information often fails to guarantee that the discovered videos and the target video share commonalities in video content, leading to low accuracy in the discovered videos. Summary of the Invention

[0003] This application provides a video processing method, apparatus, computer device, and readable storage medium, which can improve the accuracy of the extracted video segments.

[0004] One embodiment of this application provides a video processing method, including:

[0005] Obtain target video segments from the target video, and perform comment type detection on the target video segments based on the target commentary media data associated with the target video segments and the target segment content in the target video segments to obtain the type detection results;

[0006] If the type detection result indicates that the association between the target commentary media data and the target segment content meets the hot comment criteria, then the target video segment is determined to be a video segment with the hot comment type;

[0007] Retrieve video clips from the video clip library that meet the criteria of similarity to hot comments on the target video clip, and identify the video clips that meet the criteria of similarity to hot comments on the target video clip as a set of similar comment clips associated with the target video clip.

[0008] One embodiment of this application provides a video processing apparatus, including:

[0009] The type detection module is used to obtain target video segments from the target video, perform comment type detection on the target video segments based on the target commentary media data associated with the target video segments and the target segment content in the target video segments, and obtain the type detection results;

[0010] The type determination module is used to determine that the target video segment is a video segment with the hot comment type if the type detection result indicates that the correlation between the target commentary media data and the target segment content meets the hot comment condition.

[0011] The set determination module is used to obtain video segments from the video segment library that meet the criteria of similar hot comments to the target video segment, and determine the video segments that meet the criteria of similar hot comments to the target video segment as a set of similar comment segments associated with the target video segment.

[0012] The target segment content includes video plot text;

[0013] The type detection module includes:

[0014] The clause extraction unit is used to extract commentary text clauses from target commentary media data associated with the target video segment;

[0015] The clause extraction unit is used to segment the video plot text in the target video segment based on the clause characters, generating S plot text clauses; S is a positive integer.

[0016] The text matching unit is used to determine the text similarity coefficient between each plot text clause and the commentary text clause in S plot text clauses. Based on the text similarity coefficient, the number of interactions of the commentary text clause and the text length of the commentary text clause, it generates the text hotspot matching degree between each plot text clause and the commentary text clause respectively.

[0017] The text matching unit is used to merge the text hotspot matching scores corresponding to each plot text clause to obtain the merged text hotspot matching score.

[0018] The first generation unit is used to generate a type detection result indicating that the relationship between the target commentary media data and the target segment content meets the hotspot commentary condition if the hotspot matching degree of the merged text is greater than the text matching degree threshold.

[0019] The second generation unit is used to generate a type detection result indicating that the association between the target commentary media data and the target segment content does not meet the hotspot commentary condition if the hotspot matching degree of the merged text is less than or equal to the text matching degree threshold.

[0020] The type detection module is specifically used to extract text information associated with the target video segment from the subtitle file, and to use the extracted text information as the video plot text associated with the target video segment; or,

[0021] The type detection module is also specifically used to perform speech recognition on the target video segment, and to use the recognized speech text as video plot text associated with the target video segment; or,

[0022] The type detection module is also specifically used to perform semantic recognition on the action content in the target video segment, and to use the recognized semantic text as the video plot text associated with the target video segment; or,

[0023] The type detection module is also specifically used to perform character recognition on the target video segment and use the recognized character text as video plot text associated with the target video segment.

[0024] The target segment content includes video plot text, segment image frames, and segment audio frames;

[0025] The type detection module includes:

[0026] The data input unit is used to input the target commentary media data associated with the target video segment and the video plot text in the target video segment as the segment text information of the target video segment;

[0027] The data input unit is used to input the segment image frames, segment audio frames, and segment text information of the target video segment into the first target network model;

[0028] The model detection unit is used to detect the comment type of the target video segment through the first target network model and output the segment hotspot matching degree corresponding to the target video segment.

[0029] The third generation unit is used to generate a type detection result indicating that the relationship between the target commentary media data and the target segment content meets the hot commentary condition if the segment hotspot matching degree is greater than the segment matching degree threshold.

[0030] The fourth generation unit is used to generate a type detection result indicating that the association between the target commentary media data and the target segment content does not meet the hotspot commentary condition if the segment hotspot matching degree is less than or equal to the segment matching degree threshold.

[0031] Specifically, the model detection unit is used to generate a first image feature vector corresponding to the target video segment through the first target network model.

[0032] The model detection unit is specifically used to generate the first audio feature vector corresponding to the target video segment through the first target network model;

[0033] The model detection unit is specifically used to generate the first text feature vector corresponding to the target video segment through the first target network model;

[0034] The model detection unit is specifically used to perform feature fusion on the first image feature vector, the first audio feature vector, and the first text feature vector to obtain the first fused feature vector corresponding to the target video segment. The first fused feature vector is then input into the first fully connected network in the first target network model, and the first fully connected network performs fully connected processing on the first fused feature vector to obtain the first target feature vector corresponding to the target video segment.

[0035] The model detection unit is specifically used to input the first target feature vector into the first classifier in the first target network model, and the first classifier outputs the segment hotspot matching degree corresponding to the target video segment.

[0036] The first target network model includes a text subnetwork;

[0037] The model detection unit is specifically used in the text sub-network to extract fragment unit text from fragment text information, embed the fragment unit text into text, and obtain the word segmentation vector corresponding to the fragment unit text.

[0038] The model detection unit is specifically used to determine the text position of a unit text in a fragment text information, perform position embedding on the text position, and obtain the position vector corresponding to the unit text in the fragment.

[0039] The model detection unit is specifically used to determine the segmentation vector corresponding to the fragment unit text. It performs feature fusion on the word segmentation vector, the position vector, and the segmentation vector corresponding to the fragment unit text to obtain the text vector to be encoded corresponding to the fragment unit text.

[0040] The model detection unit is specifically used to perform attention processing on the text vector to be encoded corresponding to the text segment unit, to obtain the encoded text vector corresponding to the text segment unit, and to determine the first text feature vector corresponding to the target video segment based on the encoded text vector corresponding to the text segment unit.

[0041] The target segment content includes video plot text;

[0042] The set determination module includes:

[0043] The word segmentation determination unit is used to take the target commentary media data and the video plot text in the target video clip as the fragment text information of the target video clip;

[0044] The word segmentation determination unit is used to process the fragment text information into words, generate fragment text words corresponding to the fragment text information, and filter fragment key words from the fragment text words;

[0045] The segment detection unit is used to obtain a set of video segments with hot comment types from the video segment library, and to retrieve candidate video segments containing segment keywords from the video segment set based on segment keyword segmentation.

[0046] The segment determination unit is used to determine the segment similarity between the target video segment and the candidate video segments. If there are candidate video segments with a segment similarity greater than the segment similarity threshold, then the candidate video segments with a segment similarity greater than the segment similarity threshold are determined as video segments that meet the segment similarity hot comment conditions with the target video segment.

[0047] Among them, the number of key word segments in the segment must be at least two;

[0048] The segment detection unit is specifically used to retrieve a set of video segments containing at least two segment keywords from the inverted index information corresponding to the video segments in the video segment set, based on at least two segment keywords.

[0049] The segment detection unit is specifically used to generate candidate video segments containing segment key words based on video segments from at least two sets of retrieved video segments;

[0050] The segment detection unit is also specifically used to perform word segmentation on the index segment text information of video segments in the video segment set, and generate index segment text words corresponding to the index segment text information;

[0051] The segment detection unit is also specifically used to filter key words from the index segment text segmentation, and to construct inverted index information corresponding to the video segments in the video segment set based on the key words of the index segment.

[0052] The segment detection unit is specifically used to retrieve initial video segments containing segment keywords from the video segment set based on segment keyword segmentation.

[0053] The segment detection unit is specifically used to generate the text similarity between the target video segment and the initial video segment based on the segment text information and the initial segment text information of the initial video segment;

[0054] The segment detection unit is specifically used to filter initial video segments whose text similarity is greater than a similarity threshold from the initial video segments, and to determine the filtered initial video segments as candidate video segments retrieved from the video segment set.

[0055] The output of the first classifier also includes a scene type vector corresponding to the target video segment; the scene type vector includes the first scene probability corresponding to N scene type labels; N is a positive integer;

[0056] The set determination module is specifically used to obtain a set of video clips with hot comment types from the video clip library, and to retrieve candidate video clips associated with the clip text information from the video clip set;

[0057] The set determination module is specifically used to obtain the candidate scene type vector corresponding to the candidate video segment; the candidate scene type vector includes the second scene probability corresponding to each of the N scene type labels;

[0058] The set determination module is specifically used to perform a first operation on the first scene probability and the second scene probability corresponding to the same scene type label in the scene type vector and the candidate scene type vector, so as to obtain the scene matching probability corresponding to each scene type label.

[0059] The set determination module is specifically used to perform a second operation on the scene matching probability corresponding to each scene type label to obtain the first similarity, and use the first similarity as the segment similarity between the target video segment and the candidate video segment;

[0060] The set determination module is specifically used to determine, if there are candidate video segments with a segment similarity greater than the segment similarity threshold, the candidate video segments with a similarity greater than the segment similarity threshold are determined as video segments that meet the segment similarity hot comment conditions with the target video segment.

[0061] The target segment content includes segment image frames and segment audio frames; the candidate segment content in the candidate video segment includes candidate segment image frames, candidate segment audio frames, and candidate video plot text.

[0062] The segment determination unit is specifically used to take the candidate commentary media data associated with the candidate video segment and the candidate video plot text in the candidate video segment as the candidate segment text information of the candidate video segment;

[0063] The segment determination unit is specifically used to input segment image frames, segment audio frames, segment text information, candidate segment image frames, candidate segment audio frames, and candidate segment text information into the second target network model;

[0064] The segment determination unit is specifically used in the second target network model to determine the second image feature vector, the second audio feature vector, and the second text feature vector corresponding to the target video segment, and to determine the candidate image feature vector, the candidate audio feature vector, and the candidate text feature vector corresponding to the candidate video segment;

[0065] The segment determination unit is specifically used to perform feature fusion on the second image feature vector and the candidate image feature vector to obtain a fused image feature vector, to perform feature fusion on the second audio feature vector and the candidate audio feature vector to obtain a fused audio feature vector, and to perform feature fusion on the second text feature vector and the candidate text feature vector to obtain a fused text feature vector.

[0066] The segment determination unit is specifically used to perform feature fusion on the fused image feature vector, fused audio feature vector, and fused text feature vector to obtain a second fused feature vector. The second fused feature vector is then input into the second fully connected network in the second target network model, where the second fully connected network performs fully connected processing on the second fused feature vector to obtain the second target feature vector.

[0067] The segment determination unit is specifically used to input the second target feature vector into the second classifier in the second target network model, and the second classifier outputs the second similarity, which is used as the segment similarity between the target video segment and the candidate video segment.

[0068] One embodiment of this application provides a video processing method, including:

[0069] Display the target video in the video playback interface;

[0070] In response to the triggering operation of the similar comment segment function in the video playback interface, a set of similar comment segments associated with the playback progress of the target video is displayed in the target area of ​​the video playback interface; the video segments in the similar comment segment set and the target video segment meet the condition of segment similarity and hot comment; the target video segment refers to the video segment indicated by the current playback progress in the target video; both the video segments in the similar comment segment set and the target video segment are video segments of the hot comment type; the association between the commentary media data associated with the video segment of the hot comment type and the segment content in the video segment satisfies the hot comment condition.

[0071] One embodiment of this application provides a video processing apparatus, including:

[0072] The video display module is used to display the target video in the video playback interface;

[0073] The collection display module is used to respond to the trigger operation of the similar comment segment function in the video playback interface. In the target area of ​​the video playback interface, it displays a collection of similar comment segments associated with the playback progress of the target video. The video segments in the similar comment segment collection and the target video segment meet the condition of segment similarity and hot comment. The target video segment refers to the video segment indicated by the current playback progress in the target video. The video segments in the similar comment segment collection and the target video segment are both video segments of the hot comment type. The association between the commentary media data associated with the video segment with the hot comment type and the segment content in the video segment meets the condition of hot comment.

[0074] The collection display module is specifically used to respond to the trigger operation of the similar segment display control with similar comment segment function in the video playback interface, and to display a collection of similar comment segments related to the playback progress of the target video in the target area of ​​the video playback interface.

[0075] Among them, the collection display module is specifically used to respond to the trigger operation of the similar comment segment function in the video playback interface, and sequentially detect the comment type of the video segments played in the target video.

[0076] The collection display module is specifically used to display a collection of similar comment segments associated with the target video segment in the target area of ​​the video playback interface when it is detected that the currently playing video segment in the target video is a target video segment with hot comment types.

[0077] The collection display module is specifically used to display hot comment prompts with similar comment segments in the video playback interface when it is detected that the currently playing video segment in the target video is a target video segment with hot comment type;

[0078] The collection display module is specifically used to respond to the triggered operation of hot comment prompts. In the target area of ​​the video playback interface, it displays a collection of similar comment segments associated with the target video segment.

[0079] One embodiment of this application provides a computer device, including: a processor and a memory;

[0080] The processor is connected to a memory, which stores a computer program. When the computer program is executed by the processor, it causes the computer device to perform the method provided in the embodiments of this application.

[0081] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having the processor performs the method provided in this application.

[0082] One embodiment of this application provides a computer program product or computer program, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in this application embodiment.

[0083] In this embodiment, a computer device can obtain a target video segment from a target video. Based on the target commentary media data associated with the target video segment and the target segment content within the target video segment, the device performs comment type detection on the target video segment to obtain a type detection result. It is understood that if the type detection result indicates that the association between the target commentary media data and the target segment content satisfies the hot comment condition, the computer device can determine that the target video segment is a video segment with a hot comment type. Then, it retrieves video segments from the video segment library that satisfy the segment similarity hot comment condition, and identifies these video segments as a set of similar comment segments associated with the target video segment. Therefore, the video segments mined in this embodiment that satisfy the segment similarity hot comment condition will also belong to the hot comment type. Since the hot comment type is determined through analysis of the video segment content, the constraint of the segment similarity hot comment condition ensures that the mined video segments and the target video segment not only both belong to the hot comment type but also share commonalities in their video segment content. This ensures that the mined video segments are sufficiently similar to the target video segments, thereby improving the accuracy of the mined video segments. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application;

[0086] Figure 2 This is a schematic diagram of a data interaction scenario provided in an embodiment of this application;

[0087] Figure 3 This is a flowchart illustrating a video processing method provided in an embodiment of this application;

[0088] Figure 4 This is a schematic diagram of a process for determining a set of similar comment fragments provided in an embodiment of this application;

[0089] Figure 5 This is a flowchart illustrating a video processing method provided in an embodiment of this application;

[0090] Figure 6 This is a flowchart illustrating a video processing method provided in an embodiment of this application;

[0091] Figure 7 This is a schematic diagram of the structure of a first network model provided in an embodiment of this application;

[0092] Figure 8 This is a flowchart illustrating a video processing method provided in an embodiment of this application;

[0093] Figure 9 This is a flowchart illustrating a video processing method provided in an embodiment of this application;

[0094] Figure 10 This is a schematic diagram of the structure of a second network model provided in an embodiment of this application;

[0095] Figure 11 This is a flowchart illustrating a video processing method provided in an embodiment of this application;

[0096] Figure 12a This is a schematic diagram illustrating a scenario for displaying a collection of similar comment fragments, provided in an embodiment of this application.

[0097] Figure 12b This is a schematic diagram illustrating a scenario for displaying a collection of similar comment fragments, provided in an embodiment of this application.

[0098] Figure 13a This is a schematic diagram of a scene for switching video segments provided in an embodiment of this application;

[0099] Figure 13b This is a schematic diagram of a scene for switching video segments provided in an embodiment of this application;

[0100] Figure 14 This is a schematic diagram of the structure of a video processing device provided in an embodiment of this application;

[0101] Figure 15 This is a schematic diagram of the structure of a video processing device provided in an embodiment of this application;

[0102] Figure 16 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0103] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0104] It should be understood that Artificial Intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0105] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0106] The solutions provided in this application mainly involve computer vision (CV) technology and machine learning (ML) technology of artificial intelligence.

[0107] Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, and intelligent transportation technologies.

[0108] Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning. Deep learning, in particular, is a technique that utilizes deep neural network systems for machine learning.

[0109] For details, please see Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application. Figure 1 As shown, this network architecture may include a server 2000 and a terminal device cluster. The terminal device cluster may specifically include one or more terminal devices; the number of terminal devices in the cluster is not limited here. Figure 1 As shown, the multiple terminal devices may specifically include terminal device 3000a, terminal device 3000b, terminal device 3000c, ..., terminal device 3000n; terminal device 3000a, terminal device 3000b, terminal device 3000c, ..., terminal device 3000n can be directly or indirectly connected to server 2000 via wired or wireless communication, so that each terminal device can interact with server 2000 through the network connection.

[0110] Each terminal device in the terminal device cluster can include: smartphones, tablets, laptops, desktop computers, smart home appliances, wearable devices, in-vehicle terminals, and other smart terminals with video processing capabilities. It should be understood that, for example... Figure 1 Each terminal device in the terminal device cluster shown can have an application client installed. When the application client runs on each terminal device, it can interact with the aforementioned... Figure 1 The servers 2000 shown interact with each other. The application client can be a standalone client or an embedded sub-client integrated into another client; this application does not impose any limitations on this.

[0111] Specifically, the application clients can include: browsers, in-vehicle clients, smart home clients, entertainment clients, multimedia clients (e.g., video clients), social media clients, and news clients, as well as other clients with video processing capabilities. The in-vehicle terminal can be a smart terminal in a smart transportation scenario, and the application clients on this in-vehicle terminal can be the aforementioned in-vehicle clients.

[0112] Among them, server 2000 can be the server corresponding to the application client. Server 2000 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0113] For ease of understanding, the embodiments of this application may be described in detail below. Figure 1 From the multiple terminal devices shown, one terminal device is selected as the target terminal device. For example, in the embodiments of this application, a terminal device can be selected as the target terminal device. Figure 1 The terminal device 3000a shown serves as the target terminal device, which may have an application client with video processing capabilities installed. In this case, the target terminal device can interact with the server 2000 through the application client.

[0114] For ease of understanding, in this embodiment of the application, the videos (e.g., TV series) that match the user's interests in the video recommendation interface of the application client can be collectively referred to as target videos. These target videos may include multiple video segments, and the video segment indicated by the current playback progress among these multiple video segments can be the target video segment. For ease of understanding, in this embodiment of the application, the bullet comments associated with the target videos can be referred to as commentary media data. Bullet comments can represent short comments sent by users watching the video, thus achieving a presentation method where users can see comments about the video while watching it.

[0115] It should be understood that the business scenarios applicable to the above network framework may specifically include: entertainment program on-demand scenarios, online movie viewing scenarios, online classroom learning scenarios, etc. The network framework can realize the push of a set of similar comment segments associated with the target video segment in business scenarios such as entertainment program on-demand scenarios, online movie viewing scenarios, and online classroom learning scenarios. The business scenarios applicable to the network framework will not be listed one by one here.

[0116] In this application, video clips in the target video clip and the set of similar comment clips are considered similar meme videos. Similar meme videos indicate that different videos share similar memes that express similar plot effects. The set of similar comment clips consists of video clips that meet the criteria of similarity to trending comments on the target video clip. Furthermore, the mining of similar meme video clips can construct a new video format, enriching the types of video resources on video websites. In this embodiment, a "meme" can represent a popular element in a video, ranging from a specific time period to a minor plot point, such as "height meme," "classic meme," "lookalike meme," "romance meme," "creative meme," or "humorous meme."

[0117] For example, in an entertainment program on-demand scenario, the target video could be an entertainment program that matches the interests of the user Y in the video recommendation interface (e.g., a video program recommendation list), and the target video clip could be an entertaining scene from that program. Similarly, in an online movie theater scenario, the target video could be a movie that matches the interests of the user Y in the video recommendation interface (e.g., a movie recommendation list), and the target video clip could be a classic scene from that movie. Likewise, in an online classroom scenario, the target video could be a course that matches the interests of the user Y in the video recommendation interface (e.g., a course recommendation list), and the target video clip could be the knowledge explanation within that course.

[0118] It should be understood that when object Y watches a target video (e.g., video P) in the application client of the target terminal device, the application client can display video P in the video playback interface. Furthermore, when object Y needs to access the similar comment segment function of the application client, it can perform a trigger operation on the similar comment segment function in the video playback interface. In this way, the application client can respond to the trigger operation and send a segment retrieval request to server 2000 based on the playback progress of video P played in the video playback interface. Therefore, after receiving the segment retrieval request sent by the application client in the target terminal device, server 2000 can determine the target video segment indicated by the playback progress of video P. Then, when it determines that the target video segment is a video segment with a hot comment type, it retrieves a set of similar comment segments associated with the target video segment (i.e., it can include the aforementioned similar meme videos), and returns the set of similar comment segments to the application client so that the application client displays the set of similar comment segments in the target area of ​​the video playback interface.

[0119] For better understanding, please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram illustrating a data interaction scenario provided in an embodiment of this application. For example... Figure 2 The server 20a shown can be the one described above. Figure 1 The server 2000 in the corresponding embodiment, such as Figure 2 The video playback interfaces 20b and 20c shown can be video playback interfaces of the application client in the target terminal device at different times. The video playback interface 20b can be used to display target video segments in the target video, and the video playback interface 20c can be used to display a collection of similar comment segments.

[0120] like Figure 2 As shown, the video playback interface 20b can play video data corresponding to the target video. When the user (e.g., object Y) of the application client performs a trigger operation on the similar comment segment function, the application client can respond to the trigger operation performed by object Y on the similar comment segment function, determine the playback progress 21b of the target video in the video playback interface 20b, and send a segment retrieval request to the server 20a based on the playback progress 21b. The trigger operation here can include contact operations such as clicking, long pressing, and swiping, or non-contact operations such as voice and gestures; this application does not limit this.

[0121] It is understood that the video playback interface 20b may include a similar meme recognition control 21b. Therefore, the triggering operation performed by object Y for the similar comment segment function can be the click operation performed by object Y for the similar meme recognition control 21b.

[0122] like Figure 2 As shown, after receiving a segment retrieval request, server 20a can obtain the playback progress 21b from the segment retrieval request and retrieve the target video segment indicated by the playback progress 21b from the video segment 22a of the target video. For example, video segment O1 can be used as the target video segment indicated by the playback progress 21b. Video segment 22a can include M video segments obtained after dividing the target video, where M can be a positive integer. The M video segments can specifically include: video segment O1, video segment O2, and video segment O... M .

[0123] Furthermore, server 20a can obtain target commentary media data associated with the target video segment and target segment content within the target video segment. Based on the target commentary media data and target segment content, it performs comment type detection on the target video segment to obtain a type detection result. The type detection result can be used to indicate whether the association between the target commentary media data and the target segment content satisfies the hot comment criteria.

[0124] It can be understood that if the type detection result indicates that the association between the target commentary media data and the target segment content meets the hot comment criteria, then server 20a can determine that the target video segment is a video segment with the hot comment type. Optionally, if the type detection result indicates that the association between the target commentary media data and the target segment content does not meet the hot comment criteria, then server 20a can determine that the target video segment is a video segment without the hot comment type.

[0125] like Figure 2 As shown, if the target video segment is a video segment with a trending comment type, server 20a can obtain a set 22b of video segments with trending comment types from the video segment library. Based on the segment keywords contained in the target video segment, server 20a obtains video segments from video segment set 22b that satisfy the condition of similarity to the target video segment in terms of trending comment types. Specifically, all video segments in video segment set 22b are video segments with trending comment types, meaning all video segments in video segment set 22b are video segments with "memes". Correspondingly, video segments that satisfy the condition of similarity to the target video segment are also video segments with trending comment types. Furthermore, video segments that satisfy the condition of similarity to the target video segment share commonalities in their video segment content. In other words, the condition of similarity to the target video segment refers to the condition that the mined video segment has a trending comment type and shares commonalities in its video segment content with the target video segment. Video segments that satisfy the condition of similarity to the target video segment can also be called "similar meme" video segments of the target video segment.

[0126] Optionally, if the target video segment is a video segment with a hot comment type, the server 20a can traverse the video segment library to obtain video segments, determine whether the traversed video segments are video segments that meet the segment similarity hot comment condition with the target video segment, and then obtain all video segments in the video segment library that meet the segment similarity hot comment condition with the target video segment after traversing all video segments in the video segment library.

[0127] The video clip set 22b may include multiple video clip sub-libraries, specifically including: video clip sub-library 23a, video clip sub-library 23b, ..., video clip sub-library 23n. These multiple video clip sub-libraries can be associated with different keyword segments; for example, video clip sub-library 23a may be associated with keyword segment F1, video clip sub-library 23b may be associated with keyword segment F2, ..., video clip sub-library 23n may be associated with keyword segment F1, F2, ..., F3, F3, F4, F5, F6, F7, F8, F9, F1, F1, F1, F2 ...2, F1, F1, F2, F1, F2, F1, F1, F2, F1, F2, F1, F2, F1, F2, F1, F2, F1, F2, F1, F2, nRelated. For example, when the segment keyword in the target video segment is keyword F1, server 20a can obtain video segments from video segment sub-library 23a that meet the segment similarity hot comment conditions with the target video segment, and then determine the video segments obtained from video segment sub-library 23a as a set of similar comment segments associated with the target video segment. Among them, the video segments in the similar comment segment set are video segments with hot comment type, and have commonalities with the target video segment in video segment content. In addition, the video segments in the similar comment segment set and the target video segment contain the same segment keyword.

[0128] The set of similar comment segments may include one or more video segments. This example uses four video segments in the set as an illustration; however, it should be understood that this embodiment does not limit the number of segments. These four video segments can be similar meme video segments, specifically including video segment L1, video segment L2, video segment L3, and video segment L4.

[0129] like Figure 2 As shown, after obtaining the set of similar comment segments, server 20a can return the set of similar comment segments to the application client. After receiving the set of similar comment segments returned by server 20a based on the segment acquisition request, the application client can switch the video playback interface from video playback interface 20b to video playback interface 20c. In the target area of ​​video playback interface 20c, video segments L1, L2, L3 and L4 from the set of similar comment segments are displayed.

[0130] The target area in the video playback interface 20c can be located anywhere within the video playback interface 20c; it can be located in the center or at the edge of the video playback interface 20c. For ease of understanding, this embodiment uses the example of the target area being located in the center of the video playback interface 20c. Correspondingly, video segments L1, L2, L3, and L4 are located in the center of the video playback interface 20c.

[0131] Understandably, when object Y needs to play a specific video segment (e.g., video segment L1) out of four video segments in the application client, it can perform a trigger operation on video segment L1. In this way, the application client can respond to the trigger operation performed by object Y on video segment L1 and play the video data corresponding to video segment L1 in the application client's video playback interface 20d (not shown in the diagram). Here, video playback interface 20d (not shown in the diagram) can be a video playback interface at a different time than video playback interfaces 20b and 20c.

[0132] Therefore, the embodiments of this application can perform comment type detection on target video segments within a target video. When the target video segment is a segment with a trending comment type (i.e., when it is determined that the target video segment contains a meme), a similarity search is performed on the target video segment in the video segment set to discover other video segments with similar memes (i.e., video segments in the similar comment segment set), thereby improving the accuracy of mining the similar comment segment set. Therefore, by using other video segments with similar memes to achieve video recommendation for a target video segment, the accuracy of video recommendation can be improved.

[0133] Further, please see Figure 3 , Figure 3 This is a flowchart illustrating a video processing method provided in an embodiment of this application. The method can be executed by a terminal device, a server, or both. The terminal device can be one of the aforementioned... Figure 2 The terminal device in the corresponding embodiment, the server can be as described above. Figure 2 The corresponding embodiment uses server 20a. For ease of understanding, this embodiment uses the method being executed by a server as an example for explanation. The video processing method may include at least the following steps S101-S103:

[0134] Step S101: Obtain the target video segment from the target video; perform comment type detection on the target video segment based on the target commentary media data associated with the target video segment and the target segment content in the target video segment, and obtain the type detection result;

[0135] Understandably, the server can divide the target video into K video segments and extract the target video segment from these K segments, where K can be a positive integer. Specifically, the server can divide the target video into K video segments with a duration of t1 seconds. For example, when t1 equals 5 seconds, the server can divide the target video into K video segments of 5 seconds each. In this case, since the total duration of the target video may not be a multiple of t1 seconds, the duration of the first or last video segment among the K video segments may not be equal to t1 seconds. It should be understood that this embodiment does not limit the specific value of t1; t1 can be a positive number.

[0136] Optionally, the server can also cluster the video frames in the target video based on the similarity between them, grouping them into K clusters, and then determining the K clusters as K video segments. Optionally, the server can also uniformly divide the target video into K video segments, each of which has the same duration, which can be t2 seconds. It should be understood that this embodiment does not limit the specific value of t2; t2 can be a positive number.

[0137] The target segment content can include video plot text, segment image frames, and segment audio frames. It is understood that segment image frames can be image frames from the target video segment, or image frames obtained by filtering image frames from the target video segment; segment audio frames can be audio frames from the target video segment, or audio frames obtained by filtering audio frames from the target video segment; the target segment content can be text information, speech text, character text, or semantic text. Character text can include speech text, and the text information is the same as the speech text.

[0138] It should be understood that the server can obtain subtitle files associated with the target video, and then extract text information associated with the target video segment from the subtitle files, using the extracted text information as the video plot text associated with the target video segment. Optionally, the server can perform speech recognition on the target video segment using ASR (Automatic Speech Recognition), and use the recognized speech text as the video plot text associated with the target video segment. This application embodiment does not limit the speech recognition algorithm used for automatic speech recognition. Optionally, the server can perform character recognition on the target video segment using OCR (Optical Character Recognition), and use the recognized character text as the video plot text associated with the target video segment. This application embodiment does not limit the character recognition algorithm used for optical character recognition.

[0139] Optionally, the server can perform semantic recognition (i.e., action recognition) on the action content in the target video segment, and use the recognized semantic text as video plot text associated with the target video segment. Optionally, the server can perform semantic recognition (i.e., scene recognition) on the scene content in the target video segment, and use the recognized semantic text as video plot text associated with the target video segment.

[0140] Understandably, the server can perform comment type detection on the target video segment based on the target commentary media data and video plot text, and obtain the type detection result. For the specific process of the server performing comment type detection on the target video segment based on the target commentary media data and video plot text, please refer to... Figure 5 The corresponding embodiment is described below. Optionally, the server can perform comment type detection on the target video segment based on the target commentary media data, video plot text, segment image frames, and segment audio frames to obtain a type detection result. The specific process of the server performing comment type detection on the target video segment based on the target commentary media data, video plot text, segment image frames, and segment audio frames can be found in [link to documentation]. Figure 6 Description of the corresponding embodiments.

[0141] Understandably, the server can generate type detection results indicating that the association between the target commentary media data and the target segment content satisfies the hot comment criteria. Specifically, when the target commentary media data frequently matches video plot text within the target segment content, the server can determine that the association between the target commentary media data and the target segment content satisfies the hot comment criteria. Optionally, the server can generate type detection results indicating that the association between the target commentary media data and the target segment content does not satisfy the hot comment criteria.

[0142] Step S102: If the type detection result indicates that the correlation between the target commentary media data and the target segment content meets the hot comment conditions, then the target video segment is determined to be a video segment with a hot comment type.

[0143] Optionally, if the type detection result indicates that the association between the target commentary media data and the target segment content does not meet the hot comment conditions, the server can determine that the target video segment is a video segment without hot comment type. Therefore, the server does not need to perform step S103.

[0144] Step S103: Obtain video segments from the video segment library that meet the segment similarity hot comment conditions with the target video segment, and determine the video segments that meet the segment similarity hot comment conditions as a set of similar comment segments associated with the target video segment.

[0145] Specifically, the server can use the target commentary media data and the video plot text in the target video clip as the fragment text information of the target video clip. Further, the server can perform word segmentation on the fragment text information to generate fragment text segments corresponding to the fragment text information, and filter fragment key segments from the fragment text segments. Further, the server can obtain a set of video clips with hot comment types from a video clip library, and retrieve candidate video clips containing the fragment key segments from the video clip set based on the fragment key segments. Further, the server can determine the fragment similarity between the target video clip and the candidate video clips. If there are candidate video clips with a fragment similarity greater than a fragment similarity threshold, then these candidate video clips with a similarity greater than the fragment similarity threshold are identified as video clips that meet the fragment similarity hot comment condition with the target video clip, and these video clips that meet the fragment similarity hot comment condition are identified as a set of similar comment fragments associated with the target video clip.

[0146] Understandably, after generating the segmented text information corresponding to the segmented text, the server can compare the segmented text with the sample key segments in the keyword list, select the segmented text that matches the sample key segments from the segmented text, and use the selected segmented text as the segment key segments.

[0147] The video clips in the video clip set are those with trending comments. The server can obtain indexed commentary media data associated with the video clips in the video clip set, as well as indexed video plot text for the video clips in the video clip set. The indexed commentary media data and indexed video plot text are used as the index clip text information for the video clips in the video clip set. The video clips in the video clip set may include target video clips.

[0148] Optionally, the server can also add video clips with trending comment types to the video clip library. This allows the server to retrieve candidate video clips containing segment keywords from the video clip library, where the video clips in the library are those with trending comment types. Thus, when the server receives a newly uploaded sample video, it can divide the sample video into Z sample video clips and perform step S102 on each of the Z sample video clips to determine whether each sample video clip is a video clip with a trending comment type. Here, Z can be a positive integer. Furthermore, the server can add the video clips with trending comment types from the Z sample video clips to the video clip library. Therefore, this embodiment can improve the efficiency of constructing similar meme video clips in newly uploaded sample videos, further enhance the richness of video types on the video platform, and provide an effective tool for interaction on the video platform.

[0149] It should be understood that the server can perform word segmentation on the index segment text information of the video segments in the video segment set, generating index segment text segments corresponding to the index segment text information. Further, the server selects key index segment words from the index segment text segments and constructs inverted index information corresponding to the video segments in the video segment set based on these key index segment words.

[0150] The server can construct the inverted index information corresponding to the video segments in the video segment collection using the Elasticsearch algorithm (a distributed, open-source search and analysis engine, abbreviated as ES algorithm). It should be understood that the embodiments of this application do not limit the construction algorithm for constructing the inverted index information.

[0151] It is understood that the server can retrieve candidate video segments containing segment keywords from the inverted index information corresponding to the video segments in the video segment set. The number of segment keywords can be one or at least two; this embodiment uses at least two segment keywords as an example. Thus, the server can retrieve a set of retrieved video segments, each containing at least two segment keywords, from the inverted index information corresponding to the video segments in the video segment set. Furthermore, the server can generate candidate video segments containing segment keywords based on the video segments in the at least two retrieved video segment sets.

[0152] The server can perform intersection processing on video segments from at least two sets of retrieved video segments, and use the resulting video segments as candidate video segments containing segment keywords. That is, video segments present in each set of retrieved video segments are considered as candidate video segments containing segment keywords. Optionally, the server can count the occurrences of each video segment in at least two sets of retrieved video segments, and use video segments with occurrences greater than a threshold as candidate video segments containing segment keywords. This embodiment does not limit the specific value of this threshold. It should be understood that this embodiment does not limit the specific method of generating candidate video segments containing segment keywords based on video segments from at least two sets of retrieved video segments.

[0153] It should be understood that the server can retrieve initial video segments containing segment keywords from the video segment set based on segment keyword segmentation. Further, the server can generate a text similarity between the target video segment and the initial video segment based on the segment text information and the initial segment text information of the initial video segment. The initial segment text information is determined by the initial commentary media data associated with the initial video segment and the initial video plot text within the initial video segment. Further, the server can filter initial video segments from the initial video segments whose text similarity is greater than a similarity threshold, and determine the filtered initial video segments as candidate video segments retrieved from the video segment set. This embodiment does not limit the specific value of the similarity threshold.

[0154] It is understood that the server can generate the text similarity between the target video segment and the initial video segment using the BM25 (Best Matching 25) algorithm. It should be understood that the embodiments of this application do not limit the specific algorithm for generating the text similarity between the target video segment and the initial video segment.

[0155] The server can generate word segmentation similarity between each segment key word in the fragment text information and the initial segment key word in the initial fragment text information. Then, it accumulates the word segmentation similarity corresponding to each segment key word to obtain the text similarity between the fragment text information and the initial fragment text information. This text similarity is used as the text similarity between the target video fragment and the initial video fragment. The specific process of the server generating the initial segment key words in the initial fragment text information can be found in the description of generating segment key words above, and will not be repeated here.

[0156] Accordingly, the server can retrieve an initial set of video segments from the video segment set, each containing at least two segment keywords, based on at least two segment keywords. Further, the server can generate a text similarity score between the target video segment and the initial video segments based on the segment text information and the initial segment text information of the initial video segments in the initial video segment set. Further, the server can filter initial video segments from the initial video segment set whose text similarity is greater than a similarity threshold, and determine the filtered initial video segments as the retrieved video segment set, each containing at least two segment keywords. Further, the server can generate candidate video segments containing segment keywords based on the video segments in the at least two retrieved video segment sets.

[0157] Understandably, the server can identify video clips that meet the criteria for similarity to trending comments as a set of similar comment clips associated with the target video clip. The number of video clips in this set is less than or equal to a threshold (H clips, where H can be a positive integer). If the number of candidate video clips meeting the criteria for similarity to trending comments exceeds the threshold, the server can select the H candidate video clips with the highest clip similarity from these clips and identify them as the set of similar comment clips associated with the target video clip. Optionally, if the number of candidate video clips meeting the criteria for similarity to trending comments exceeds the threshold, the server can select the H candidate video clips that match the user corresponding to the terminal device from these clips and identify them as the set of similar comment clips associated with the target video clip, thus achieving personalized identification of the set of similar comment clips. Optionally, if the number of candidate video segments that meet the criteria of similarity to popular comments is greater than the set threshold, the server can sort the candidate video segments that meet the criteria of similarity to popular comments based on their segment popularity, and obtain the H candidate video segments with the highest segment popularity from the candidate video segments that meet the criteria of similarity to popular comments. The H candidate video segments with the highest segment popularity are determined as the set of similar comment segments associated with the target video segment.

[0158] For easier understanding, please refer to Figure 4 , Figure 4 This is a schematic diagram of a process for determining a set of similar comment fragments according to an embodiment of this application. For example... Figure 4 As shown, after obtaining the target video segment from the target video, the server can execute step S401. Based on the comparison and mining of video bullet comments (i.e. target commentary media data) and dialogue (i.e. video plot text), step S401 determines whether the video segment (i.e. target video segment) has a plot type (i.e. meme), that is, whether the target video segment is a video segment with a hot comment type.

[0159] like Figure 4 As shown, if the target video segment is a video segment with trending comments, the server can execute step S402 to recall similar video segments. These similar video segments are candidate video segments retrieved from the video segment set that contain segment keyword segments; that is, other video segments with similar memes to the target video segment. Optionally, if the target video segment is a video segment without trending comments, the server executes step S404, at which point the process ends.

[0160] Furthermore, the server can execute step S403, which further determines similar video segments, that is, based on the segment similarity between the target video segment and the candidate video segments, determines the video segments that meet the segment similarity hot comment conditions with the target video segment, and determines the video segments that meet the segment similarity hot comment conditions as a set of similar comment segments associated with the target video segment.

[0161] Based on this, this application proposes a method for mining similar meme videos. It involves deep mining of the bullet comment data of a target video segment and comparing it with the dialogue of the target video segment to identify memes. Then, through a massive similarity search of video segments on a video platform, other candidate video segments with similar memes are obtained. Further similarity judgment is then performed on these other similar meme videos to obtain a set of similar comment segments. Therefore, this application can improve the accuracy of similar meme video segment mining, achieve automated identification of similar meme video segments, further enhance the richness of video resource types on video platforms, and provide a basis for interactive operation of video platforms.

[0162] The specific process by which the server retrieves video clips from the video clip library that meet the criteria of similarity to trending comments on the target video clip can be found in [link to relevant documentation]. Figure 8 or Figure 9 Description of the corresponding embodiments.

[0163] Therefore, it can be seen that the video segments that meet the criteria for similarity to hot comments in the target video segments mined in this application embodiment will also belong to the hot comment type. The determination of the hot comment type is obtained by analyzing the content of the video segments. Therefore, by constraining the criteria for similarity to hot comments, it can be ensured that the mined video segments and the target video segments not only both belong to the hot comment type, but also have commonalities in the content of the video segments. This can ensure that the mined video segments are sufficiently similar to the target video segments, thereby improving the accuracy of the mined video segments.

[0164] Further, please see Figure 5 , Figure 5 This is a flowchart illustrating a video processing method provided in an embodiment of this application. The video processing method may include the following steps S11-S16, and steps S11-S16 are... Figure 3 A specific embodiment of step S101 in the corresponding embodiment.

[0165] Step S11: Extract commentary text clauses from the target commentary media data associated with the target video segment;

[0166] The target commentary media data consists of emoticons, symbols, and text. The server can remove emoticons and symbols from the target commentary media data and use the text in each target commentary media data as the corresponding commentary text clause.

[0167] Step S12: Based on the sentence segmentation characters, the video plot text in the target video segment is segmented into S plot text clauses;

[0168] Here, S can be a positive integer. For ease of understanding, this application uses an example where S is a positive integer greater than 1. Clause characters (i.e., clause symbols) can be commas, periods, exclamation marks, semicolons, etc. This application does not list all clause characters.

[0169] Step S13: Determine the text similarity coefficient between each plot text clause and the commentary text clause in the S plot text clauses. Based on the text similarity coefficient, the number of interactions of the commentary text clause, and the text length of the commentary text clause, generate the text hotspot matching degree between each plot text clause and the commentary text clause respectively.

[0170] The server can determine the text similarity coefficient between each plot text clause and commentary text clause using the Jaccard similarity coefficient. The Jaccard similarity coefficient represents the proportion of the intersection elements of set A and set B in the union elements of set A and set B, where set A can be each plot text clause and set B can be commentary text clauses.

[0171] It should be understood that here, S is taken as an example of plot text clause S out of S plot text clauses. i To illustrate, let's take an example with G commentary text clauses, where i can be a positive integer less than or equal to S, and G can be a positive integer. The server can determine the plot text clauses S. i The text similarity coefficients between each of the G commentary text clauses are calculated. Then, based on the text similarity coefficient of each commentary text clause, the number of interactions with each commentary text clause, and the text length of each commentary text clause, plot text clause S is generated. i The server can further analyze the text hotspot matching degree between each commentary text clause and the plot text clause S. i The text hotspot matching scores between each commentary text clause are merged to obtain the plot text clause S. i The corresponding text hotspot matching degree (i.e., plot text clause S) i (Text hotspot matching degree between the text and the commentary text clauses).

[0172] Specifically, the server can use the product of the text similarity coefficient corresponding to each commentary text clause, the number of interactions for each commentary text clause, and the text length of each commentary text clause as the plot text clause S. i The server calculates the first text match degree between each commentary text clause and the first text match degree between each commentary text clause; the server can use the product of the number of interactions and the text length of each commentary text clause as the plot text clause S. i The second text match score between each commentary text clause and the first text match score. Therefore, the server can use the ratio of the first text match score to the second text match score for each commentary text clause as the plot text clause S. i The text hotspot matching degree between each commentary text clause and the text hotspot matching degree. For example, G and the commentary text clauses may include the commentary text clause G. j Here, j can be a positive integer less than or equal to G, so that the server can process the commentary text clause G. j The corresponding text similarity coefficient and commentary text clause G j Number of interactions and commentary text clauses G j The product of the text lengths is used as the plot text clause S. i With commentary text clause G j The first text match between them; the server can then use the commentary text clause G j Number of interactions and commentary text clauses G j The product of the text lengths is used as the plot text clause S. i With commentary text clause G j The second text match degree between them. Therefore, the server can use the commentary text clause G j The corresponding first text match score and commentary text clause G j The ratio of the corresponding second text matching degree is used as the plot text clause S. i With commentary text clause G j The degree of text hotspot matching between them.

[0173] Among them, the server can process the plot text clause S i The text hotspot matching scores between each commentary text clause are accumulated to obtain the plot text clause S. i The corresponding text hotspot matching degree. In other words, the above merging process can be a cumulative process.

[0174] Step S14: Merge the text hotspot matching scores corresponding to each plot text clause to obtain the merged text hotspot matching scores;

[0175] The server can accumulate the text hotspot matching scores corresponding to each plot text clause to obtain a merged text hotspot matching score. Optionally, the server can compare the text hotspot matching scores corresponding to each plot text clause and use the maximum text hotspot matching score among the text hotspot matching scores corresponding to each plot text clause as the merged text hotspot matching score.

[0176] Understandably, the server can determine whether the hotspot matching degree of the merged text is greater than the text matching degree threshold. If the hotspot matching degree of the merged text is greater than the text matching degree threshold, then step S15 is executed; if the hotspot matching degree of the merged text is less than or equal to the text matching degree threshold, then step S16 is executed.

[0177] Step S15: If the hotspot matching degree of the merged text is greater than the text matching degree threshold, a type detection result is generated to indicate that the association between the target commentary media data and the target segment content meets the hotspot commentary condition.

[0178] Step S16: If the hotspot matching degree of the merged text is less than or equal to the text matching degree threshold, a type detection result is generated to indicate that the association between the target commentary media data and the target segment content does not meet the hotspot commentary conditions.

[0179] In other words, the embodiments of this application can determine whether a certain pair of white sentences in a target video segment appears frequently in the bullet comments (i.e., target commentary media data) within the corresponding time range of the target video segment. If a certain pair of white sentences appears frequently in the bullet comments within the corresponding time range of the target video segment, a type detection result is generated to indicate that the relationship between the target commentary media data and the content of the target segment meets the hot comment conditions, that is, the type detection result indicates that the target video segment contains a meme; if a certain pair of white sentences does not appear frequently in the bullet comments within the corresponding time range of the target video segment, a type detection result is generated to indicate that the relationship between the target commentary media data and the content of the target segment does not meet the hot comment conditions, that is, the type detection result indicates that the target video segment does not contain a meme.

[0180] Optionally, the server can determine whether there exists a text hotspot match score greater than a text matching score threshold in the text hotspot match score corresponding to each plot text clause. If there is a text hotspot match score greater than the text matching score threshold in the text hotspot match score corresponding to each plot text clause, a type detection result is generated to indicate that the association between the target commentary media data and the target segment content meets the hotspot comment conditions. Optionally, if there is no text hotspot match score greater than the text matching score threshold in the text hotspot match score corresponding to each plot text clause, a type detection result is generated to indicate that the association between the target commentary media data and the target segment content does not meet the hotspot comment conditions.

[0181] Therefore, this application embodiment can determine the text hotspot matching degree between each plot text clause and the comment text clause in the target commentary media data and S plot text clauses in the video plot text, and then determine the combined text hotspot matching degree between the target commentary media data and the video plot text based on the text hotspot matching degree between each plot text clause and the comment text clause. Based on this, comment type detection can be performed on the target video segment using the target commentary media data (i.e., bullet comments) and the video plot text (i.e., dialogue). Based on the combined text hotspot matching degree generated by the comment type detection, a type detection result is generated to indicate whether the association between the target commentary media data and the target segment content meets the hotspot comment condition, thus improving the accuracy of comment type detection for the target video segment.

[0182] Further, please see Figure 6 , Figure 6 This is a flowchart illustrating a video processing method provided in an embodiment of this application. The video processing method may include the following steps S21-S25, and steps S21-S25 are... Figure 3 A specific embodiment of step S101 in the corresponding embodiment.

[0183] Step S21: Use the target commentary media data associated with the target video segment and the video plot text in the target video segment as the segment text information of the target video segment;

[0184] Step S22: Input the segment image frames, segment audio frames, and segment text information of the target video segment into the first target network model;

[0185] The first target network model is obtained by training the first initial network model. The first target network model may include a text sub-network (i.e., the first text sub-network), an image sub-network (i.e., the first image sub-network), and an audio sub-network (i.e., the first audio sub-network).

[0186] It should be understood that the first initial network model and the first target network model can be collectively referred to as the first network model. The first initial network model and the first target network model are names of the first network model at different times. During the training phase, the first network model can be called the first initial network model, and during the prediction phase, the first network model can be called the first target network model.

[0187] Step S23: Detect comment type for the target video segment using the first target network model, and output the segment hotspot matching degree corresponding to the target video segment;

[0188] Specifically, the server can generate a first image feature vector corresponding to the target video segment using the first target network model. Further, the server can generate a first audio feature vector corresponding to the target video segment using the first target network model. Further, the server can generate a first text feature vector corresponding to the target video segment using the first target network model. Further, the server can perform feature fusion on the first image feature vector, the first audio feature vector, and the first text feature vector to obtain a first fused feature vector corresponding to the target video segment. This first fused feature vector is then input into the first fully connected network in the first target network model, where it undergoes fully connected processing to obtain the first target feature vector corresponding to the target video segment. Further, the server can input the first target feature vector into the first classifier in the first target network model, where the first classifier outputs the segment hotspot matching degree corresponding to the target video segment.

[0189] It should be understood that the server can input segment image frames into the first image sub-network, and the first image sub-network can extract image features from the segment image frames to obtain the first image feature vector corresponding to the target video segment; the server can input segment audio frames into the first audio sub-network, and the first audio sub-network can extract audio features from the segment audio frames to obtain the first audio feature vector corresponding to the target video segment; the server can input segment text information into the first text sub-network, and the first text sub-network can extract text features from the segment text information to obtain the first text feature vector corresponding to the target video segment.

[0190] The first image sub-network may include a first image network layer and a second image network layer. Understandably, the server can input a segment image frame into the first image network layer, which extracts image features from the segment image frame to obtain the corresponding image feature vector. Further, the server can input the image feature vector corresponding to the segment image frame into the second image network layer, which performs attention processing on the image feature vector to obtain an attention image feature vector. Based on the attention image feature vector, the first image feature vector corresponding to the target video segment is determined.

[0191] It should be understood that the first image network layer can be an EfficientNet (Rethinking Model Scaling for Convolutional Neural Networks, a multi-dimensional hybrid model scaling method) model, and the second image network layer can be a self-attention model. The embodiments of this application do not limit the specific types of the first and second image network layers.

[0192] The first audio sub-network can include a first audio network layer and a second audio network layer. Understandably, the server can input a segment of audio frame into the first audio network layer, which extracts audio features from the segment to obtain an audio feature vector. Further, the server can input the audio feature vector into the second audio network layer, which performs attention processing on the feature vector to obtain an attention-based audio feature vector. Based on this attention-based audio feature vector, the first audio feature vector corresponding to the target video segment is determined.

[0193] It should be understood that the first audio network layer can be a VGGish model (i.e., a model pre-trained on YouTube's AudioSet data), and the second audio network layer can be a self-attention model. This application does not limit the specific types of the first and second audio network layers.

[0194] Specifically, it should be understood that the server can extract segment unit text from the segment text information in the first text sub-network, embed the segment unit text into text, and obtain the word segmentation vector corresponding to the segment unit text. Further, the server can determine the text position of the segment unit text in the segment text information, embed the text position into position, and obtain the position vector corresponding to the segment unit text. Further, the server can determine the segmentation vector corresponding to the segment unit text, fuse the word segmentation vector, the position vector, and the segmentation vector corresponding to the segment unit text to obtain the text vector to be encoded corresponding to the segment unit text. Further, the server can perform attention processing on the text vector to be encoded corresponding to the segment unit text to obtain the encoded text vector corresponding to the segment unit text, and based on the encoded text vector corresponding to the segment unit text, determine the first text feature vector corresponding to the target video segment.

[0195] It is understood that the first text sub-network can be a lightweight BERT model (A Lite BERT for Self-supervised Learning of Language Representations, abbreviated as ALBERT). Optionally, the first text sub-network can also be a Bidirectional Encoder Representations from Transformers (BERT) model. It should be understood that the embodiments of this application do not limit the specific type of the first text sub-network.

[0196] It is understood that the server can perform feature fusion on the first image feature vector, the first audio feature vector, and the first text feature vector using either a weighted average or vector concatenation. It should be understood that this application embodiment does not limit the specific method of feature fusion.

[0197] It is understandable that the first fully connected network can be a multi-layer fully connected network, which can perform non-linear transformation on the input features (i.e., the first fused feature vector) to obtain the output features (i.e., the first target feature vector). In addition, the multi-layer fully connected network can also compress the dimensions of the first fused feature vector to obtain the compressed first target feature vector. For example, the 4000-dimensional first fused feature vector can be compressed into a 1000-dimensional first target feature vector.

[0198] Understandably, the server can determine whether the segment hotspot matching degree is greater than the segment matching degree threshold. If the segment hotspot matching degree is greater than the segment matching degree threshold, then step S24 is executed; if the segment hotspot matching degree is less than or equal to the segment matching degree threshold, then step S25 is executed.

[0199] Step S24: If the segment hotspot matching degree is greater than the segment matching degree threshold, a type detection result is generated to indicate that the association between the target commentary media data and the target segment content meets the hotspot commentary condition.

[0200] Step S25: If the segment hotspot matching degree is less than or equal to the segment matching degree threshold, a type detection result is generated to indicate that the association between the target commentary media data and the target segment content does not meet the hotspot commentary condition.

[0201] Optionally, the output of the first classifier may also include a scene type vector corresponding to the target video segment. The scene type vector can be used in step S103 to determine the segment similarity between the target video segment and the candidate video segments. The scene type vector can be a one-dimensional list of floating-point numbers, where each element represents the probability that the target video segment belongs to a different atmosphere type.

[0202] It should be understood that the specific process by which the server trains the first initial network model to obtain the first target network model can be described as follows: The server can obtain the hot comment type labels and scene type labels of the first sample video segment, and input the first sample segment image frame, the first sample segment audio frame, and the first sample segment text information of the first sample video segment into the first initial network model. Further, the server can determine the first sample image feature vector, the first sample audio feature vector, and the first sample text feature vector corresponding to the first sample video segment within the first initial network model. Further, the server can perform feature fusion on the first sample image feature vector, the first sample audio feature vector, and the first sample text feature vector to obtain the first sample fused feature vector corresponding to the first sample video segment. This first sample fused feature vector is then input into the first initial fully connected network in the first initial network model, where the first initial fully connected network performs fully connected processing on the first sample fused feature vector to obtain the first sample target feature vector corresponding to the first sample video segment. Furthermore, the server can input the first sample target feature vector into the first initial classifier in the first initial network model, and the first initial classifier outputs the sample segment hotspot matching degree and sample scene type vector corresponding to the first sample video segment. Further, the server can determine the segment hotspot loss value of the first initial network model based on the sample segment hotspot matching degree and hotspot comment type label, and determine the scene type loss value of the first initial network model based on the sample scene type vector and scene type label. Further, the server can determine the total loss value of the first initial network model based on the segment hotspot loss value and scene type loss value, and adjust the model parameters in the first initial network model according to the total loss value to obtain the first target network model.

[0203] The "Hot Comment Type" label indicates whether the first sample video segment has a hot comment type. If it does, the label is "1"; otherwise, it is "0". The "Scene Type" label indicates the atmosphere of the first sample video segment, which can be humorous, sad, serious, sweet, fighting, or debating. Therefore, the hot comment matching degree represents the probability that the first sample video segment has a hot comment type, and the scene type vector represents the probability distribution of the first sample video segment belonging to that atmosphere type.

[0204] It should be understood that the specific process by which the server determines the first sample image feature vector, the first sample audio feature vector, and the first sample text feature vector through the first initial network model can be referred to in the above description of determining the first image feature vector, the first audio feature vector, and the first text feature vector through the first target network model, and will not be repeated here.

[0205] It should be understood that the specific process of the server performing feature fusion on the first sample image feature vector, the first sample audio feature vector, and the first sample text feature vector can be found in the above description of feature fusion on the first image feature vector, the first audio feature vector, and the first text feature vector, and will not be repeated here.

[0206] For easier understanding, please refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of a first network model provided in an embodiment of this application. For example... Figure 7 The model structure shown can be either the model structure of the first initial network model or the model structure of the first target network model. For ease of understanding, it is shown here as follows. Figure 7 The model structure shown is an example of the first target network model.

[0207] In such Figure 7 When the model structure shown corresponds to the first target network model, the image subnetwork 70a can be a first image subnetwork, the audio subnetwork 70b can be a first audio subnetwork, and the text subnetwork 70c can be a first text subnetwork. Specifically, the image network layer 71a in the image subnetwork 70a can be a first image network layer, and the image network layer 71b in the image subnetwork 70a can be a second image network layer; similarly, the audio network layer 72a in the audio subnetwork 70b can be a first audio network layer, and the audio network layer 72b in the image subnetwork 70b can be a second audio network layer.

[0208] At this time, as Figure 7The video clip shown can be a target video clip, the image frame sequence can be a clip image frame, the audio frame sequence can be a clip audio frame, and the video clip text can be clip text information. For example... Figure 7 As shown, the server can input segment image frames into the image sub-network 70a, and determine the image feature vector (i.e., the first image feature vector) corresponding to the target video segment through the image sub-network 70a; the server can input segment audio frames into the audio sub-network 70b, and determine the audio feature vector (i.e., the first audio feature vector) corresponding to the target video segment through the audio sub-network 70b; the server can perform text processing on the segment text information to obtain the text vector to be encoded corresponding to the segment unit text in the segment text information, and input the text vector to be encoded corresponding to the segment unit text into the text sub-network 70c, and determine the text feature vector (i.e., the first text feature vector) corresponding to the target video segment through the text sub-network 70c.

[0209] like Figure 7 As shown, the server can generate a multi-dimensional feature fusion representation of the target video segment based on image feature vectors, audio feature vectors, and text feature vectors. Then, based on this multi-dimensional feature fusion representation, it can generate the segment feature matching degree (i.e., the probability of a meme in the target video segment) and scene type vector (i.e., the atmosphere type distribution of the target video segment). By jointly modeling these two tasks—segment feature matching degree and scene type vector—the modeling capability of the first target network model can be improved.

[0210] Understandably, the server can be based on Figure 5 The corresponding embodiment merges text hotspot matching scores to generate type detection results for the target video segment; the server can also be based on... Figure 6 The segment hotspot matching degree in the corresponding embodiment generates the type detection result of the target video segment.

[0211] Optionally, the server can also use the product of the merged text hotspot matching score and the segment hotspot matching score as the merged matching score, and then generate a type detection result for the target video segment based on the merged matching score, the merged text hotspot matching score, and the segment hotspot matching score. Specifically, if the merged text hotspot matching score is greater than the text matching score threshold, the segment hotspot matching score is greater than the segment matching score threshold, and the merged matching score is greater than the merged matching score threshold, then a type detection result is generated indicating that the association between the target commentary media data and the target segment content meets the hotspot comment conditions. If the merged text hotspot matching score is not greater than the text matching score threshold, or the segment hotspot matching score is not greater than the segment matching score threshold, or the merged matching score is not greater than the merged matching score threshold, then a type detection result is generated indicating that the association between the target commentary media data and the target segment content does not meet the hotspot comment conditions.

[0212] Therefore, the embodiments of this application can perform multi-dimensional modeling of target commentary media data, segment image frames, segment audio frames, and segment text information. A first target network model is used to perform multimodal similarity judgment on the target video segment to obtain the segment hotspot matching degree corresponding to the target video segment. Based on this, comment type detection can be performed on the target video segment using the target commentary media data (i.e., bullet comments) and video plot text (i.e., dialogue). Then, based on the segment hotspot matching degree generated by the comment type detection, a type detection result is generated to indicate whether the association between the target commentary media data and the target segment content meets the hotspot comment conditions, thus improving the accuracy of comment type detection for the target video segment.

[0213] Further, please see Figure 8 , Figure 8 This is a flowchart illustrating a video processing method provided in an embodiment of this application. The video processing method may include the following steps S31-S35, and steps S31-S35 are... Figure 3 A specific embodiment of step S103 in the corresponding embodiment.

[0214] Step S31: Obtain a set of video clips with hot comment types from the video clip library, and retrieve candidate video clips associated with the clip text information from the video clip set;

[0215] The specific process by which the server retrieves candidate video segments associated with segment text information from the video segment set can be found above. Figure 3 The description of step S103 in the corresponding embodiments will not be repeated here.

[0216] Step S32: Obtain the candidate scene type vector corresponding to the candidate video segment;

[0217] Understandable, Figure 6 The output of the first classifier in the corresponding embodiment also includes a scene type vector corresponding to the target video segment; the scene type vector includes the first scene probability corresponding to N scene type labels. Here, N can be a positive integer.

[0218] The candidate scene type vector includes the second scene probability corresponding to N scene type labels. This candidate scene type vector is generated by inputting the candidate video segment image frames, candidate segment audio frames, and candidate segment text information into... Figure 6Following the first target network model in the corresponding embodiment, the output from the first classifier in the first target network model should be understood as follows: the specific process by which the server outputs candidate scene type vectors through the first target network model can be found in the above description of outputting scene type vectors through the first target network model, and will not be repeated here. Here, one scene type label corresponds to one first scene probability and one second scene probability.

[0219] It should be understood that the scene type label corresponding to the highest first scene probability in the scene type vector can be the atmosphere type of the target video segment, and the scene type label corresponding to the highest second scene probability in the candidate scene type vector can be the atmosphere type of the candidate video segment.

[0220] Step S33: In the scene type vector and the candidate scene type vector, perform a first operation on the first scene probability and the second scene probability corresponding to the same scene type label to obtain the scene matching probability corresponding to each scene type label.

[0221] The first operation can be a multiplication operation.

[0222] Step S34: Perform a second operation on the scene matching probability corresponding to each scene type label to obtain the first similarity, and use the first similarity as the segment similarity between the target video segment and the candidate video segment;

[0223] The second operation can be an addition operation.

[0224] Step S35: If there are candidate video segments with a similarity greater than the segment similarity threshold, then the candidate video segments with a similarity greater than the segment similarity threshold are determined as video segments that meet the segment similarity hot comment conditions with the target video segment.

[0225] It is understood that the number of candidate video segments is one or at least two. This embodiment of the application uses at least two candidate video segments as an example for illustration. In this way, the server can determine the first similarity between the target video segment and at least two candidate video segments, and then determine whether the first similarity is greater than the segment similarity threshold. If there are candidate video segments among the first similarities that are greater than the segment similarity threshold, then the candidate video segments among the at least two candidate video segments that are greater than the segment similarity threshold are determined as video segments that meet the segment similarity hot comment conditions with the target video segment.

[0226] Optionally, if there are no candidate video segments with a similarity greater than the segment similarity threshold in the first similarity, the server cannot determine a video segment that meets the segment similarity hot comment condition with the target video segment.

[0227] Therefore, this application embodiment can obtain the scene type vector corresponding to the target video segment and the candidate scene type vector corresponding to the candidate video segment. Based on the first scene probability and the second scene probability corresponding to the same scene type label, a segment similarity between the target video segment and the candidate video segment is generated. Then, through the segment similarity, similar comment video segments associated with the target video segment are automatically mined from the video segment library. The target video segment and the similar comment video segments share commonalities in content and both belong to the category of trending comment videos, thereby expanding the richness of video resources on the video platform and improving the accuracy of the mined similar comment video segments. Based on this, the accuracy of video recommendations can be improved when using similar comment video segments for video recommendation.

[0228] Further, please see Figure 9 , Figure 9 This is a flowchart illustrating a video processing method provided in an embodiment of this application. The video processing method may include the following steps S41-S48, and steps S41-S48 are... Figure 3 A specific embodiment of step S103 in the corresponding embodiment.

[0229] Step S41: Obtain a set of video clips with hot comment types from the video clip library, and retrieve candidate video clips associated with the clip text information from the video clip set;

[0230] The specific process by which the server retrieves candidate video segments associated with segment text information from the video segment set can be found above. Figure 3 The description of step S103 in the corresponding embodiments will not be repeated here.

[0231] The candidate video clip content can include candidate clip image frames, candidate clip audio frames, and candidate video plot text.

[0232] Step S42: The candidate commentary media data associated with the candidate video segment and the candidate video plot text in the candidate video segment are used as the candidate segment text information of the candidate video segment;

[0233] Step S43: Input the fragment image frame, fragment audio frame, fragment text information, candidate fragment image frame, candidate fragment audio frame, and candidate fragment text information into the second target network model;

[0234] The second target network model is obtained by training the second initial network model. The second target network model may include a second text subnetwork, a second image subnetwork, a second audio subnetwork, a third text subnetwork, a third image subnetwork, and a third audio subnetwork. The second and third text subnetworks have the same network structure, the second and third image subnetworks have the same network structure, and the second and third audio subnetworks have the same network structure.

[0235] It should be understood that the second initial network model and the second target network model can be collectively referred to as the second network model. The second initial network model and the second target network model are simply names for the second network model at different times. During the training phase, the second network model can be called the second initial network model, and during the prediction phase, the second network model can be called the second target network model.

[0236] Step S44: In the second target network model, determine the second image feature vector, the second audio feature vector, and the second text feature vector corresponding to the target video segment; and determine the candidate image feature vector, the candidate audio feature vector, and the candidate text feature vector corresponding to the candidate video segment.

[0237] It should be understood that the server can input segment image frames into the second image sub-network, which will extract image features from the segment image frames to obtain the second image feature vector corresponding to the target video segment; the server can input segment audio frames into the second audio sub-network, which will extract audio features from the segment audio frames to obtain the second audio feature vector corresponding to the target video segment; the server can input segment text information into the second text sub-network, which will extract text features from the segment text information to obtain the second text feature vector corresponding to the target video segment.

[0238] The specific process of outputting the second image feature vector through the second image sub-network can be found in the above. Figure 6 The description of outputting the first image feature vector through the first image sub-network in the corresponding embodiment will not be repeated here. The specific process of outputting the second audio feature vector through the second audio sub-network can be found above. Figure 6 The description of outputting the first audio feature vector through the first audio sub-network in the corresponding embodiment will not be repeated here. The specific process of outputting the second text feature vector through the second text sub-network can be found above. Figure 6 The description of the first text feature vector output by the first text sub-network in the corresponding embodiment will not be repeated here.

[0239] Similarly, it should be understood that the server can input candidate segment image frames into the third image sub-network, which will extract image features from the candidate segment image frames to obtain the third image feature vector corresponding to the candidate video segment; the server can input candidate segment audio frames into the third audio sub-network, which will extract audio features from the candidate segment audio frames to obtain the third audio feature vector corresponding to the candidate video segment; the server can input candidate segment text information into the third text sub-network, which will extract text features from the candidate segment text information to obtain the third text feature vector corresponding to the candidate video segment.

[0240] The specific process of outputting candidate image feature vectors through the third image sub-network can be found in the above. Figure 6 The description of outputting the first image feature vector through the first image sub-network in the corresponding embodiment will not be repeated here. The specific process of outputting the candidate audio feature vector through the third audio sub-network can be found above. Figure 6 The description of outputting the first audio feature vector through the first audio sub-network in the corresponding embodiment will not be repeated here. The specific process of outputting the candidate text feature vector through the third text sub-network can be found above. Figure 6 The description of the first text feature vector output by the first text sub-network in the corresponding embodiment will not be repeated here.

[0241] Step S45: Perform feature fusion on the second image feature vector and the candidate image feature vector to obtain a fused image feature vector; perform feature fusion on the second audio feature vector and the candidate audio feature vector to obtain a fused audio feature vector; perform feature fusion on the second text feature vector and the candidate text feature vector to obtain a fused text feature vector.

[0242] It is understood that the server may fuse the second image feature vector and the candidate image feature vector, the second audio feature vector and the candidate audio feature vector, and the second text feature vector and the candidate text feature vector using either a weighted average or a vector concatenation method. It should be understood that this application does not limit the specific method of feature fusion.

[0243] Optionally, the server can fuse the second image feature vector and the candidate image feature vector to obtain an initial fused image feature vector. This initial fused image feature vector is then input into the fully connected image layer of the second target network model, where it performs fully connected processing on the initial fused image features to obtain the fused image feature vector. Similarly, the server can fuse the second audio feature vector and the candidate audio feature vector to obtain an initial fused audio feature vector. This initial fused audio feature vector is then input into the fully connected audio layer of the second target network model, where it performs fully connected processing on the initial fused audio features to obtain the fused audio feature vector. Likewise, the server can fuse the second text feature vector and the candidate text feature vector to obtain an initial fused text feature vector. This initial fused text feature vector is then input into the fully connected text layer of the second target network model, where it performs fully connected processing on the initial fused text features to obtain the fused text feature vector.

[0244] Step S46: Perform feature fusion on the fused image feature vector, fused audio feature vector, and fused text feature vector to obtain a second fused feature vector. Input the second fused feature vector into the second fully connected network in the second target network model. The second fully connected network performs fully connected processing on the second fused feature vector to obtain the second target feature vector.

[0245] It is understood that the server can perform feature fusion on the fused image feature vector, fused audio feature vector, and fused text feature vector in either a weighted average or a vector concatenation method. It should be understood that this application embodiment does not limit the specific method of feature fusion.

[0246] It is understandable that the second fully connected network can be a multi-layer fully connected network, which can perform nonlinear transformation on the input features (i.e., the second fused feature vector) to obtain the output features (i.e., the second target feature vector). In addition, the multi-layer fully connected network can also perform dimensional compression on the second fused feature vector to obtain the compressed second target feature vector.

[0247] Step S47: Input the second target feature vector into the second classifier in the second target network model, and the second classifier outputs the second similarity. Use the second similarity as the segment similarity between the target video segment and the candidate video segment.

[0248] Step S48: If there are candidate video segments with a similarity greater than the segment similarity threshold, then the candidate video segments with a similarity greater than the segment similarity threshold are determined as video segments that meet the segment similarity hot comment conditions with the target video segment.

[0249] It is understood that the number of candidate video segments is one or at least two. This embodiment of the application uses at least two candidate video segments as an example for illustration. In this way, the server can determine the second similarity between the target video segment and at least two candidate video segments, and then determine whether the second similarity is greater than the segment similarity threshold. If there are candidate video segments whose second similarity is greater than the segment similarity threshold, then the candidate video segments with a similarity to the segment similarity threshold among the at least two candidate video segments are determined as video segments that meet the segment similarity hot comment conditions with the target video segment.

[0250] Optionally, if there are no candidate video segments with a similarity greater than the segment similarity threshold in the second similarity, the server cannot determine the video segment that meets the segment similarity hot comment condition with the target video segment.

[0251] It should be understood that the specific process by which the server trains the second initial network model to obtain the second target network model can be described as follows: The server can obtain the segment similarity labels of the second sample video segment and the third sample video segment, and input the second sample segment image frame, the second sample segment audio frame, the second sample segment text information of the second sample video segment, the third sample segment image frame, the third sample segment audio frame, and the third sample segment text information of the third sample video segment into the second initial network model. Further, the server can determine the second sample image feature vector, the second sample audio feature vector, and the second sample text feature vector corresponding to the second sample video segment in the second initial network model, and determine the third sample image feature vector, the third sample audio feature vector, and the third sample text feature vector corresponding to the third sample video segment. Furthermore, the server can fuse the feature vectors of the second and third sample images to obtain a fused image feature vector, fuse the feature vectors of the second and third sample audios to obtain a fused audio feature vector, and fuse the feature vectors of the second and third sample texts to obtain a fused text feature vector. Further, the server can fuse the fused image, fused audio, and fused text feature vectors to obtain a second fused feature vector. This second fused feature vector is then input into the second fully connected network in the second initial network model. The second fully connected network performs fully connected processing on the second fused feature vector to obtain the second sample target feature vector. Further, the server can input the second sample target feature vector into the second initial classifier in the second initial network model. The second initial classifier outputs the predicted segment similarity between the second and third sample video segments. Further, the server can determine the similarity loss value of the second initial network model based on the predicted segment similarity and segment similarity labels. Based on the similarity loss value, the server adjusts the model parameters in the second initial network model to obtain the second target network model.

[0252] The segment similarity label can represent the classification type of the second sample video segment and the third sample video segment. If the second sample video segment and the third sample video segment match (i.e., the same melodies), the segment similarity label can be "1". If the second sample video segment and the third sample video segment do not match (i.e., the same melodies), the segment similarity label can be "0".

[0253] It should be understood that the specific process by which the server determines the feature vectors of the second sample image, the second sample audio, and the second sample text through the second initial network model can be referred to the above description of determining the feature vectors of the first image, the first audio, and the first text through the first target network model, and will not be repeated here. Similarly, it should be understood that the specific process by which the server determines the feature vectors of the third sample image, the third sample audio, and the third sample text through the third initial network model can be referred to the above description of determining the feature vectors of the first image, the first audio, and the first text through the first target network model, and will not be repeated here.

[0254] It should be understood that the specific process of the server performing feature fusion on the feature vectors of the second and third sample images can be found in the above description of feature fusion on the feature vectors of the second and candidate images, and will not be repeated here. Similarly, the specific process of the server performing feature fusion on the feature vectors of the second and third sample audio images can be found in the above description of feature fusion on the second and candidate audio images, and will not be repeated here. Likewise, the specific process of the server performing feature fusion on the feature vectors of the second and third sample text images can be found in the above description of feature fusion on the second and candidate text images, and will not be repeated here.

[0255] It should be understood that the specific process of the server performing feature fusion on the sample fused image feature vector, sample fused audio feature vector, and sample fused text feature vector can be found in the above description of feature fusion on the fused image feature vector, fused audio feature vector, and fused text feature vector, and will not be repeated here.

[0256] For easier understanding, please refer to Figure 10 , Figure 10 This is a schematic diagram of the structure of a second network model provided in an embodiment of this application. For example... Figure 10 The model structure shown can be either the model structure of the second initial network model or the model structure of the second target network model. For ease of understanding, it is shown here as follows. Figure 10 The model structure shown is an example of the model structure of the second target network model.

[0257] In such Figure 10When the model structure shown corresponds to the second target network model, image subnetwork 100a can be the second image subnetwork, audio subnetwork 100b can be the second audio subnetwork, text subnetwork 100c can be the second text subnetwork, image subnetwork 100d can be the third image subnetwork, audio subnetwork 100e can be the third audio subnetwork, and text subnetwork 100f can be the third text subnetwork. Specifically, image network layer 101a in image subnetwork 100a can be the third image network layer, and image network layer 101b in image subnetwork 100a can be the fourth image network layer; similarly, audio network layer 102a in audio subnetwork 100b can be the third audio network layer, and audio network layer 102b in image subnetwork 100b can be the fourth audio network layer. Among them, the image network layer 103a in the image sub-network 100d can be the fifth image network layer, and the image network layer 103b in the image sub-network 100d can be the sixth image network layer; the audio network layer 104a in the audio sub-network 100e can be the fifth audio network layer, and the audio network layer 104b in the image sub-network 100e can be the sixth audio network layer.

[0258] At this time, as Figure 10 The current video clip shown can be the target video clip. The image frame sequence of the target video clip can be clip image frames, the audio frame sequence of the target video clip can be clip audio frames, and the video clip text of the target video clip can be clip text information; for example... Figure 7 The recalled video clips shown can be candidate video clips, the image frame sequence of the candidate video clips can be candidate clip image frames, the audio frame sequence of the candidate video clips can be candidate clip audio frames, and the video clip text of the candidate video clips can be candidate clip text information.

[0259] like Figure 10 As shown, the server can input the segment image frame into the image sub-network 100a, and determine the image feature vector 102a (i.e., the second image feature vector) corresponding to the target video segment through the image sub-network 100a; the server can input the segment audio frame into the audio sub-network 100b, and determine the audio feature vector 102b (i.e., the second audio feature vector) corresponding to the target video segment through the audio sub-network 100b; the server can perform text processing on the segment text information to obtain the text vector to be encoded corresponding to the segment unit text in the segment text information, and input the text vector to be encoded corresponding to the segment unit text into the text sub-network 100c, and determine the text feature vector 102c (i.e., the second text feature vector) corresponding to the target video segment through the text sub-network 100c.

[0260] like Figure 10As shown, the server can input candidate segment image frames into the image sub-network 100d, and determine the image feature vector 103a (i.e., candidate image feature vector) corresponding to the candidate video segment through the image sub-network 100d; the server can input candidate segment audio frames into the audio sub-network 100e, and determine the audio feature vector 103b (i.e., candidate audio feature vector) corresponding to the candidate video segment through the audio sub-network 100e; the server can perform text processing on the candidate segment text information to obtain the candidate text vector to be encoded corresponding to the candidate segment unit text in the candidate segment text information, and input the candidate text vector to be encoded corresponding to the candidate segment unit text into the text sub-network 100f, and determine the text feature vector 103c (i.e., candidate text feature vector) corresponding to the candidate video segment through the text sub-network 100f.

[0261] like Figure 10 As shown, the server can perform feature fusion on image feature vector 102a and image feature vector 103a to obtain an image dimension interactive representation (i.e., fused image feature vector), perform feature fusion on audio feature vector 102b and audio feature vector 103b to obtain an audio dimension interactive representation (i.e., fused audio feature vector), and perform feature fusion on text feature vector 102c and text feature vector 103c to obtain a text dimension interactive representation (i.e., fused text feature vector).

[0262] like Figure 10 As shown, the server can generate the video segment classification type distribution (i.e., second similarity) of the target video segment and the candidate video segment based on the fused image feature vector, fused audio feature vector, and fused text feature vector. Here, the second similarity is the stroke consistency rate between the target video segment and the candidate video segment.

[0263] Understandably, the server can be based on Figure 8 In the corresponding embodiment, the first similarity is obtained from the video clip library, which retrieves video clips that meet the criteria of clip similarity to hot comments related to the target video clip; the server can also be based on... Figure 9 In the corresponding embodiment, the second similarity is obtained from the video clip library, which retrieves video clips that meet the criteria of similarity to hot comments in the target video clip.

[0264] Optionally, the server can also retrieve video segments from the video segment library that satisfy the segment similarity hot comment condition with the target video segment based on the first similarity and the second similarity. Specifically, if there are candidate video segments whose first similarity is greater than the first segment similarity threshold and whose second similarity is greater than the second segment similarity threshold, then these candidate video segments are identified as those satisfying the segment similarity hot comment condition with the target video segment. In other words, if there are candidate video segments with a first similarity greater than the first segment similarity threshold, then these candidate video segments are identified as the first candidate video segments. Furthermore, if there are first candidate video segments with a second similarity greater than the second segment similarity threshold, then these first candidate video segments are identified as the second candidate video segments. These second candidate video segments are the video segments that satisfy the segment similarity hot comment condition with the target video segment.

[0265] Therefore, the embodiments of this application can perform multimodal similarity judgment on the segment image frames, segment audio frames, and segment text information of the target video segment, as well as the candidate segment image frames, candidate segment audio frames, and candidate segment text information of the candidate video segment. A second target network model is used to generate segment similarity between the target video segment and the candidate video segments. Then, based on the segment similarity, similar comment video segments associated with the target video segment are automatically mined from the video segment library. The target video segment and the similar comment video segments share commonalities in content and both belong to the category of trending comment video segments. This can expand the richness of video resources on the video platform and improve the accuracy of the mined similar comment video segments. Based on this, the accuracy of video recommendations can be improved when using similar comment video segments for video recommendation.

[0266] Further, please see Figure 11 , Figure 11 This is a flowchart illustrating a video processing method provided in an embodiment of this application. The method can be executed by a terminal device, a server, or both. The terminal device can be one of the aforementioned... Figure 2 The terminal device in the corresponding embodiment, the server can be as described above. Figure 2 The corresponding embodiment uses server 20a. For ease of understanding, this embodiment uses the method executed by a terminal device as an example for explanation. The video processing method may include at least the following steps S201-S202:

[0267] Step S201: Display the target video in the video playback interface;

[0268] It is understandable that when a user (e.g., object Y) of a terminal device needs to watch a certain video (e.g., the target video), a trigger operation can be performed on the target video in the video recommendation interface of the terminal device. In this way, the terminal device can respond to the trigger operation performed by object Y on the target video, and send a video playback request to the server corresponding to the terminal device based on the video identifier of the target video. After receiving the target video data corresponding to the target video returned by the server, the target video data is played in the video playback interface of the terminal device.

[0269] Step S202: In response to the trigger operation of the similar comment segment function in the video playback interface, display a set of similar comment segments related to the playback progress of the target video in the target area of ​​the video playback interface.

[0270] Specifically, the video segments in the similar comment segment set and the target video segment satisfy the condition of segment similarity and trending comments; the target video segment refers to the video segment indicated by the current playback progress in the target video; both the video segments in the similar comment segment set and the target video segment belong to the trending comment type; and the association between the commentary media data associated with the video segment of the trending comment type and the segment content in the corresponding video segment satisfies the trending comment condition. In other words, the association between the target commentary media data associated with the target video segment and the target segment content in the target video segment satisfies the trending comment condition.

[0271] Understandably, the terminal device can respond to the triggering operation of the similar segment display control with similar comment segment function in the video playback interface, and display a set of similar comment segments related to the playback progress of the target video in the target area of ​​the video playback interface.

[0272] Specifically, when the terminal device responds to a trigger operation on the similar segment display control, it can detect the comment type of the currently playing video segment in the target video. If the currently playing video segment is a target video segment with popular comment types, it will display a set of similar comment segments associated with the target video segment in the target area of ​​the video playback interface. At this time, the terminal device can only display the set of similar comment segments associated with the target video segment.

[0273] For easier understanding, please refer to Figure 12a , Figure 12a This is a schematic diagram illustrating a scenario for displaying a collection of similar comment fragments, provided in an embodiment of this application. For example... Figure 12aThe video playback interfaces 120a and 120b shown can be the video playback interfaces of the terminal device at different times. The video playback interface 120a may include a similar clip display control 121a with a similar comment clip function.

[0274] like Figure 12a As shown, the terminal device can respond to the trigger operation performed by object Y on the similar segment display control 121a, perform comment type detection on the video segment (i.e., video segment O1) to which the playback progress 121b of the target video belongs, and then, when video segment O1 is a video segment with a hot comment type, switch the video playback interface from video playback interface 120a to video playback interface 120b, and display the set of similar comment segments associated with video segment O1 in the target area of ​​video playback interface 120b. The set of similar comment segments associated with video segment O1 may include one or more similar meme video segments, which may specifically include: video segment L1, video segment L2, video segment L3, and video segment L4.

[0275] Optionally, the terminal device can respond to the trigger operation of the similar comment segment function in the video playback interface, and sequentially perform comment type detection on the video segments played in the target video. Furthermore, when it is detected that the currently playing video segment in the target video is a target video segment with a popular comment type, the terminal device can display a set of similar comment segments associated with the target video segment in the target area of ​​the video playback interface.

[0276] Specifically, the terminal device can respond to trigger operations on the similar segment display control with similar comment segment functionality in the video interface. It sequentially detects the comment types of the video segments played in the target video, and then, when a video segment played in the target video is a segment with a popular comment type, it sequentially displays a set of similar comment segments associated with that video segment in the target video playback interface. At this point, the terminal device can prioritize displaying the set of similar comment segments associated with the target video segment.

[0277] Please see again. Figure 12aIf object Y needs to watch a specific video segment from video segments L1, L2, L3, and L4, a trigger operation can be performed on that video segment. In this case, the terminal device can play the video data corresponding to that video segment on the video playback interface 120b. Optionally, if object Y does not need to watch a specific video segment from video segments L1, L2, L3, and L4, a trigger operation can be performed on any location other than the target area. In this case, the terminal device can cancel the display of the similar comment segment set on the video playback interface 120b.

[0278] Furthermore, when the playback progress of the target video in the video playback interface 120b switches from playback progress 121b to playback progress 121c (not shown in the figure), the terminal device can perform comment type detection on the video segment (i.e., video segment O2) to which playback progress 121c (not shown in the figure) belongs. If video segment O2 is a video segment with a popular comment type, the video playback interface is switched from video playback interface 120b to video playback interface 120c (not shown in the figure). In the target area of ​​video playback interface 120c (not shown in the figure), a set of similar comment segments associated with video segment O2 is displayed. This set of similar comment segments associated with video segment O2 may include one or more similar meme video segments, specifically including video segment L5.

[0279] Optionally, when the currently playing video segment in the target video is detected to be a target video segment with a trending comment type, the terminal device can display trending comment prompts with similar comment segments in the video playback interface. Furthermore, in response to a trigger operation on the trending comment prompts, the terminal device can display a set of similar comment segments associated with the target video segment in the target area of ​​the video playback interface.

[0280] Specifically, when the target video is detected to be a video segment with a hot comment type, the terminal device can display a hot comment prompt area with similar comment segment function in the video playback interface, display hot comment prompt information in the hot comment prompt area, and then respond to the trigger operation of the hot comment prompt information in the hot comment prompt area. In the target area of ​​the video playback interface, a set of similar comment segments associated with the target video segment is displayed.

[0281] Understandably, when a terminal device activates the hot comment type segment recognition function, it can sequentially detect the comment type of video segments played in the video playback interface of the target video. Then, when a video segment played in the target video is a video segment with a hot comment type, it will sequentially display hot comment prompts with similar comment segment functions in the video playback interface.

[0282] Specifically, when the terminal device has not enabled the hot comment segment recognition function, it can respond to a trigger operation on the similar segment display control with similar comment segment functionality in the video interface to enable the hot comment segment recognition function. Optionally, when the terminal device has enabled the hot comment segment recognition function, it can respond to a trigger operation on the similar segment display control to disable the hot comment segment recognition function.

[0283] For easier understanding, please refer to Figure 12b , Figure 12b This is a schematic diagram illustrating a scenario for displaying a collection of similar comment fragments, provided in an embodiment of this application. For example... Figure 12b The video playback interfaces 122a, 122b, 122c, and 122d shown can be the video playback interfaces of the terminal device at different times. Among them, video playback interface 122a may include a similar segment display control 123a with similar comment segment functionality.

[0284] like Figure 12b As shown, the terminal device can respond to the trigger operation performed by object Y on the similar segment display control 123a, and sequentially perform comment type detection on the video segments played in the target video. It can be understood that when the target video progresses from the playback progress in video playback interface 122a to the playback progress 123b in video playback interface 122b, the terminal device can perform comment type detection on the video segment (i.e., video segment O1) to which playback progress 123b belongs. Then, if video segment O1 is a video segment with a hot comment type, the video playback interface is switched from video playback interface 122b to video playback interface 122c. In video playback interface 122c, a hot comment prompt message 123c with similar comment segment functionality is displayed. The hot comment prompt message 123c can be "Hot video segment!".

[0285] like Figure 12b As shown, the terminal device can respond to the trigger operation performed by object Y in response to the hot comment prompt information 123c, and switch the video playback interface from video playback interface 122c to video playback interface 122d. In the target area of ​​video playback interface 122d, a set of similar comment segments associated with video segment O1 is displayed.

[0286] It is understandable that video clips in the set of similar comment segments and the target video clip are similar meme videos. For an explanation of similar meme videos, please refer to... Figure 13a and Figure 13b , Figure 13a and Figure 13b This is a schematic diagram of a scene for switching video segments provided in an embodiment of this application.

[0287] like Figure 13a As shown, the video clips belonging to video playback interface 131a and video playback interface 131b are similar meme videos. The similar meme expresses the idea that "one party asks the other party to close the door and leave, but the other party intentionally or unintentionally misunderstands it as just closing the door and not having to leave himself." This similar meme can create a humorous atmosphere.

[0288] like Figure 13b As shown, the video clips belonging to video playback interface 132a and video playback interface 132b are similar meme videos. The similar meme expresses "three years after three years, describing the long time it takes to succeed". Similar video clips can be obtained through this similar meme, thereby providing inspirational content.

[0289] Therefore, this application embodiment can display the target video in the video playback interface, and then respond to the trigger operation of the similar comment segment function in the video playback interface. In the target area of ​​the video playback interface, a set of similar comment segments associated with the target video segment is displayed. The video segments in the similar comment segment set satisfy the condition of similarity to hot comments with the target video segment, and both the video segments in the similar comment segment set and the target video segment belong to the hot comment type (i.e., similar meme video segments). Therefore, this application embodiment can generate different sets of similar comment segments for different video segments in the target video. The video segments in different sets of similar comment segments can have a high similarity to different videos in the target video, improving the accuracy of mining video segments in the similar comment segment set, thereby improving the accuracy of video recommendation. Furthermore, the video segments in the similar comment segment set are obtained after dividing the complete video. Recommending videos using video segments from the complete video can enrich the video resources associated with the target video segment. In addition, the similar meme video segments in this application embodiment can serve as a new video type, enriching the resource formats of video sites, providing basic capabilities for video site interaction and distribution, and improving the viewing experience of the video platform.

[0290] Further, please see Figure 14 , Figure 14This is a schematic diagram of the structure of a video processing device provided in an embodiment of this application. The video processing device 1 may include: a type detection module 11, a type determination module 12, and a set determination module 13.

[0291] The type detection module 11 is used to obtain target video segments from the target video, perform comment type detection on the target video segments based on the target commentary media data associated with the target video segments and the target segment content in the target video segments, and obtain the type detection result;

[0292] The target segment content includes video plot text;

[0293] The type detection module 11 includes: a clause extraction unit 111, a text matching unit 112, a first generation unit 113, and a second generation unit 114;

[0294] Clause extraction unit 111 is used to extract commentary text clauses from target commentary media data associated with the target video segment;

[0295] Clause extraction unit 111 is used to segment the video plot text in the target video segment based on the clause characters, and generate S plot text clauses; S is a positive integer;

[0296] The text matching unit 112 is used to determine the text similarity coefficient between each plot text clause and the commentary text clause in S plot text clauses, and to generate the text hotspot matching degree between each plot text clause and the commentary text clause based on the text similarity coefficient, the number of interactions of the commentary text clause and the text length of the commentary text clause;

[0297] The text matching unit 112 is used to merge the text hotspot matching scores corresponding to each plot text clause to obtain the merged text hotspot matching score.

[0298] The first generation unit 113 is used to generate a type detection result indicating that the relationship between the target commentary media data and the target segment content meets the hotspot commentary conditions if the hotspot matching degree of the merged text is greater than the text matching degree threshold.

[0299] The second generation unit 114 is used to generate a type detection result indicating that the association between the target commentary media data and the target segment content does not meet the hotspot commentary conditions if the hotspot matching degree of the merged text is less than or equal to the text matching degree threshold.

[0300] The specific implementation methods of the clause extraction unit 111, text matching unit 112, first generation unit 113, and second generation unit 114 can be found above. Figure 5The descriptions of steps S11-S16 in the corresponding embodiments will not be repeated here.

[0301] Specifically, the type detection module 11 is further used to extract text information associated with the target video segment from the subtitle file, and to use the extracted text information as video plot text associated with the target video segment; or,

[0302] The type detection module 11 is also specifically used to perform speech recognition on the target video segment, and to use the recognized speech text as video plot text associated with the target video segment; or,

[0303] The type detection module 11 is also specifically used to perform semantic recognition on the action content in the target video segment, and to use the recognized semantic text as video plot text associated with the target video segment; or,

[0304] The type detection module 11 is also specifically used to perform character recognition on the target video segment and use the recognized character text as video plot text associated with the target video segment.

[0305] The target segment content includes video plot text, segment image frames, and segment audio frames;

[0306] The type detection module 11 includes: a data input unit 115, a model detection unit 116, a third generation unit 117, and a fourth generation unit 118;

[0307] The data input unit 115 is used to input the target commentary media data associated with the target video segment and the video plot text in the target video segment as the segment text information of the target video segment;

[0308] Data input unit 115 is used to input the segment image frames, segment audio frames, and segment text information of the target video segment into the first target network model;

[0309] The model detection unit 116 is used to detect the comment type of the target video segment through the first target network model and output the segment hotspot matching degree corresponding to the target video segment.

[0310] Specifically, the model detection unit 116 is used to generate the first image feature vector corresponding to the target video segment through the first target network model;

[0311] The model detection unit 116 is specifically used to generate the first audio feature vector corresponding to the target video segment through the first target network model;

[0312] The model detection unit 116 is specifically used to generate the first text feature vector corresponding to the target video segment through the first target network model;

[0313] The model detection unit 116 is specifically used to perform feature fusion on the first image feature vector, the first audio feature vector and the first text feature vector to obtain the first fused feature vector corresponding to the target video segment, and input the first fused feature vector into the first fully connected network in the first target network model. The first fully connected network performs fully connected processing on the first fused feature vector to obtain the first target feature vector corresponding to the target video segment.

[0314] The model detection unit 116 is specifically used to input the first target feature vector into the first classifier in the first target network model, and the first classifier outputs the segment hotspot matching degree corresponding to the target video segment.

[0315] The first target network model includes a text subnetwork;

[0316] The model detection unit 116 is specifically used to extract fragment unit text from fragment text information in the text sub-network, embed the fragment unit text into text, and obtain the word segmentation vector corresponding to the fragment unit text.

[0317] The model detection unit 116 is specifically used to determine the text position of the unit text in the fragment text information, perform position embedding on the text position, and obtain the position vector corresponding to the unit text in the fragment.

[0318] The model detection unit 116 is specifically used to determine the segmentation vector corresponding to the fragment unit text, and to fuse the word segmentation vector, the position vector, and the segmentation vector corresponding to the fragment unit text to obtain the text vector to be encoded corresponding to the fragment unit text.

[0319] The model detection unit 116 is specifically used to perform attention processing on the text vector to be encoded corresponding to the fragment unit text to obtain the encoded text vector corresponding to the fragment unit text, and to determine the first text feature vector corresponding to the target video fragment based on the encoded text vector corresponding to the fragment unit text.

[0320] The third generation unit 117 is used to generate a type detection result indicating that the relationship between the target commentary media data and the target segment content meets the hot commentary conditions if the segment hotspot matching degree is greater than the segment matching degree threshold.

[0321] The fourth generation unit 118 is used to generate a type detection result indicating that the relationship between the target commentary media data and the target segment content does not meet the hotspot commentary condition if the segment hotspot matching degree is less than or equal to the segment matching degree threshold.

[0322] The specific implementation methods of the data input unit 115, the model detection unit 116, the third generation unit 117, and the fourth generation unit 118 can be found in the above description. Figure 6 The descriptions of steps S21-S25 in the corresponding embodiments will not be repeated here.

[0323] The type determination module 12 is used to determine the target video segment as a video segment with a hot comment type if the type detection result indicates that the correlation between the target commentary media data and the target segment content meets the hot comment conditions;

[0324] The set determination module 13 is used to obtain video segments from the video segment library that meet the segment similarity hot comment conditions with the target video segment, and determine the video segments that meet the segment similarity hot comment conditions as a set of similar comment segments associated with the target video segment.

[0325] The target segment content includes video plot text;

[0326] The set determination module 13 includes: a word segmentation determination unit 131, a segment detection unit 132, and a segment determination unit 133;

[0327] The word segmentation determination unit 131 is used to use the target commentary media data and the video plot text in the target video segment as the segment text information of the target video segment;

[0328] The word segmentation determination unit 131 is used to perform word segmentation processing on the fragment text information, generate fragment text segments corresponding to the fragment text information, and filter fragment key segments from the fragment text segments.

[0329] The segment detection unit 132 is used to obtain a set of video segments with hot comment types from the video segment library, and retrieve candidate video segments containing segment key words from the video segment set based on segment key word segmentation;

[0330] Among them, the number of key word segments in the segment must be at least two;

[0331] The segment detection unit 132 is specifically used to retrieve a set of video segments containing at least two segment keywords from the inverted index information corresponding to the video segments in the video segment set, based on at least two segment keywords.

[0332] The segment detection unit 132 is specifically used to generate candidate video segments containing segment key words based on video segments in at least two retrieved video segment sets;

[0333] The segment detection unit 132 is also specifically used to perform word segmentation on the index segment text information of the video segments in the video segment set, and generate the index segment text segment corresponding to the index segment text information;

[0334] The segment detection unit 132 is also specifically used to filter key words of index segments from the text segmentation of index segments, and to construct inverted index information corresponding to video segments in the video segment set based on the key words of index segments.

[0335] The segment detection unit 132 is specifically used to retrieve an initial video segment containing segment key words from the video segment set based on segment key word segmentation.

[0336] The segment detection unit 132 is specifically used to generate a text similarity between the target video segment and the initial video segment based on the segment text information and the initial segment text information of the initial video segment;

[0337] The segment detection unit 132 is specifically used to filter initial video segments from initial video segments whose text similarity is greater than a similarity threshold, and to determine the filtered initial video segments as candidate video segments retrieved from the video segment set.

[0338] The segment determination unit 133 is used to determine the segment similarity between the target video segment and the candidate video segments. If there are candidate video segments with a segment similarity greater than the segment similarity threshold, then the candidate video segments with a segment similarity greater than the segment similarity threshold are determined as video segments that meet the segment similarity hot comment conditions with the target video segment.

[0339] The target segment content includes segment image frames and segment audio frames; the candidate segment content in the candidate video segment includes candidate segment image frames, candidate segment audio frames, and candidate video plot text.

[0340] The segment determination unit 133 is specifically used to take the candidate commentary media data associated with the candidate video segment and the candidate video plot text in the candidate video segment as the candidate segment text information of the candidate video segment;

[0341] The segment determination unit 133 is specifically used to input the segment image frame, segment audio frame, segment text information, candidate segment image frame, candidate segment audio frame and candidate segment text information into the second target network model;

[0342] The segment determination unit 133 is specifically used in the second target network model to determine the second image feature vector, the second audio feature vector, and the second text feature vector corresponding to the target video segment, and to determine the candidate image feature vector, the candidate audio feature vector, and the candidate text feature vector corresponding to the candidate video segment;

[0343] The segment determination unit 133 is specifically used to perform feature fusion on the second image feature vector and the candidate image feature vector to obtain a fused image feature vector, to perform feature fusion on the second audio feature vector and the candidate audio feature vector to obtain a fused audio feature vector, and to perform feature fusion on the second text feature vector and the candidate text feature vector to obtain a fused text feature vector.

[0344] The segment determination unit 133 is specifically used to perform feature fusion on the fused image feature vector, fused audio feature vector and fused text feature vector to obtain a second fused feature vector, and input the second fused feature vector into the second fully connected network in the second target network model, and the second fully connected network performs fully connected processing on the second fused feature vector to obtain the second target feature vector;

[0345] The segment determination unit 133 is specifically used to input the second target feature vector into the second classifier in the second target network model, and the second classifier outputs the second similarity, which is used as the segment similarity between the target video segment and the candidate video segment.

[0346] The output of the first classifier also includes a scene type vector corresponding to the target video segment; the scene type vector includes the first scene probability corresponding to N scene type labels; N is a positive integer;

[0347] The set determination module 13 is specifically used to obtain a set of video clips with hot comment types from the video clip library, and to retrieve candidate video clips associated with the clip text information from the video clip set;

[0348] The set determination module 13 is specifically used to obtain the candidate scene type vector corresponding to the candidate video segment; the candidate scene type vector includes the second scene probability corresponding to N scene type labels respectively;

[0349] The set determination module 13 is specifically used to perform a first operation on the first scene probability and the second scene probability corresponding to the same scene type label in the scene type vector and the candidate scene type vector, so as to obtain the scene matching probability corresponding to each scene type label.

[0350] The set determination module 13 is specifically used to perform a second operation on the scene matching probability corresponding to each scene type label to obtain the first similarity, and use the first similarity as the segment similarity between the target video segment and the candidate video segment;

[0351] The set determination module 13 is specifically used to determine, if there are candidate video segments with a similarity greater than the segment similarity threshold, the candidate video segments with a similarity greater than the segment similarity threshold as video segments that meet the segment similarity hot comment conditions with the target video segment.

[0352] The specific implementation methods of the type detection module 11, type determination module 12, and set determination module 13 can be found in the above description. Figure 3 In the corresponding embodiment, steps S101-S103, Figure 5 In the corresponding embodiment, steps S11-S16, Figure 6 In the corresponding embodiment, steps S21-S25, Figure 8 In the corresponding embodiments, steps S31-S35, and Figure 9 The descriptions of steps S41-S48 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0353] Further, please see Figure 15 , Figure 15 This is a schematic diagram of the structure of a video processing device provided in an embodiment of this application. The video processing device 2 may include: a video display module 21 and a collection display module 22.

[0354] The video display module 21 is used to display the target video in the video playback interface;

[0355] The collection display module 22 is used to respond to the trigger operation of the similar comment segment function in the video playback interface. In the target area of ​​the video playback interface, it displays a collection of similar comment segments associated with the playback progress of the target video. The video segments in the similar comment segment collection and the target video segment meet the condition of segment similarity and hot comment. The target video segment refers to the video segment indicated by the current playback progress in the target video. The video segments in the similar comment segment collection and the target video segment are both video segments of the hot comment type. The association between the commentary media data associated with the video segment of the hot comment type and the segment content in the video segment satisfies the condition of hot comment.

[0356] The collection display module 22 is specifically used to respond to the trigger operation of the similar segment display control with similar comment segment function in the video playback interface, and to display a collection of similar comment segments related to the playback progress of the target video in the target area of ​​the video playback interface.

[0357] Among them, the collection display module 22 is specifically used to respond to the trigger operation of the similar comment segment function in the video playback interface, and sequentially detect the comment type of the video segments played in the target video;

[0358] The collection display module 22 is specifically used to display a collection of similar comment segments associated with the target video segment in the target area of ​​the video playback interface when it is detected that the currently playing video segment in the target video is a target video segment with hot comment type.

[0359] Among them, the collection display module 22 is specifically used to display hot comment prompt information with similar comment segment function in the video playback interface when it is detected that the currently playing video segment in the target video is a target video segment with hot comment type;

[0360] The collection display module 22 is specifically used to respond to the triggered operation of hot comment prompts and to display a collection of similar comment segments associated with the target video segment in the target area of ​​the video playback interface.

[0361] For details on the implementation of the video display module 21 and the collection display module 22, please refer to the above. Figure 11 The descriptions of steps S201-S202 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0362] Further, please see Figure 16 , Figure 16 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may be a terminal device or a server. Figure 16As shown, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. In some embodiments, the user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. Optionally, the network interface 1004 may include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the processor 1001. Figure 16 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0363] In such Figure 16 In the computer device 1000 shown, the network interface 1004 provides network communication functions; the user interface 1003 is mainly used to provide an input interface for users; and the processor 1001 can be used to call the device control application stored in the memory 1005.

[0364] It should be understood that the computer device 1000 described in the embodiments of this application can execute the foregoing text. Figure 3 , Figure 5 , Figure 6 , Figure 8 , Figure 9 and Figure 11 The description of the video processing method in the corresponding embodiments can also be performed as described above. Figure 14 In the corresponding embodiments, video processing device 1 and Figure 15 The description of the video processing device 2 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0365] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned video processing apparatus 1 and video processing apparatus 2. When the processor executes the computer program, it can execute the aforementioned... Figure 3 , Figure 5 , Figure 6 , Figure 8 , Figure 9 and Figure 11The video processing method described in the corresponding embodiments is already known and will therefore not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application.

[0366] Furthermore, it should be noted that this application also provides a computer program product, which may include a computer program that can be stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium, and the processor can execute the computer program, causing the computer device to perform the aforementioned... Figure 3 , Figure 5 , Figure 6 , Figure 8 , Figure 9 and Figure 11 The video processing method described in the corresponding embodiments is already known and will therefore not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer program product embodiments related to this application, please refer to the description of the method embodiments of this application.

[0367] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0368] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method of video processing, the method comprising: The method comprises the following steps: obtaining a target video segment from a target video, performing comment type detection on the target video segment according to target comment media data associated with the target video segment and target segment content in the target video segment, and obtaining a type detection result; if the type detection result indicates that the association between the target comment media data and the target segment content meets a hot comment condition, determining that the target video segment is a video segment with a hot comment type; obtaining a video segment that meets a segment similar hot comment condition with the target video segment from a video segment library, and determining the video segment that meets the segment similar hot comment condition as a similar comment segment set associated with the target video segment; if the target segment content includes video plot text, the comment type detection on the target video segment according to the target comment media data associated with the target video segment and the target segment content in the target video segment comprises the following steps: extracting a comment text clause from the target comment media data associated with the target video segment; performing text segmentation on the video plot text in the target video segment based on segmented characters to generate S plot text clauses; S is a positive integer; determining a text similarity coefficient between each plot text clause in the S plot text clauses and the comment text clause, and generating a text hot match degree between each plot text clause and the comment text clause according to the text similarity coefficient, the interaction times of the comment text clause, and the text length of the comment text clause; merging the text hot match degrees corresponding to each plot text clause to obtain a merged text hot match degree; if the merged text hot match degree is greater than a text match degree threshold, generating a type detection result indicating that the association between the target comment media data and the target segment content meets the hot comment condition.

2. The method of claim 1, wherein, The method further comprises: if the merged text hot match degree is less than or equal to the text match degree threshold, generating a type detection result indicating that the association between the target comment media data and the target segment content does not meet the hot comment condition.

3. The method of claim 1, wherein, The method further comprises: extracting text information associated with the target video segment from a subtitle file, and taking the extracted text information as the video plot text associated with the target video segment; or performing speech recognition on the target video segment, and taking the recognized speech text as the video plot text associated with the target video segment; or performing semantic recognition on the action content in the target video segment, and taking the recognized semantic text as the video plot text associated with the target video segment; or performing character recognition on the target video segment, and taking the recognized character text as the video plot text associated with the target video segment.

4. The method of claim 1, wherein, If the target segment content includes video episode text, segment image frames and segment audio frames, the comment type detection on the target video segment according to the target comment media data associated with the target video segment and the target segment content in the target video segment is performed to obtain a type detection result, including: inputting the target comment media data associated with the target video segment and the video episode text in the target video segment as segment text information of the target video segment; inputting the segment image frames in the target video segment, the segment audio frames in the target video segment and the segment text information into a first target network model; performing comment type detection on the target video segment by the first target network model, and outputting a segment hot spot matching degree corresponding to the target video segment; if the segment hot spot matching degree is greater than a segment matching degree threshold, generating a type detection result indicating that the association between the target comment media data and the target segment content satisfies a hot spot comment condition; if the segment hot spot matching degree is less than or equal to the segment matching degree threshold, generating a type detection result indicating that the association between the target comment media data and the target segment content does not satisfy the hot spot comment condition.

5. The method of claim 4, wherein, The comment type detection on the target video segment by the first target network model and the output of the segment hot spot matching degree corresponding to the target video segment include: generating a first image feature vector corresponding to the target video segment by the first target network model; generating a first audio feature vector corresponding to the target video segment by the first target network model; generating a first text feature vector corresponding to the target video segment by the first target network model; performing feature fusion on the first image feature vector, the first audio feature vector and the first text feature vector to obtain a first fusion feature vector corresponding to the target video segment, inputting the first fusion feature vector into a first full connection network in the first target network model, performing full connection processing on the first fusion feature vector by the first full connection network to obtain a first target feature vector corresponding to the target video segment; inputting the first target feature vector into a first classifier in the first target network model, and outputting a segment hot spot matching degree corresponding to the target video segment by the first classifier.

6. The method of claim 5, wherein, The first target network model includes a text sub-network; The generation of the first text feature vector corresponding to the target video segment by the first target network model includes: extracting a segment unit text from the segment text information in the text sub-network, and performing text embedding on the segment unit text to obtain a word segmentation vector corresponding to the segment unit text; determining a text position of the segment unit text in the segment text information, and performing position embedding on the text position to obtain a position vector corresponding to the segment unit text; determine a segmentation vector corresponding to the segment unit text, perform feature fusion on the word segmentation vector corresponding to the segment unit text, the position vector corresponding to the segment unit text, and the segmentation vector corresponding to the segment unit text to obtain a to-be-encoded text vector corresponding to the segment unit text; perform attention processing on the to-be-encoded text vector corresponding to the segment unit text to obtain an encoded text vector corresponding to the segment unit text, and determine a first text feature vector corresponding to the target video segment based on the encoded text vector corresponding to the segment unit text.

7. The method of claim 1, wherein, If the target segment content includes video plot text, the obtaining, from a video segment library, a video segment that meets a segment similar hot comment condition with the target video segment includes: taking the target comment media data and the video plot text in the target video segment as segment text information of the target video segment; performing word segmentation processing on the segment text information to generate segment text word segmentation corresponding to the segment text information, and screening segment key word segmentation from the segment text word segmentation; obtaining a video segment set with the hot comment type from a video segment library, and retrieving a candidate video segment containing the segment key word segmentation from the video segment set according to the segment key word segmentation; determining a segment similarity between the target video segment and the candidate video segment, and determining a candidate video segment greater than a segment similarity threshold as a video segment that meets a segment similar hot comment condition with the target video segment if the segment similarity is greater than the segment similarity threshold.

8. The method of claim 7, wherein, The number of segment key word segmentations is at least two. The retrieving a candidate video segment containing the segment key word segmentation from the video segment set according to the segment key word segmentation includes: retrieving a retrieval video segment set containing at least two segment key word segmentations from inverted index information corresponding to video segments in the video segment set according to the at least two segment key word segmentations; generating a candidate video segment containing the segment key word segmentation according to video segments in at least two retrieval video segment sets; The method further includes: performing word segmentation processing on index segment text information of video segments in the video segment set to generate index segment text word segmentation corresponding to the index segment text information; screening index segment key word segmentation from the index segment text word segmentation, and constructing the inverted index information corresponding to the video segments in the video segment set based on the index segment key word segmentation.

9. The method of claim 7, wherein, The retrieving a candidate video segment containing the segment key word segmentation from the video segment set according to the segment key word segmentation includes: retrieving an initial video segment containing the segment key word segmentation from the video segment set according to the segment key word segmentation; generating a text similarity between the target video segment and the initial video segment according to the segment text information and initial segment text information of the initial video segment; and screening the initial video clips from the initial video clips, wherein the text similarity is greater than a similarity threshold, and determining the screened initial video clips as candidate video clips retrieved from the video clip set.

10. The method of claim 5, wherein, The output of the first classifier further includes a scene type vector corresponding to the target video clip; the scene type vector includes first scene probabilities corresponding to N scene type labels respectively; The N is a positive integer; The method further includes: retrieving a video clip set having the hot comment type from a video clip library, and retrieving candidate video clips associated with the clip text information from the video clip set; retrieving a video clip set having the hot comment type from a video clip library, and retrieving candidate video clips associated with the clip text information from the video clip set; retrieving a video clip set having the hot comment type from a video clip library, and retrieving candidate video clips associated with the clip text information from the video clip set; performing first operation processing on the first scene probabilities and the second scene probabilities corresponding to the same scene type labels in the scene type vector and the candidate scene type vector to obtain scene matching probabilities corresponding to each scene type label respectively; performing second operation processing on the scene matching probabilities corresponding to each scene type label respectively to obtain a first similarity, and taking the first similarity as a clip similarity between the target video clip and the candidate video clip; 11. The method of claim 7, wherein, If there is a candidate video clip greater than a clip similarity threshold in the clip similarity, the candidate video clip greater than the clip similarity threshold is determined as a video clip satisfying the clip similarity hot comment condition with the target video clip. If the target clip content further includes clip image frames and clip audio frames, and candidate clip content in the candidate video clip includes candidate clip image frames, candidate clip audio frames, and candidate video plot text, the method further includes: taking the candidate comment media data associated with the candidate video clip and the candidate video plot text in the candidate video clip as candidate clip text information of the candidate video clip; inputting the clip image frames, the clip audio frames, the clip text information, the candidate clip image frames, the candidate clip audio frames, and the candidate clip text information into a second target network model; In the second target network model, a second image feature vector corresponding to the target video clip, a second audio feature vector corresponding to the target video clip, and a second text feature vector corresponding to the target video clip are determined, and a candidate image feature vector corresponding to the candidate video clip, a candidate audio feature vector corresponding to the candidate video clip, and a candidate text feature vector corresponding to the candidate video clip are determined. perform feature fusion on the second image feature vector, the second audio feature vector and the second text feature vector to obtain a second fusion feature vector, input the second fusion feature vector into a second full connection network in the second target network model, perform full connection processing on the second fusion feature vector by the second full connection network to obtain a second target feature vector, and input the second target feature vector into a second classifier in the second target network model to output a second similarity, wherein the second similarity is taken as a segment similarity between the target video segment and the candidate video segment. The method comprises the following steps: displaying a target video in a video playing interface; 12. A method for video processing, comprising: in response to a triggering operation of a similar comment segment function in the video playing interface, displaying a similar comment segment set associated with a playing progress of the target video in a target region of the video playing interface; a video segment in the similar comment segment set meets a segment similarity hot comment condition with a target video segment; the target video segment refers to a video segment indicated by a current playing progress in the target video; the video segment in the similar comment segment set and the target video segment are both video segments of a hot comment type; an association between comment media data associated with the video segment of the hot comment type and segment content in the video segment meets a hot comment condition; when a merged text hot comment matching degree is greater than a text matching degree threshold, an association between target comment media data associated with the target video segment and target segment content in the target video segment meets the hot comment condition; the merged text hot comment matching degree is obtained by performing merging processing on text hot comment matching degrees corresponding to S scenario text clauses; the S scenario text clauses are obtained by performing text segmentation on video scenario text in the target segment content based on sentence characters; S is a positive integer; the text hot comment matching degrees corresponding to the S scenario text clauses are generated according to a text similarity coefficient between each scenario text clause and a comment text clause in the target comment media data, an interaction frequency of the comment text clause and a text length of the comment text clause. The method comprises the following steps: in response to a triggering operation of a similar comment segment function in the video playing interface, displaying a similar comment segment set associated with a playing progress of the target video in a target region of the video playing interface; a video segment in the similar comment segment set meets a segment similarity hot comment condition with a target video segment; the target video segment refers to a video segment indicated by a current playing progress in the target video; the video segment in the similar comment segment set and the target video segment are both video segments of a hot comment type; an association between comment media data associated with the video segment of the hot comment type and segment content in the video segment meets a hot comment condition; when a merged text hot comment matching degree is greater than a text matching degree threshold, an association between target comment media data associated with the target video segment and target segment content in the target video segment meets the hot comment condition; the merged text hot comment matching degree is obtained by performing merging processing on text hot comment matching degrees corresponding to S scenario text clauses; the S scenario text clauses are obtained by performing text segmentation on video scenario text in the target segment content based on sentence characters; S is a positive integer; the text hot comment matching degrees corresponding to the S scenario text clauses are generated according to a text similarity coefficient between each scenario text clause and a comment text clause in the target comment media data, an interaction frequency of the comment text clause and a text length of the comment text clause.

13. The method of claim 12, wherein, ​ In response to a triggering operation on a similar segment display control with a similar comment segment function in the video playing interface, a set of similar comment segments associated with a playing progress of the target video is displayed in a target region of the video playing interface.

14. The method of claim 12, wherein, The displaying, in response to the triggering operation on the similar comment segment function in the video playing interface, of the set of similar comment segments associated with the playing progress of the target video in the target region of the video playing interface includes: In response to the triggering operation on the similar comment segment function in the video playing interface, a comment type detection is performed on video segments played in the target video in sequence. When it is detected that a currently played video segment in the target video is a target video segment with the hot comment type, a set of similar comment segments associated with the target video segment is displayed in a target region of the video playing interface.

15. The method of claim 12, wherein, The displaying, in response to the triggering operation on the similar comment segment function in the video playing interface, of the set of similar comment segments associated with the playing progress of the target video in the target region of the video playing interface includes: When it is detected that a currently played video segment in the target video is a target video segment with the hot comment type, a hot comment prompt information with a similar comment segment function is displayed in the video playing interface. In response to a triggering operation on the hot comment prompt information, a set of similar comment segments associated with the target video segment is displayed in a target region of the video playing interface.

16. A video processing apparatus, comprising: The method comprises: The type detection module is configured to obtain a target video segment from a target video, and perform a comment type detection on the target video segment according to target comment media data associated with the target video segment and target segment content in the target video segment, to obtain a type detection result. The type determination module is configured to determine the target video segment as a video segment with a hot comment type if the type detection result indicates that an association between the target comment media data and the target segment content satisfies a hot comment condition. The set determination module is configured to obtain, from a video segment library, a video segment that satisfies a segment similar hot comment condition with the target video segment, and determine the video segment that satisfies the segment similar hot comment condition as a set of similar comment segments associated with the target video segment. The target segment content comprises video plot text. The type detection module comprises: The clause extraction unit is configured to extract a comment text clause from the target comment media data associated with the target video segment. The clause extraction unit is configured to perform text segmentation on the video plot text in the target video segment based on segmentation characters, to generate S plot text clauses; S is a positive integer. The clause extraction unit is configured to perform text segmentation on the video plot text in the target video segment based on segmentation characters, to generate S plot text clauses; S is a positive integer. The text matching unit is configured to determine a text similarity coefficient between each of the S plot text clauses and the comment text clause, and generate a text hot spot matching degree between each of the S plot text clauses and the comment text clause according to the text similarity coefficient, the interaction times of the comment text clause, and the text length of the comment text clause. The text matching unit is configured to combine the text hot spot matching degrees corresponding to each of the S plot text clauses to obtain a combined text hot spot matching degree. The first generation unit is configured to generate a type detection result indicating that the association relationship between the target comment media data and the target segment content satisfies the hot spot comment condition, if the combined text hot spot matching degree is greater than the text matching degree threshold.

17. A video processing apparatus, comprising: The video display module is configured to display a target video in a video playing interface. The collection display module is configured to display, in a target region of the video playing interface, a similar comment segment collection associated with a playing progress of the target video, in response to a triggering operation of a similar comment segment function in the video playing interface. A video segment in the similar comment segment collection satisfies a segment similarity hot spot comment condition with a target video segment. The target video segment refers to a video segment indicated by the current playing progress in the target video. The video segment in the similar comment segment collection and the target video segment both belong to a video segment of a hot spot comment type. An association relationship between comment media data associated with the video segment of the hot spot comment type and segment content in the video segment satisfies a hot spot comment condition. When a combined text hot spot matching degree is greater than a text matching degree threshold, an association relationship between target comment media data associated with the target video segment and target segment content in the target video segment satisfies a hot spot comment condition. The combined text hot spot matching degree is obtained by combining the text hot spot matching degrees corresponding to the S plot text clauses. The S plot text clauses are obtained by text segmentation of video plot text in the target segment content based on sentence characters. S is a positive integer. The text hot spot matching degrees corresponding to the S plot text clauses are generated according to a text similarity coefficient between each of the plot text clauses and a comment text clause in the target comment media data, interaction times of the comment text clause, and a text length of the comment text clause. The processor and the memory are included.

18. A computer device, comprising: The memory is configured to store a computer program, and the processor is configured to invoke the computer program to enable the computer device to perform the method in any one of claims 1-15. The computer readable storage medium stores a computer program adapted to be loaded and executed by the processor to enable the computer device having the processor to perform the method in any one of claims 1-15. ​ 19. A computer-readable storage medium, characterized in that, ​ 20. A computer program product, characterised in that, The computer program product comprises a computer program stored in a computer readable storage medium and adapted to be read and executed by a processor to cause a computer device having the processor to perform the method of any of claims 1-15.

Images