Video processing method and device, equipment, storage medium and program product

Abstract

The invention discloses a video processing method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a plurality of sub-lens segments in a target video; obtaining modal feature information of a plurality of modals of each sub-lens section in the plurality of sub-lens sections; on the basis of the modal feature information of the multiple modals of every two adjacent sub-lens segments in the sub-lens segments, obtaining the relevance of every two adjacent sub-lens segments in the sub-lens segments in each of the multiple modals; and obtaining plot demarcation points of the target video based on the relevance of every two adjacent sub-lens segments in each sub-lens segment in each of the multiple modes.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] In web applications, many video playback applications provide videos with storyline dividing points so that users can understand the location of each storyline in the video while watching it.

[0003] In related technologies, video providers typically use manual methods to mark the plot dividing points in the videos they provide. For example, before publishing the video online, the video provider manually determines the starting point of each storyline in the video. After the video is published, the starting point of each storyline is displayed on the playback progress bar of the video playback interface on the user's side, so that the user can understand the position of each storyline in the video.

[0004] However, among the aforementioned related technologies, the method of manually determining the dividing points of storylines in videos requires a large amount of human resources, and the accuracy is limited by the personal subjective feelings of the annotators. Therefore, the efficiency and accuracy of determining the dividing points of storylines in videos are relatively low. Summary of the Invention

[0005] This application provides a video processing method, apparatus, device, storage medium, and program product, which can improve the efficiency and accuracy of determining the dividing points of storylines in videos. The technical solution is as follows:

[0006] On the one hand, a video processing method is provided, the method comprising:

[0007] Acquire multiple scene segments from the target video; each scene segment consists of at least two consecutive video image frames, corresponding to the same shot.

[0008] Obtain modal feature information of multiple modalities for each of the multiple scene segments;

[0009] Based on the modal feature information of each pair of adjacent storyboard segments in each storyboard segment, the correlation of each pair of adjacent storyboard segments in each of the multiple modalities is obtained. The correlation is used to indicate whether the two adjacent storyboard segments belong to the same storyline.

[0010] Based on the correlation between each pair of adjacent story segments in each of the multiple modalities, the plot dividing point of the target video is obtained; the plot dividing point is the dividing point between different storylines in the target video.

[0011] On the other hand, a video processing apparatus is provided, the apparatus comprising:

[0012] The segment acquisition module is used to acquire multiple scene segments from the target video; the scene segment is composed of at least two consecutive video image frames, corresponding to the same shot.

[0013] The modal feature acquisition module is used to acquire modal feature information of multiple modalities for each of the multiple scene segments;

[0014] The correlation acquisition module is used to acquire the correlation of each pair of adjacent story segments in each of the multiple modal features based on the modal feature information of each pair of adjacent story segments in each of the multiple modalities. The correlation is used to indicate whether the two adjacent story segments belong to the same storyline.

[0015] The plot dividing point acquisition module is used to acquire the plot dividing point of the target video based on the correlation between each two adjacent plot segments in each of the multiple modalities; the plot dividing point is the dividing point between different storylines in the target video.

[0016] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the video processing method as described in the above embodiments of this application.

[0017] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, the computer program being loaded and executed by a processor to implement the video processing method as described in the above embodiments of this application.

[0018] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the video processing methods described in the above embodiments.

[0019] The beneficial effects of the technical solutions provided in this application include at least the following:

[0020] For modal feature information of each segment in multiple scenes of a target video, the correlation between each pair of adjacent segments in each modality can be obtained. Then, based on the correlation between each pair of adjacent segments in each modality, the plot dividing point of the target video can be obtained. On the one hand, it can automatically integrate information from multiple modalities to determine the dividing point between storylines in the video. On the other hand, when information from one or more modalities is missing at two adjacent segments, it can also determine whether the two adjacent segments belong to the same storyline through modal feature information from other modalities. This can improve the efficiency and accuracy of determining the dividing point of storylines in the video. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a system configuration diagram of a video processing system involved in this application;

[0023] Figure 2 This is a flowchart illustrating a video processing method according to an exemplary embodiment;

[0024] Figure 3 This is a flowchart illustrating a video processing method according to an exemplary embodiment;

[0025] Figure 4 This is a technical flowchart of the video processing solution involved in this application;

[0026] Figure 5 It is a schematic diagram of multimodal data, bullet comments, and scene division;

[0027] Figure 6 This is a schematic diagram of the Swin-Transformer involved in this application;

[0028] Figure 7 This is a schematic diagram of the improved Swin-Transformer involved in this application;

[0029] Figure 8 This is a structural diagram of the Patch Partition module involved in this application;

[0030] Figure 9 These are structural diagrams of the two ST blocks involved in this application;

[0031] Figure 10 This is a schematic diagram of the relevance curve involved in this application;

[0032] Figure 11 This is an architecture diagram of the multimodal and bullet screen data segmentation judgment module involved in this application;

[0033] Figure 12 This is a schematic diagram of the background music modal segmentation involved in this application;

[0034] Figure 13 This is a network structure diagram of the background music and human voice categories involved in this application;

[0035] Figure 14 This is a flowchart of the judgment module based on human voice lines and subtitle lines involved in this application;

[0036] Figure 15 This is a schematic diagram illustrating the judgment based on human voice dialogue involved in this application;

[0037] Figure 16 This is a schematic diagram illustrating the judgment based on subtitle dialogue involved in this application;

[0038] Figure 17 This is a schematic diagram illustrating the role-based judgment involved in this application;

[0039] Figure 18 This is a network structure diagram of the scene classification involved in this application;

[0040] Figure 19 This is a network structure diagram of the scene classification involved in this application;

[0041] Figure 20 This is a schematic diagram of the bullet screen data involved in this application;

[0042] Figure 21 This is a schematic diagram of fusion regularization involved in this application;

[0043] Figure 22 This is a structural block diagram of a video processing apparatus provided in an illustrative embodiment of this application;

[0044] Figure 23 This is a schematic diagram of the structure of a server provided in an exemplary embodiment of this application. Detailed Implementation

[0045] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0046] Before describing the various embodiments shown in this application, several concepts involved in this application will be introduced first.

[0047] 1) AI (Artificial Intelligence)

[0048] AI (Artificial Intelligence) is the theory, methods, technology, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. Artificial intelligence technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0049] 2) Computer Vision (CV) technology

[0050] Computer vision is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, detecting, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include video processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, virtual reality, augmented reality, and simultaneous localization and mapping (SLAM).

[0051] 3) ML (Machine Learning)

[0052] Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0053] 4) Conv: Convolutional Neural Network (CNN)

[0054] A CNN is a feedforward neural network whose artificial neurons can respond to surrounding units within a certain coverage area, making it excellent for large-scale image processing. A convolutional neural network consists of one or more convolutional layers and a fully connected layer at the top (corresponding to a classic neural network), and also includes associated weights and pooling layers.

[0055] 5) Spatial similarity

[0056] Spatial similarity, also known as spatial distance, measures the similarity between two vectors by measuring the cosine of the angle between them. The cosine of a 0-degree angle in space is 1, and the cosine of any other angle is no greater than 1; its minimum value is -1. Therefore, the cosine of the angle between two vectors determines their spatial similarity, that is, the degree of coincidence of the spatial angle and direction between the two vectors. When two vectors have the same direction, the similarity is high, and the cosine similarity value is 1; when the spatial angle between two vectors is 90°, the similarity is low, and the cosine similarity value is 0; when two vectors point in completely opposite directions and are completely dissimilar, the cosine similarity value is -1. This result is independent of the length of the vectors, only related to their direction. Cosine similarity is usually used in positive space, so the given value is between 0 and 1.

[0057] 6) Storyboard

[0058] The term "storyboard" as used in this application can refer to a micro-unit of the entire video. Storyboard switching refers to the switching between two storyboards, which is equivalent to the switching of scenes or shots in a film or television drama. All content within each storyboard is strongly interconnected, with most storyboard frames exhibiting highly similar content, although significant differences exist between different storyboards. All storyboards collectively constitute the entire video of the film or television drama. The plot division in this scheme can be based on storyboards as the unit.

[0059] 7) Plot / Storyline

[0060] Plot / storyline is one of the essential elements constituting the content of a film or television drama. It refers to the development of a series of life events that depict the relationships between characters throughout the entire video. It consists of a series of specific events that showcase character traits and demonstrate the relationships between characters and between characters and their environment. A film or television drama is typically composed of multiple plot segments arranged in chronological order. Each plot segment contains a complete story or character portrayal, meaning that each episode of a film or television drama consists of several relatively independent stories or character portrayals.

[0061] 8) Bullet screen

[0062] In this application, "bullet comments" refers to commentary text that pops up when watching videos online. Bullet comments can give viewers a sense of "real-time interaction." Although different bullet comments are sent at different times, they only appear at a specific point in the video. Therefore, bullet comments sent at the same time generally have the same theme, and when participating in the comments, viewers will feel like they are commenting at the same time as other viewers.

[0063] Please refer to Figure 1 This illustrates a system configuration diagram of a video processing system according to various embodiments of this application. Figure 1 As shown, the system includes a terminal 140 and a server 160; optionally, the system may also include a database 180.

[0064] Terminal 140 can be a terminal device with certain processing capabilities and interface display functions. For example, terminal 140 can be a mobile phone, tablet computer, e-book reader, smart glasses, laptop computer, desktop computer, smart TV, vehicle terminal, etc.

[0065] Terminal 140 may include terminals used by developers and terminals used by users.

[0066] When terminal 140 is implemented as a terminal for developers, developers can use terminal 140 to develop client and server-side applications for video playback (optionally, also for video publishing). These applications can be video applications, browser applications, news feed applications, etc. For example, the server-side application can be deployed to server 160, and the client-side application can be deployed to the user's terminal.

[0067] When terminal 140 is implemented as a user terminal, terminal 140 can download and install the client installation package of the aforementioned video playback application. After installing the application, terminal 140 can download / cache video data from server 160 through the application and play it in the playback interface of the application.

[0068] When terminal 140 is implemented as a terminal for developers, developers can also use terminal 140 to develop a plot segmentation application for determining the dividing points of different storylines in a video. This plot segmentation application can be deployed on the terminal or server of the video provider (such as the video provider corresponding to the video playback application mentioned above) to automatically determine the dividing points of different storylines in the video provided by the video provider, so that the corresponding dividing points can be displayed in the video playback interface when the video is provided later. Alternatively, the plot segmentation application can also be deployed on the user's terminal to automatically determine the dividing points of different storylines in the video to be uploaded by the user or the video provided by the video provider, so that the user can upload a video with the dividing points of the storylines, or so that the user can perform secondary development based on the dividing points of the storylines in the video.

[0069] Among them, server 160 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0070] The aforementioned server 160 may be a server that provides background services for the application installed in the terminal 140. This background server may be responsible for application version management, pushing videos and video-related information (such as storyline dividing point information) to the application's client, etc.

[0071] The aforementioned database 180 can be a Redis database, or it can be another type of database. Database 180 is used to store various types of data. Database 180 can be configured independently of server 160, in which case server 160 and database 180 can be connected via a communication network; alternatively, database 180 can also be configured within server 160.

[0072] Optionally, terminal 140 and server 160 are connected via a communication network. Optionally, the communication network can be a wired network or a wireless network.

[0073] Optionally, the system may also include a management device ( Figure 1 (Not shown), the management device is connected to the server 160 via a communication network. Optionally, the communication network can be a wired network or a wireless network.

[0074] Figure 2 This is a flowchart illustrating a video processing method according to an exemplary embodiment. The method can be executed by a computer device, such as a server, a terminal, or a combination of both. The server can be one of the aforementioned... Figure 1 The server 160 in the illustrated embodiment can be the terminal described above. Figure 1 Terminal 140 in the illustrated embodiment. (As shown) Figure 2 As shown, the video processing method may include the following steps.

[0075] Step 210: Obtain multiple scene segments from the target video; a scene segment consists of at least two consecutive video image frames, corresponding to the same shot.

[0076] A target video can be divided into multiple shot segments, each of which is a segment composed of consecutive video image frames from the same shot; multiple shot segments are arranged in chronological order to form a complete target video.

[0077] Step 220: Obtain modal feature information of multiple modalities for each of the multiple storyboard segments.

[0078] In the embodiments of this application, multiple modalities refer to multiple types of information related to the video, such as images, audio, text of various types / sources, people involved, etc.

[0079] For the same storyboard segment, different modalities correspond to different modalities and have different modal feature information; for example, the same storyboard segment has image features, audio features, text features, the names / numbers of the people involved, etc.

[0080] Step 230: Based on the modal feature information of each pair of adjacent storyboard segments in each storyboard segment, obtain the correlation of each pair of adjacent storyboard segments in each modality in multiple modalities. The correlation is used to indicate whether the two adjacent storyboard segments belong to the same storyline.

[0081] For a video, there will be certain correlations between different storylines and the modal feature information of the video in multiple modes. This can be used to determine whether two story segments belong to the same storyline. Therefore, by the similarity or correlation between the modal feature information of each two adjacent story segments in multiple modes, the correlation of each two adjacent story segments in each modality can be obtained.

[0082] For example, within the same storyline segment, video image frames from different scene segments are generally quite similar. Therefore, if the image features of video image frames from two adjacent scene segments are similar, or in other words, the image feature similarity of video image frames from two adjacent scene segments is high, it can be determined that these two adjacent scene segments belong to the same storyline under the modality corresponding to their image features. Conversely, if the image features of video image frames from two adjacent scene segments differ significantly, or in other words, the image feature similarity of video image frames from two adjacent scene segments is low, it can be determined that these two adjacent scene segments do not belong to the same storyline under the modality corresponding to their image features.

[0083] For example, within the same storyline, the audio features of different scene segments are generally quite similar. Therefore, if the audio features of two adjacent scene segments are similar, or in other words, the similarity of their audio features is high, it can be determined that these two adjacent scene segments belong to the same storyline under the modality corresponding to their audio features. Conversely, if the audio features of two adjacent scene segments differ significantly, or in other words, the similarity of their audio features is low, it can be determined that these two adjacent scene segments do not belong to the same storyline under the modality corresponding to their audio features.

[0084] For example, within the same storyline, the text features (such as at least one of audio text and dialogue subtitle text) of different scene segments are generally quite similar. Therefore, if the text features of two adjacent scene segments are similar, or in other words, the text feature similarity of two adjacent scene segments is high, it can be determined that these two adjacent scene segments belong to the same storyline under the modality corresponding to the text features. Conversely, if the text features of two adjacent scene segments differ greatly, or in other words, the text feature similarity of two adjacent scene segments is low, it can be determined that these two adjacent scene segments do not belong to the same storyline under the modality corresponding to the text features.

[0085] For example, in the same storyline, the characters in different storyboard segments are generally quite similar. Therefore, if two adjacent storyboard segments contain the same character, it can be determined that these two adjacent storyboard segments belong to the same storyline in the modality corresponding to the character. Conversely, if two adjacent storyboard segments contain completely different characters, it can be determined that these two adjacent storyboard segments do not belong to the same storyline in the modality corresponding to the character.

[0086] Step 240: Based on the correlation between each pair of adjacent story segments in each modality, obtain the plot dividing point of the target video; the plot dividing point is the dividing point between different story segments in the target video.

[0087] In this embodiment of the application, for each modality of two adjacent storyboard segments, a correlation can be determined. By combining the correlations of multiple modalities, it can be determined whether the two adjacent storyboard segments belong to the same storyline, thereby determining the dividing point between different storylines in the target video.

[0088] In some embodiments, the modal feature information of multiple modalities includes at least one of the following:

[0089] Background music information corresponding to the background music modality;

[0090] Human voice modality corresponds to human voice speech information;

[0091] The dialogue text information corresponding to the dialogue modality;

[0092] Character information corresponding to the character modality;

[0093] Scene information corresponding to scene modality;

[0094] And the bullet screen information corresponding to the bullet screen mode.

[0095] In summary, the solution provided in this application can obtain the modal feature information of each segment in multiple segments of a target video, and the correlation between each pair of adjacent segments in each modality. Then, based on the correlation between each pair of adjacent segments in each modality, the plot boundary point of the target video can be obtained. On the one hand, it can automatically integrate information from multiple modalities to determine the boundary point between storylines in the video. On the other hand, when information from one or more modalities is missing at two adjacent segments, it can also determine whether two adjacent segments belong to the same storyline through modal feature information from other modalities, thereby improving the efficiency and accuracy of determining the boundary point of storylines in the video.

[0096] The above application Figure 2 The proposed solution can be applied to, but is not limited to, the following scenarios:

[0097] 1) Many online video platforms possess extensive capabilities for structuring video content. For example, intelligent fast-forwarding allows users to automatically select their preferred scenes based on plot divisions, enabling fast-forwarding or skipping. The solution presented in this application divides video content into multiple shot segments. For each shot segment, modal feature information across various modes is extracted. Then, the correlation between any two adjacent shot segments within each modality is obtained. Based on this correlation, plot boundaries are identified, enabling rapid and efficient location of plot lines in films and television dramas. This provides fast algorithmic data services for this video content structuring capability, improving the efficiency of the entire implementation.

[0098] 2) In many short video derivative platforms, the official platform provides creators with a massive material library, which consists of a vast number of film and television video clips. The solution presented in this application allows for the division of films and television dramas into multiple shot segments. For each shot segment, modal feature information of various modes is extracted. Then, the correlation between each pair of adjacent shot segments in each modality is obtained. Based on the correlation between each pair of adjacent shot segments in each modality, the plot dividing points of the film and television drama are obtained. This provides automatic plot segmentation capabilities for derivative short video platforms, allowing all films and television dramas on online video platforms to be segmented at the plot level using this system. The segmented video clips are then input into the massive material library, providing short video creators with more high-quality plot and story materials and improving the efficiency of derivative creation.

[0099] 3) On many online video platforms, many videos, especially TV series, are serialized, so a certain number are updated daily, along with trailers for the next few episodes. The solution presented in this application can automatically divide and segment the content of the next day's episodes based on plot. Specifically, the content of the next day's episodes is divided according to shots, resulting in multiple scene segments. For each scene segment, modal feature information of multiple modes is extracted. Then, the correlation between each pair of adjacent scene segments in each modality is obtained. Based on the correlation between each pair of adjacent scene segments in each modality, the plot dividing point of the episode content is obtained. Because the video is segmented according to plot, each segment is a complete story, and the segmented plot videos can be directly selected as trailers for release, empowering the trailer function of the entire platform.

[0100] Based on the above Figure 2 Please refer to the embodiments shown. Figure 3 This is a flowchart illustrating a video processing method according to an exemplary embodiment. Figure 3 As shown above, Figure 2 Step 240 may include step 240a.

[0101] Step 240a: For adjacent first and second scene segments in each scene segment, in response to the fact that the correlation between the first and second scene segments in multiple modalities satisfies the specified conditions, the boundary point between the first and second scene segments is taken as a plot boundary point of the target video.

[0102] In this embodiment of the application, when determining whether two adjacent story segments belong to the same storyline by comprehensively considering the correlation of multiple modalities, and thus determining the dividing point between different storylines in the target video, the computer device can detect whether the correlation of two adjacent story segments in multiple modalities meets certain conditions, and determine whether the dividing point between two adjacent story segments is a plot dividing point of the target video based on the detection results.

[0103] In this embodiment, for two adjacent storyboard segments, a correlation can be determined for each modality. Multiple modalities will correspond to multiple correlations. In this case, combining multiple correlations to determine whether the boundary point between the first and second storyboard segments meets the conditions for a plot boundary point can improve the accuracy of determination compared to the scheme of determining whether the boundary point between the first and second storyboard segments meets the conditions for a plot boundary point through the correlation under a single modality. In other words, the above scheme can more accurately determine the plot boundary point of the target video.

[0104] In one possible implementation, the specified conditions mentioned above may include the following conditions:

[0105] 1) The correlation between the first and second storyboard segments in multiple modalities all indicate that the first and second storyboard segments do not belong to the same storyline.

[0106] In this embodiment of the application, when the correlation between two adjacent storyboard segments in various modalities indicates that the two storyboard segments do not belong to the same storyline, the dividing point between the two storyboard segments can be used as a plot dividing point of the target video.

[0107] In this embodiment of the application, when the first and second storyboard segments do not belong to the same storyline in multiple modalities, the computer device can determine that the dividing point between the two storyboard segments is a plot dividing point of the target video. This can avoid the situation where the dividing point between the first and second storyboard segments is mistakenly taken as a plot dividing point of the target video because the first and second storyboard segments are incorrectly divided into different storylines in one or more modalities, thereby improving the accuracy of plot dividing point determination.

[0108] In one possible implementation, the correlation between the first and second scene segments across multiple modalities all indicate that the first and second scene segments do not belong to the same storyline. This means that the correlations corresponding to the various modal feature information of the first and second scene segments that are not missing all indicate that the first and second scene segments do not belong to the same storyline. In other words, when comprehensively determining whether two adjacent scene segments belong to the same storyline through the correlations of multiple modalities, modalities with missing modal feature information can be disregarded, and only the correlations corresponding to modalities with existing modal feature information can be considered. This ensures that even when certain modalities are present, the plot boundary point can still be correctly determined, guaranteeing the accuracy of the plot boundary point determination.

[0109] 2) Among multiple modalities, the number of modalities in which the first storyboard segment and the second storyboard segment do not belong to the same storyline is not less than the number threshold.

[0110] In the embodiments of this application, when two adjacent storyboard segments have more than a specified number of modal correlations in various modalities indicating that the two storyboard segments do not belong to the same storyline, the dividing point between the two storyboard segments can be used as a plot dividing point of the target video.

[0111] In cases involving correlation across multiple modalities, it's possible that the correlation between any two adjacent scene segments across all modalities / all unmissing modalities indicates that these two scene segments do not belong to the same storyline. In such cases, if the storyline boundary point is determined according to condition 1) above, it would be impossible to distinguish the boundary points of different storylines. To address this, in the solution shown in the embodiments of this application, the condition restriction can be relaxed based on condition 1). It is only necessary that, across all modalities / all unmissing modalities, there exists a correlation in some modalities greater than a specified threshold that indicates that these two scene segments do not belong to the same storyline. This is sufficient to determine the boundary point between these two scene segments as a plot boundary point of the target video, thereby improving the accuracy of plot boundary point determination.

[0112] 3) Among multiple modalities, the sum of the importance scores of modalities whose corresponding relevance indicators, the first and second storyboard segments, do not belong to the same storyline is not less than the score threshold; each modality has its own importance score.

[0113] In this embodiment, since the importance of different modalities may vary, for example, the correlation between background music and voice modalities is more indicative of whether two story segments do not belong to the same storyline than the correlation between scene modalities. In this case, a higher importance score can be set for the background music and voice modalities, and a lower importance score can be set for the scene modalities. When comprehensively determining whether two adjacent story segments belong to the same storyline by considering the correlation of multiple modalities, the modalities that indicate the first and second story segments do not belong to the same storyline can be identified first. The importance scores of these modalities can then be summed, and the summed result can be compared with a score threshold. If the summed result is greater than the score threshold, it can be determined that the two story segments do not belong to the same storyline. Thus, the boundary point between the two story segments can be used as a plot boundary point of the target video, thereby improving the accuracy of plot boundary point determination.

[0114] Optionally, the importance score for each modality can be correlated with the similarity or matching degree of the two current scene clips in that modality; the higher the similarity or matching degree, the higher the importance score. In other words, the importance score for each modality will change depending on the scene clips being processed. For example, the importance score of multiple modalities when a computer determines that the relevance of scene clip 1 and scene clip 2 in multiple modalities meets specified conditions may differ from the importance score of multiple modalities when a computer determines that the relevance of scene clip 2 and scene clip 3 in multiple modalities meets specified conditions.

[0115] The similarity or matching degree of the two current scene segments in this modality can refer to the similarity or matching degree between the modal feature information corresponding to the two current scene segments in this modality.

[0116] For example, the importance score of the background music modality is related to the similarity between the background music information of the two current scene segments; the higher the similarity, the higher the importance score.

[0117] For example, the importance score of the human voice modality is related to the matching degree of the human voice information of the two current scene segments (such as the correlation / continuity of the human voice information of the two current scene segments, which can be judged by the changes in volume / pitch / timbre of the human voice information at the boundary point of the two scene segments. For example, the higher the similarity of volume / pitch / timbre at the boundary point of the two scene segments, the higher the matching degree of the human voice information of the two current scene segments). The higher the matching degree, the higher the importance score.

[0118] For example, the importance score of the dialogue modality is related to the similarity of the dialogue text information of the two current scene segments. The higher the similarity, the higher the importance score. For instance, if the dialogue text information of the two current scene segments is the same, the importance score of the dialogue modality can be high, such as 1. If the dialogue text information of the two current scene segments is different, the importance score of the dialogue modality can be low, such as 0.

[0119] For example, the importance score of a character modality is related to the similarity of the character information in the two current storyboard segments. The higher the similarity, the higher the importance score. For instance, character information includes the names or numbers of the characters in the storyboard segments. The similarity of the character information in the two current storyboard segments can be the similarity between the names or numbers of the characters contained in the two current storyboard segments.

[0120] For example, the importance score of a scene modality is related to the similarity of the scene information of the two current storyboard clips. The higher the similarity, the higher the importance score. For example, the scene information mentioned above can be scene names (such as indoor, outdoor, street, road, hillside, grassland, etc.). There can be a pre-set importance score between each pair of scenes (for example, it can be pre-set by the developers), and the higher the similarity between the two scenes, the higher the pre-set importance score. Another example is that the scene information mentioned above can be a feature matrix / feature vector extracted from the storyboard clips, which is related to the scene. The higher the similarity between the feature matrices / feature vectors of the two current storyboard clips, the higher the corresponding importance score of the scene modality.

[0121] For example, the importance score of the bullet screen modality is related to the matching degree of the bullet screen information of the two current scene segments (such as the correlation / continuity of the bullet screen information of the two current scene segments, which can be judged by the change in the number of bullet screens in the two scene segments. For example, the greater the difference in the number of bullet screens in the two scene segments, the higher the matching degree of the bullet screen information of the two current scene segments). The higher the matching degree, the higher the importance score.

[0122] Optionally, the importance score can be the product of the similarity between the two current scene segments in that modality and the importance score base. Each modality has its own importance score base, which is preset.

[0123] based on Figure 2 or Figure 3 In one possible implementation of the scheme shown, the modal feature information responding to multiple modalities includes background music information corresponding to the background music modality.

[0124] Obtain modal feature information of multiple modalities for each scene segment in multiple scene segments, including:

[0125] Obtain the background music for the first storyboard segment and the background music for the second storyboard segment; the first storyboard segment and the second storyboard segment are two adjacent storyboard segments among multiple storyboard segments;

[0126] Audio features were extracted from the background music of the first storyboard segment to obtain the background music information of the first storyboard segment;

[0127] Audio features were extracted from the background music of the second scene segment to obtain the background music information of the second scene segment;

[0128] Based on the modal feature information of each pair of adjacent scene segments in each segment, the correlation between each pair of adjacent scene segments in each modality is obtained, including:

[0129] Calculate the first similarity between the background music information of the first storyboard segment and the background music information of the second storyboard segment;

[0130] Based on the first similarity, the first correlation between the first scene segment and the second scene segment in the background music modality is obtained.

[0131] In this embodiment of the application, the computer device can extract the background music of two adjacent storyboard segments respectively, and then extract the audio features of the background music of the two storyboard segments as the background music information of the two storyboard segments.

[0132] Computer equipment can calculate the spatial distance between the audio features of the background music of two storyboard segments, such as the cosine value of the angle between the audio feature vectors of the background music of the two storyboard segments or the Euclidean distance, etc., as the first similarity between the background music information of the two storyboard segments. Then, when the above-mentioned first similarity is higher than a certain similarity threshold, for example, when the above-mentioned cosine value is greater than a certain threshold (e.g., greater than 0.9), or when the reciprocal of the above-mentioned Euclidean distance is greater than a certain threshold, it can be considered that the two storyboard segments belong to the same storyline under the background music modality (i.e., the first correlation indicates that the first storyboard segment and the second storyboard segment belong to the same storyline), otherwise, it is considered that the two storyboard segments do not belong to the same storyline under the background music modality (i.e., the first correlation indicates that the first storyboard segment and the second storyboard segment do not belong to the same storyline).

[0133] The above solution provides an effective way to determine whether two adjacent story segments belong to the same storyline by using background music information, thus ensuring the accuracy of subsequent storyline boundary point determination.

[0134] based on Figure 2 or Figure 3 In one possible implementation of the scheme shown, the modal feature information responding to multiple modalities includes human voice speech information corresponding to the human voice modality, and dialogue text information corresponding to the dialogue modality.

[0135] Obtain modal feature information of multiple modalities for each scene segment in multiple scene segments, including:

[0136] Obtain the voice recordings of the first and second storyboard segments; the first and second storyboard segments are two adjacent storyboard segments among multiple storyboard segments.

[0137] Audio features are extracted from the human voice in the first scene segment to obtain the human voice information of the first scene segment;

[0138] Audio features are extracted from the human voice in the second scene segment to obtain the human voice information of the second scene segment.

[0139] Get the dialogue text of the first storyboard segment and the dialogue text of the second storyboard segment;

[0140] Text features are extracted from the dialogue text of the first storyboard segment to obtain the dialogue text information of the first storyboard segment;

[0141] Text features were extracted from the dialogue text of the second storyboard segment to obtain the dialogue text information of the second storyboard segment;

[0142] Based on the modal feature information of each pair of adjacent scene segments in each segment, the correlation between each pair of adjacent scene segments in each modality is obtained, including:

[0143] Calculate the second similarity between the human voice information of the first storyboard segment and the human voice information of the second storyboard segment;

[0144] Calculate the third similarity between the dialogue text information of the first storyboard segment and the dialogue text information of the second storyboard segment;

[0145] Based on the second and third similarities, the second correlation between the first and second scene segments in the human voice modality and the dialogue modality is obtained.

[0146] The aforementioned voice information can be extracted from storyboard segments, and the aforementioned dialogue text information can be obtained by recognizing the image frames of storyboard segments, or obtained from the subtitle text corresponding to the storyboard segments in the subtitle file.

[0147] The aforementioned voice information can be used to represent the characteristics of a speaker's voice. For example, the voice information may include the speaker's voiceprint features. In other words, if the voice information in two adjacent scene segments has a high similarity (e.g., greater than a certain similarity threshold), then the speakers in these two scene segments are considered to be the same. In this case, the two scene segments are considered to be continuous in terms of plot and belong to the same storyline. Conversely, if the voice information in two adjacent scene segments has a low similarity, then the speakers in these two scene segments are considered to be different. In this case, the two scene segments are considered to be discontinuous in terms of plot and belong to different storylines.

[0148] In this embodiment of the application, the computer device can combine the human voice speech information corresponding to the human voice modality and the dialogue text information corresponding to the dialogue modality to determine the correlation. Specifically, based on the human voice speech information corresponding to the human voice modality, the computer device can determine the similarity of the human voice speech information of two adjacent scene segments and the similarity of the dialogue text information of these two scene segments respectively, and comprehensively determine the correlation of the two scene segments in the human voice modality and the dialogue modality based on these two similarities.

[0149] For example, after a computer device determines the similarity of the voice information of two adjacent scene segments and the similarity of the dialogue text information of these two scene segments, it can take the larger value of these two similarities and determine the correlation between the two scene segments in the voice modality and the dialogue modality (i.e., the second correlation mentioned above) based on the larger value of these two similarities. In other words, it can determine whether the two scene segments belong to the same storyline in the voice modality and the dialogue modality. For example, based on the relationship between the larger value of these two similarities and the similarity threshold, it can be determined whether the two scene segments belong to the same storyline in the voice modality and the dialogue modality.

[0150] For example, after the computer equipment determines the similarity of the voice information of two adjacent scene segments and the similarity of the dialogue text information of the two scene segments, it can take the average of the two similarities and determine the correlation between the two scene segments in the voice modality and the dialogue modality based on the average of the two similarities, that is, determine whether the two scene segments belong to the same storyline in the voice modality and the dialogue modality.

[0151] The above solution provides an effective way to determine whether two adjacent story segments belong to the same plot by combining human voice information and dialogue text information, thus ensuring the accuracy of subsequent plot dividing point determination.

[0152] based on Figure 2 or Figure 3 In one possible implementation of the scheme shown, the modal feature information responding to multiple modalities includes character information corresponding to the character modality, and the character information is used to indicate the characters contained in the corresponding storyboard segment.

[0153] Obtain modal feature information of multiple modalities for each scene segment in multiple scene segments, including:

[0154] Character identification is performed on the first and second storyboard segments respectively to obtain character information for both the first and second storyboard segments; the first and second storyboard segments are two adjacent storyboard segments among multiple storyboard segments.

[0155] Based on the modal feature information of each pair of adjacent scene segments in each segment, the correlation between each pair of adjacent scene segments in each modality is obtained, including:

[0156] Calculate the third similarity between the character information of the first storyboard segment and the character information of the second storyboard segment;

[0157] Based on the third similarity, the third correlation between the first and second storyboard segments under the character modality is obtained.

[0158] In this application embodiment, for two adjacent storyboard segments, if the characters contained in the two storyboard segments are the same or partially the same, it can be considered that the two storyboard segments may belong to the same storyline, and the higher the proportion of the same characters, the higher the probability that the two storyboard segments belong to the same storyline. Based on this, the solution shown in the embodiments of this application can identify characters in each storyboard segment to obtain character information contained in each storyboard segment. Then, for two adjacent storyboard segments, the similarity between the character information of these two storyboard segments is calculated. For example, the computer device can count the total number of characters appearing in two adjacent storyboard segments and the number of identical characters appearing in the two adjacent storyboard segments. Then, the number of identical characters appearing in two adjacent storyboard segments is divided by the total number of characters appearing in two adjacent storyboard segments to obtain the third similarity. The third association is determined based on the third similarity. For example, when the third similarity is greater than a certain similarity threshold, it is determined that the two adjacent storyboard segments belong to the same storyline. When the third similarity is not greater than a certain similarity threshold, it is determined that the two adjacent storyboard segments do not belong to the same storyline. For another example, the computer device can determine whether the character information contained in two adjacent storyboard segments is the same. If so, it is determined that the two adjacent storyboard segments belong to the same storyline; otherwise, it is determined that the two adjacent storyboard segments do not belong to the same storyline.

[0159] The above solution provides an effective way to determine whether two adjacent storyboard segments belong to the same storyline by examining the characters they contain, thus ensuring the accuracy of subsequent storyline boundary point determination.

[0160] based on Figure 2 or Figure 3 In one possible implementation of the scheme shown, the modal feature information responding to multiple modalities includes scene information corresponding to the scene modality, and the scene information is used to indicate the scene type of the corresponding storyboard segment.

[0161] Obtain modal feature information of multiple modalities for each scene segment in multiple scene segments, including:

[0162] Scene recognition is performed on the first and second storyboard segments respectively to obtain scene information of the first and second storyboard segments; the first and second storyboard segments are two adjacent storyboard segments among multiple storyboard segments.

[0163] Based on the modal feature information of each pair of adjacent scene segments in each segment, the correlation between each pair of adjacent scene segments in each modality is obtained, including:

[0164] Based on the scene type indicated by the scene information of the first scene segment and the scene type indicated by the scene information of the second scene segment, the fourth correlation between the first scene segment and the second scene segment in the scene modality is obtained.

[0165] In this embodiment, for two adjacent storyboard segments, if the scenes of the two storyboard segments are the same (e.g., both are living rooms) or similar (e.g., living room and bedroom), then the two storyboard segments may belong to the same storyline; otherwise, they are considered not to belong to the same storyline. Based on this, the solution shown in this application can perform scene recognition on each storyboard segment to obtain scene information for each segment. Then, for two adjacent storyboard segments, it is determined whether the two storyboard segments belong to the same storyline based on whether the scene types indicated by the scene information of the two storyboard segments are the same or similar. For example, if the scene types indicated by the scene information of the two storyboard segments are the same, then the two storyboard segments are determined to belong to the same storyline; otherwise, the two storyboard segments are determined not to belong to the same storyline. Furthermore, if the scene types indicated by the scene information of the two storyboard segments are the same or similar, then the two storyboard segments are determined to belong to the same storyline; otherwise, the two storyboard segments are determined not to belong to the same storyline. The similarity of the two scene types can be preset by the developer.

[0166] The above solution provides an effective way to determine whether two adjacent storyboard segments belong to the same storyline by examining the scenes of two adjacent storyboard segments, thus ensuring the accuracy of subsequent storyline boundary point determination.

[0167] based on Figure 2 or Figure 3 In one possible implementation of the scheme shown, the modal feature information responding to multiple modalities includes the bullet screen information corresponding to the bullet screen modality.

[0168] Obtain modal feature information of multiple modalities for each scene segment in multiple scene segments, including:

[0169] Based on the number of bullet comments at each time point in the first and second scene segments, the bullet comment number curves that change with time are obtained and used as the bullet comment information for the first and second scene segments; the first and second scene segments are two adjacent scene segments among multiple scene segments.

[0170] Based on the modal feature information of each pair of adjacent scene segments in each segment, the correlation between each pair of adjacent scene segments in each modality is obtained, including:

[0171] Based on the bullet comment count curve, the fifth correlation between the first and second scene segments in the bullet comment modality is obtained.

[0172] For example, when the barrage curve reaches a trough at the dividing point between two adjacent story segments, and the number of barrages at the dividing point is less than the number threshold, it can be determined that the two adjacent story segments do not belong to the same storyline; or, for example, when the barrage curve in two adjacent story segments generally shows an upward / downward trend, and the number of barrages at the dividing point is less than the number threshold, it can be determined that the two adjacent story segments do not belong to the same storyline.

[0173] In this embodiment, for a given time period corresponding to a storyline, the number of bullet comments may follow certain rules. For example, the number of bullet comments may rise to a certain level in a short period and remain above that level, then slowly decrease. At the end of the storyline, the number of bullet comments may drop to a low point. Then, in the next storyline, the number of bullet comments may rise again and remain there, before decreasing again before the storyline ends. Based on this, the solution shown in this embodiment can obtain the curves of the number of bullet comments in two adjacent story segments over time, analyze the trend of the number of bullet comments in these two story segments, and if the number of bullet comments in these two story segments reaches a low point at the boundary between the two story segments, then these two story segments can be considered not to belong to the same storyline.

[0174] The above solution provides an effective way to determine whether two adjacent story segments belong to the same storyline by observing the changing trend of the number of bullet comments between them, thus ensuring the accuracy of subsequent storyline boundary point determination.

[0175] based on Figure 2 or Figure 3 The scheme shown, in one possible implementation, involves acquiring multiple scene clips from the target video, including:

[0176] Obtain the image similarity between any two adjacent video image frames in each video image frame of the target video;

[0177] Based on the image similarity between any two adjacent video image frames in the target video, the target video is divided into multiple segment segments.

[0178] Since a storyboard segment is typically a video clip shot from a fixed or slowly moving camera angle, the differences between video image frames within a storyboard segment are relatively small. Based on this, the solution shown in the embodiments of this application can segment the storyboard segment by measuring the image similarity between any two adjacent video image frames in the target video. That is, when the image similarity between two adjacent video image frames is high (e.g., higher than a certain similarity threshold), the two video image frames are assigned to the same storyboard segment; conversely, when the image similarity between two adjacent video image frames is low (e.g., not higher than the aforementioned similarity threshold), the two video image frames are assigned to different storyboard segments.

[0179] In some embodiments, obtaining the image similarity between any two adjacent video image frames in each video image frame of the target video includes:

[0180] The first video image frame is processed by an image feature extraction network to obtain the first image features output by at least two specified network layers in the image feature extraction network.

[0181] The second video image frame is processed by an image feature extraction network to obtain the second image features output by at least two specified network layers; the first video image frame and the second video image frame are two adjacent video image frames in each video image frame;

[0182] Based on the first image features and the second image features, the image similarity between the first video image frame and the second video image frame is obtained.

[0183] In this embodiment of the application, the computer device can represent the image similarity between video image frames by the similarity between the image features of the video image frames. The image features of the video image frames can be extracted by means of an image feature extraction network, thereby ensuring the accuracy of image feature extraction and thus ensuring the accuracy of segment division.

[0184] This application proposes a method for locating the boundary points of complete plot segments in film and television dramas based on multimodal and bullet screen data complementary judgment. This method uses multiple modalities to perform complementary judgments and calculate the boundary points between various plot segments in film and television dramas. By using multimodal complementarity, the system can be applied to plot boundaries in different types of film and television videos, especially for situations with frequent scene transitions and camera jumps. The system also enhances robustness, capable of handling various video formats, particularly when one modality is missing; it can directly apply other modalities to divide the plot within the entire video. Furthermore, this solution incorporates user-generated bullet screen data (bullet screen data) related to the film and television video content. This bullet screen data supplements the multimodal judgment in this solution, making the final calculated plot boundary points more aligned with user preferences and subjective feelings, thereby improving the overall system performance.

[0185] This innovative system for locating the boundary points of complete plot segments in film and television dramas first uses an AI-based feature extraction network (e.g., an improved Swin-Transformer model) to locate all the boundary lines of the entire input video, thereby calculating the micro-elements (i.e., the plot segments) of the entire system. Then, the multimodal features of all the plot segments are input into a multimodal plot judgment module built in this solution. This module uses visual information, audio information, background music, and dialogue text from the video to calculate the final result. The system can determine whether all plot segments belong to the same storyline from different modalities and modal derivatives (scenes, characters) or combinations thereof. Furthermore, the system can incorporate bullet screen data from the film and television drama to complement the multimodal judgment, making the plot boundary points calculated for the entire film and television drama video more in line with the audience's preferences and subjective feelings.

[0186] Taking the modal feature information of the aforementioned multiple modalities, including background music information corresponding to the background music modality, human voice information corresponding to the human voice modality, dialogue text information corresponding to the dialogue modality, character information corresponding to the character modality, scene information corresponding to the scene modality, and bullet screen information corresponding to the bullet screen modality, as examples, the above-mentioned information in this application... Figure 2 or Figure 3 The technical flowchart of the video processing solution shown can be as follows: Figure 4 As shown.

[0187] like Figure 4As shown, the solution involved in this application can be divided into three modules: a first module 410 based on the improved Swin-Transformer (hereinafter referred to as the ST model) model for scene judgment, a second module 420 for scene judgment using multimodal and film and television barrage data, and a third module 430 for fusion regularization judgment.

[0188] First, there is a first module 410 based on the improved Swin-Transformer for scene determination. The input of the first module 410 is the video data information of the original film and television video. Specifically, the input to the model is the image of every two adjacent video frames. After the calculation by the first module 410, the correlation between the two video frames can be obtained. Thus, the time boundary line / boundary point between multiple scene segments in the whole video can be determined by the calculated correlation. This scheme can accurately, quickly and in one go locate the scene.

[0189] The second module 420 is a plot segment judgment module based on multimodal and bullet screen data. The input of the second module 420 is also all the information of the original video. The entire module is divided into five branches, which are for five modal and bullet screen data. The second module 420 first uses an audio classification model to classify audio, judging the human voice and music in the audio tracks of the entire video, thereby classifying the background music and human voice dialogue. The human voice dialogue refers to the human voice part of the corresponding lines, that is, the human voice audio of the person speaking the lines. Then, the background music can be judged separately for plot. The two modalities of human voice dialogue and dialogue text are fused to perform plot division based on dialogue information. In addition, the character information in the storyboard can also be used to calculate the plot division result. Finally, the scene information of the storyboard is used to merge the storyboard segments, merging the storyboard segments with high similarity into the same storyline, and dividing the storyboard segments with low similarity into different storylines, or the dividing point between the low similarity storyboard segments as the dividing point of the storyline, thereby completing the scene-based plot division. Then, the plot division results of the bullet screen data are combined and input into the third module 430 for complementary fusion and regularization, so as to calculate and judge the final plot dividing point time.

[0190] The diagrams illustrating the multimodal data, bullet comments, and scene segmentation can be seen as follows: Figure 5 As shown. In Figure 5 In this process, one or more modal feature information can be extracted from each segment, including background music, voice lines, subtitle lines, characters, scenes, and bullet screen data curves (which can represent the number of bullet screens at each time point).

[0191] The three modules mentioned above will be described in detail below:

[0192] Module 1.410: Scene Determination Module of the Improved Swin-Transformer

[0193] Please refer to Figure 6 and Figure 7 ,in, Figure 6 A schematic diagram of the Swin-Transformer involved in this application is shown. Figure 7 A schematic diagram of the improved Swin-Transformer involved in this application is shown.

[0194] The Swin-Transformer first uses video image frames as the raw feature input, then inputs the images into the Patch Partition module 610 for segmentation, where each 4x4 adjacent pixels form a patch, and then the images are flattened in the channel direction; please refer to [reference needed]. Figure 8 It shows an operational schematic diagram of the block segmentation module involved in this application, such as... Figure 8 As shown, the block segmentation module first extracts feature maps from the input image according to the block position (S801), then concatenates them according to the feature map channels (S802), and then performs stretching (S803) and normalization (S804). Afterwards, it performs dimensionality reduction according to the channels using a linear transformation (S805). Assuming the input to this block segmentation module is an RGB three-channel image, each patch has 4x4 = 16 pixels, and each pixel has three values: R, G, and B. Therefore, after flattening, there are 16x3 = 48 values. After Patch Partition, the image data structure (shape) changes from [H, W, 3] to [H / 4, W / 4, 48]. Then, the Swin-Transformer performs a linear transformation on the channel data of each pixel through a linear embedding layer, changing the value from 48 to C. That is, the image shape changes from [H / 4, W / 4, 48] to [H / 4, W / 4, C]. In the source code, PatchPartition and Linear... Embedding can be achieved using a convolutional layer.

[0195] The Swin-Transformer then constructs feature maps of different sizes through four stages. Except for Stage 1, which first uses a Linear Embedding layer 620, the remaining three stages all begin with a Patch Merging layer 630 for downsampling. Each stage then contains repeatedly stacked Swin-Transformer Blocks (ST blocks) 640. These Swin-Transformer Blocks can have two structures; please refer to [reference needed]. Figure 9 It shows structural diagrams of the two ST blocks involved in this application, such as Figure 9 As shown, the input to these two structures is feature z. The only difference between the two structures is that one ST block uses a W-MSA (Windows Multi-head Self-Attention) structure 901, while the other ST block uses an SW-MSA (Shifted Windows Multi-Head Self-Attention) structure 902. Moreover, these two structures are used in pairs, that is, a W-MSA structure is used in one ST block first, and then an SW-MSA structure is used in the next ST block. Finally, for the classification network, in each stage, the W-MSA / SW-MSA structure can be followed by a normalization layer (such as Layer Norm, LN), a multilayer perceptron (MLP), a global pooling layer, and a fully connected layer to obtain the final output.

[0196] The Patch Merging layer is used for downsampling. This module's function is to reduce resolution, adjust the number of channels, and thus create a hierarchical design while saving computational resources. In Convolutional Neural Networks (CNNs), this is done before each Stage using convolutional / pooling layers with a stride of 2. Patch Merging is similar to pooling, but more complex. Pooling loses information, while patch Merging does not. Patch Merging downsamples by a factor of two each time, so elements are selected at intervals of 2 in both row and column directions, concatenated into a new patch, and then all patches are concatenated into a single tensor, which is then expanded. At this point, the channel dimension becomes four times the original (because H and W are each reduced by a factor of 2).

[0197] The Swin-Transformer can obtain the corresponding embedding vector by adding a 650-fold reshape layer after the four stages, which is the embedding encoding of the current input image.

[0198] Please refer to Figure 7 The improved Swin-Transformer is a superposition of two Swin-Transformers. The first model 710 takes the information from the previous frame as input, and the second model 720 takes the information from the next frame as input. Figure 7 As you can see, there are three parts (i.e.) Figure 7 The relevance of P1, P2, and P3 in the model is calculated (e.g., cosine similarity COS). Using only the last information for relevance calculation would cause the embedding to lose the underlying information. By combining the relevance calculations of different parts, the loss of the model can be finally formed for training. The final relevance is also composed of these three parts. For example, the final loss can be determined by the weighted sum of the three parts above. For example, the final loss Loss = 0.2*P1 + 0.3*P2 + 0.5*P3.

[0199] By using an improved Swing-Transformer, the correlation between every two adjacent video image frames in the entire video frame sequence can be calculated, resulting in a correlation curve. A schematic diagram of this correlation curve can be derived from... Figure 10 As shown, Figure 10 The horizontal axis represents time, and the vertical axis represents relevance. Figure 10 As can be seen, three points (points 1001, 1002, and 1003) have particularly low correlation. The two adjacent frames corresponding to these points are the boundary lines between two scenes, hence the low correlation, which allows us to determine the distance between the boundary lines of these two scenes. After calculation, all the scene boundary lines in the entire video can be located. The entire solution involves merging scenes based on micro-elements, ultimately generating individual plot segments.

[0200] Module 2.420: Multimodal and Bullet Screen Data Segmentation Module

[0201] Please refer to Figure 11 It shows the architecture diagram of the multimodal and bullet screen data segmentation judgment module involved in this application, such as Figure 11 As shown, the overall process of this module is divided into five branches ( Figure 11The diagram shows branches 1101 to 1105. These five branches can each locate the dividing lines, and then the final dividing lines are input into the third module for fusion and normalization, ultimately locating accurate dividing lines for all plots. The following is a description of each of these five branches.

[0202] A) Judgment based on background music (corresponding to) Figure 11 Branch road 1101)

[0203] Please refer to Figure 12 This illustrates a schematic diagram of background music modal segmentation related to this application. Figure 12 As shown, a complete storyline is often accompanied by a single, continuous piece of background music. Therefore, when a single piece of background music spans two or more scenes, these scenes are highly likely to belong to the same storyline. Thus, the second module 420 needs to identify the background music. Simultaneously, since the dialogue in this solution also falls under the category of audio modal features, the second module 420 can construct a multi-classification network for music and dialogue to locate the timing of background music and dialogue segments within the entire video's audio track.

[0204] Please refer to Figure 13 This illustrates a network structure diagram for classifying background music and human voices, as described in this application. Figure 13 As shown:

[0205] (1) The input 1301 of the classification network uses the audio sampling point sequence of film and television dramas, that is, the input of the entire network is the original sequence of audio signals; then the classification network is divided into two branches, which use the original audio sampling points of the model input to extract the basic feature map of the audio, and then the basic feature map can be used as the input of the frequency domain branch of the classification network at the same time. The entire classification network can be divided into a time domain processing branch and a frequency domain processing branch.

[0206] (2) Figure 13 The left-hand branch is the time-domain processing branch, which uses multiple one-dimensional convolutional layers 1302. Using convolutional layers in the time-domain signal can directly learn the time-domain characteristics of the audio signal, especially information such as audio loudness and sampling point amplitude. After multiple one-dimensional convolutional layers, the generated one-dimensional sequence is resized to obtain a two-dimensional wavegram. The purpose is to combine the output of this branch with the output of the frequency-domain branch.

[0207] (3) Figure 13The right-hand branch is the frequency domain processing branch. The input to this branch is also the original audio signal. Then, the frequency domain spectrum of the audio is obtained. Here, the obtained spectrum can be the log-mel spectrum, which can be the Mel frequency. Then, the obtained frequency domain spectrum is input into multiple two-dimensional convolutional layers 1303, and the output is a feature map. This feature map has the same dimension as the feature map obtained by the left-hand time domain branch.

[0208] (4) Figure 13 In the middle of the network, there is a switch module 1304 controlled by a switch mechanism. This module is used to input user data into the entire classification network. The input to this module is a segment of the current user data curve. If the switch is turned on, it means that user data is input. Then, the user data is input into a fully connected layer and then expanded in dimension to generate new data features. Then, the new data features and the data from the left and right branches are input into the attention mechanism layer, which allows the user data to be combined with the data from the two branches. The two outputs are then concatenated and input into the top layer overlay module 1305. This mechanism is designed to maintain the complementarity of information in the time domain and frequency domain, while also allowing the higher-level network to perceive the information of the lower-level network.

[0209] (5) Figure 13 The classification network shown superimposes the output feature maps of the left and right branches of the same dimension and the feature maps of the two middle branches to form a set of two-dimensional frequency domain feature maps. The generated two-dimensional frequency domain feature maps are input into the two-dimensional convolutional neural network (CNN) module 1306. Then, the mean and maximum values ​​are calculated according to the last dimension. The mean and maximum values ​​are added together and then passed through an activation function layer 1307 (such as a ReLU network layer) to generate the final audio semantic feature vector.

[0210] (6) Figure 13 The classification network shown inputs the generated audio semantic feature vectors into the final classification network to classify background music, human voices, and silence, thereby enabling the entire system to calculate and locate the background music and human voice segments.

[0211] B) Judgment based on human voice dialogue and subtitle dialogue (corresponding to) Figure 11 Branch road 1102)

[0212] The input of branch 1102 is Figure 13The classification network shown identifies voice-over dialogue fragments and subtitles from the film / TV series. Branch 1102 then uses these two modalities to jointly determine adjacent scenes, thereby calculating the plot boundary. Both voice-over dialogue and subtitles involve textual semantic similarity calculations. Therefore, this scene merging process, based on dialogue similarity calculations, utilizes a combination of voice-over dialogue fragments and subtitles from the film / TV series.

[0213] Please refer to Figure 14 The diagram illustrates the flowchart of the judgment module based on human voice dialogue and subtitle dialogue involved in this application. Figure 14 As shown, the final storyboard similarity calculation based on human voice data and subtitles is divided into two branches: a calculation branch 1401 based on human voice (used to output human voice similarity) and a calculation branch 1402 based on subtitles (used to output subtitles similarity). When there is a missing piece of information in a storyboard, it is automatically reset to zero. Then, the similarity outputs from the two calculation branches are fused to obtain the storyboard similarity. The storyboard similarity with the highest value is taken as the final value of the similarity between human voice / subtitles.

[0214] Please refer to Figure 15 This illustrates a schematic diagram of the judgment based on human voice dialogue involved in this application. For example... Figure 15 As shown, the process of calculating the similarity of human voices first requires extracting the human voice tracks from the storyboard using the human voice background separation module 1501 (this can be achieved using an audio track separation tool, for example, by using...). Figure 13 The classification network shown can directly locate the time segment of the human voice, and then the encoding network 1502 performs network encoding on the corresponding human voice track to obtain the corresponding embedding. The embedding can contain the feature information of the human voice in the scene, such as at least one of the voiceprint information. Then, the similarity distance between the two embeddings can be calculated, so that the similarity relationship between the two scene segments based on the human voice can be obtained.

[0215] Please refer to Figure 16 This illustrates a schematic diagram of the judgment based on subtitle dialogue involved in this application. For example... Figure 16As shown, there are two methods for obtaining subtitle text: one is to obtain subtitle text by performing Optical Character Recognition (OCR) on video image frames, and the other is to obtain subtitle text from the video's built-in subtitle text file. After obtaining the subtitle text of two storyboard segments, in the process of calculating the similarity between the subtitle text of the two storyboard segments, the subtitle text in the storyboard is processed by a text encoding network 1601 (for example, a Bidirectional Encoder Representations from Transformers (BERT) model network can be used to obtain text representations (tokens), and then the similarity between the tokens is calculated, which is the similarity between the storyboard segments based on the subtitle text.

[0216] After obtaining the similarity scores of the storyboard segments based on voice lines and dialogue text respectively, the relationship between two adjacent storyboard segments (whether they belong to the same storyline) can be determined, thus deciding whether to merge them. For example, if the similarity between the voice lines / dialogue text of two adjacent storyboard segments is greater than a certain threshold, the two adjacent storyboard segments are merged into the same storyline. Conversely, if the similarity between the voice lines / dialogue text of two adjacent storyboard segments is greater than a certain threshold, the two adjacent storyboard segments are divided into different storylines.

[0217] C) Judgment based on role information (corresponding to) Figure 11 Branch road 1103)

[0218] Figure 11 Branch 1103 uses character information from two storyboard segments to calculate the similarity between adjacent segments. For example, branch 1103 uses a recognition model to identify and calculate the characters appearing in each storyboard segment, then counts the number and IDs of characters in each segment, and compares the differences in characters between two adjacent storyboard segments to determine the similarity between the two segments based on character information. Branch 1103 relies on the character information in two adjacent storyboard segments; if they belong to the same plot, then the same or similar characters will appear in the two storyboard segments. Please refer to [reference needed]. Figure 17 It illustrates a diagram of the role-based judgment involved in this application, such as... Figure 17 As shown, the character statistics calculated by the recognition model in the three storyboard segments are as follows: Celebrities 1, 2, and 3 appear in storyboard segment 1, Celebrities 1 and 3 appear in storyboard segment 2, and Celebrities 2 and 3 appear in storyboard segment 3. Based on this, it can be considered that these three storyboard segments may belong to the same storyline.

[0219] D) Judgment based on scene information (corresponding to) Figure 11 Branch road 1104)

[0220] Figure 11 Branch 1104 uses scene information to calculate the similarity relationship between two storyboard segments. For example, branch 1104 can use the last layer embedding in the Swin-Transformer model in the first module 410 to calculate the scene information in the current storyboard, and then determine whether the two storyboards are in the same scene based on the scene information, thereby deciding whether the two storyboards should be merged.

[0221] Please refer to Figure 18 and Figure 19 ,in, Figure 18 The diagram shows the network structure of the scene classification involved in this application. Figure 19 This application illustrates a scenario involving information; as shown in the diagram. Figure 18 and Figure 19 As shown, for each storyboard segment, the embedding encoding output by the Swin-Transformer model 1801 is first used for classification to obtain the scene information 1802 of the storyboard segment. For example, in Figure 19 In the example, both storyboard segments 1 and 2 are classified as scene 1 by the Swin-Transformer model, while scene segment 3 is located in scene 2. Therefore, storyboard segments 1 and 2 belong to the same scene and can be merged into the same storyline.

[0222] E) Judgment based on bullet screen data (corresponding to) Figure 11 Branch road 1105)

[0223] Figure 11 Branch 1105 uses bullet screen data to calculate the relationship between different scenes. For example, usually, when the peak of the bullet screen curve (i.e. the peak number of bullet screens) is relatively high, the content in the current time segment is more exciting. Usually, such time segments belong to the same storyline. If the time segment where such bullet screen data peak is located spans two scenes, then these two scene segments are likely to belong to the same storyline. In the process of dividing the storyline, we can consider classifying multiple scene segments that are spanned by a single bullet screen peak into the same storyline.

[0224] Figure 20 This application illustrates a diagram of the bullet screen data involved. Please refer to the diagram provided. Figure 20The overall trend of the bullet comment curve shown generally follows the plot development. The trend of the bullet comment curve (representing the number of bullet comments at a corresponding point in time) is strongly correlated with the plot, especially for popular TV dramas, where the amount of bullet comment data is relatively large. As shown in Figure 20, the entire bullet comment curve 2001 covers three storyboard segments. There is a clear trough in the bullet comment curve between storyboard segment 2 and storyboard segment 3. However, within the time period of storyboard segment 1 and storyboard segment 2, the height of the entire bullet comment curve does not have a clear trough. Based on this, it can be determined that the trough between storyboard segment 2 and storyboard segment 3 is the point where the plot is divided. Storyboard segment 1 and storyboard segment 2 belong to the same storyline, while storyboard segment 3 and storyboard segment 2 belong to different storylines.

[0225] Module 3.430: Fusion and Regularization Module

[0226] The third module 430 receives the storyline dividing points (dividing times) determined by the second module 420 through multimodal and bullet screen data. Then, the third module 430 merges the calculated storyline dividing points. If there is overlap between the time periods divided by the dividing points of storylines determined by different modalities, the different storylines need to be merged to obtain a larger time segment of the storyline. After merging the five timelines multiple times, the third module 430 obtains the final time segmentation of the storyline, ultimately determining the time of the storyline dividing points. Please refer to [reference needed]. Figure 21 It shows a schematic diagram of merging and regularization involved in this application, which illustrates the method of merging five timelines; such as Figure 21 As shown, for the storyline dividing points obtained from the five branches, a dividing point 2101 is determined as the final storyline dividing point. Here, the dividing point 2101 is the time point at which all five branches are determined as storyline dividing points.

[0227] The solution presented in this application, when dealing with various types of film and television videos, can automatically and quickly locate all plot dividing points in the entire video based on the video's inherent modality and combined with the video's bullet screen data. This allows the entire video to be divided according to different plot points, providing precise plot segmentation time for downstream video creation and video structuring tasks. The solution presented in this application has the following characteristics:

[0228] Firstly, this solution constructs a fully automated method for locating plot segment boundaries in film and television. The entire process enables completely industrial-grade location of plot boundaries, eliminating manual operation and intervention, greatly improving the efficiency of the entire system, and reducing the cost of the entire video structured analysis process. Secondly, this solution uses fully automated calculations based on model standardization, avoiding discrepancies in location results caused by subjective differences among annotators, thus standardizing the timing of plot boundary points throughout the system.

[0229] This solution uses an improved Swin-Transformer to locate the boundaries of video scenes. The model takes two adjacent video frames as input and automatically calculates the similarity and correlation between the two frames. This method, which takes adjacent original frames as input and automatically calculates the final correlation, avoids the problem that the grayscale histogram method cannot locate the boundaries due to the color of the images. At the same time, based on the Swin-Transformer, it can learn better image representations, making the whole system more accurate in locating the boundaries of the scenes.

[0230] This innovative solution constructs a method for locating the dividing points of complete plot segments in film and television dramas based on multimodal complementarity judgment. Through the complementary correlation between different modalities, this system is applicable to the plot division of various types of film and television videos. Furthermore, because this solution is designed as a multimodal complementarity system, it can handle situations where modalities are missing in the video—that is, when visual or audio modalities are missing from the film or television video. Even then, this solution can still accurately locate the plot dividing points in the video using the modalities that remain. Therefore, this solution has high versatility and robustness, is applicable to different types of film and television videos, and can also solve the problem of modal missingness in videos.

[0231] This case innovatively incorporates bullet screen data as a data modality into the entire system as a complementary judgment for locating plot dividing points in film and television videos. This allows users to input their real feelings and information about the direction of the plot into the system. Bullet screen data, to a certain extent, reflects the audience's judgment on the content of the film and television drama. Therefore, the trend of the bullet screen data curve is strongly correlated with the changes in the plot of the film and television drama. Thus, the bullet screen curve sent by users can assist the entire system in locating plot dividing points. This mechanism makes the plot segments segmented by the entire system more in line with the actual user's preferences and feelings, improves the performance of the entire system in plot judgment, and is more conducive to the practical implementation of the system.

[0232] This solution innovatively integrates voice and dialogue text modalities in its multimodal approach as features for plot analysis. Since voice is often accompanied by dialogue text, these two modalities are strongly correlated. Therefore, integrating them complements each other. Furthermore, the correlation of dialogue text content allows the system to more accurately segment plots, particularly in scenes involving phone calls. These scenes often switch frequently, especially in scenarios where two characters are on the phone. While there may be separation in the voice aspect, the dialogue text modal shows a strong correlation, enabling the system to determine that these back-and-forth phone call scenes belong to the same plot. This solves the problem of traditional methods failing to accurately calculate plot boundaries in such scenarios.

[0233] Figure 22 This is a structural block diagram of a video processing apparatus provided in an exemplary embodiment of this application, such as... Figure 22 As shown, the device includes the following parts:

[0234] The segment acquisition module 2201 is used to acquire multiple scene segments in the target video; the scene segment is composed of at least two consecutive video image frames, corresponding to the same shot video segment;

[0235] Modal feature acquisition module 2202 is used to acquire modal feature information of multiple modes of each of the multiple scene segments;

[0236] The correlation acquisition module 2203 is used to acquire the correlation of each pair of adjacent story segments in each of the multiple modal features based on the modal feature information of each pair of adjacent story segments in each story segment. The correlation is used to indicate whether the two adjacent story segments belong to the same storyline.

[0237] The plot dividing point acquisition module 2204 is used to acquire the plot dividing point of the target video based on the correlation between each two adjacent plot segments in each of the multiple modalities; the plot dividing point is the dividing point between different storylines in the target video.

[0238] In one possible implementation, the modal feature information of the multiple modalities includes at least one of the following:

[0239] Background music information corresponding to the background music modality;

[0240] Human voice modality corresponds to human voice speech information;

[0241] The dialogue text information corresponding to the dialogue modality;

[0242] Character information corresponding to the character modality;

[0243] Scene information corresponding to scene modality;

[0244] And the bullet screen information corresponding to the bullet screen mode.

[0245] In one possible implementation, the plot dividing point acquisition module 2204 is used for,

[0246] For adjacent first and second scene segments in each scene segment, in response to the fact that the correlation between the first and second scene segments in the multiple modalities meets a specified condition, the boundary point between the first and second scene segments is taken as a plot boundary point of the target video.

[0247] In one possible implementation, the plot dividing point acquisition module 2204 is used for,

[0248] The correlation between the first and second storyboard segments across all modalities indicates that the first and second storyboard segments do not belong to the same storyline;

[0249] Among the multiple modalities, the number of modalities corresponding to the correlation indicator that the first storyboard segment and the second storyboard segment do not belong to the same storyline is not less than the number threshold.

[0250] Among the various modalities, the sum of the importance scores of modalities that indicate the first storyboard segment and the second storyboard segment do not belong to the same storyline is not less than a score threshold; each of the various modalities corresponds to its own importance score.

[0251] In one possible implementation, the modal feature information responding to the multiple modalities includes background music information corresponding to the background music modality, and the modal feature acquisition module 2202 is used to,

[0252] Obtain the background music for the first storyboard segment and the background music for the second storyboard segment; the first storyboard segment and the second storyboard segment are two adjacent storyboard segments among the plurality of storyboard segments;

[0253] Audio features are extracted from the background music of the first storyboard segment to obtain the background music information of the first storyboard segment;

[0254] Audio features are extracted from the background music of the second storyboard segment to obtain the background music information of the second storyboard segment.

[0255] The correlation acquisition module 2203 is used for,

[0256] Calculate the first similarity between the background music information of the first storyboard segment and the background music information of the second storyboard segment;

[0257] Based on the first similarity, the first correlation between the first scene segment and the second scene segment under the background music modality is obtained.

[0258] In one possible implementation, the modal feature information responding to the multiple modalities includes human voice speech information corresponding to the human voice modality, and dialogue text information corresponding to the dialogue modality. The modal feature acquisition module 2202 is used to...

[0259] Acquire the voice recordings of the first and second scene segments; the first and second scene segments are two adjacent scene segments among the plurality of scene segments.

[0260] Audio features are extracted from the human voice in the first storyboard segment to obtain the human voice information of the first storyboard segment.

[0261] Audio features are extracted from the human voice in the second storyboard segment to obtain the human voice information of the second storyboard segment.

[0262] Get the dialogue text of the first storyboard segment and the dialogue text of the second storyboard segment;

[0263] Text features are extracted from the dialogue text of the first storyboard segment to obtain the dialogue text information of the first storyboard segment;

[0264] Text features are extracted from the dialogue text of the second storyboard segment to obtain the dialogue text information of the second storyboard segment;

[0265] The correlation acquisition module 2203 is used for,

[0266] Calculate the second similarity between the human voice information of the first storyboard segment and the human voice information of the second storyboard segment;

[0267] Calculate the third similarity between the dialogue text information of the first storyboard segment and the dialogue text information of the second storyboard segment;

[0268] Based on the second similarity and the third similarity, the second correlation between the first scene segment and the second scene segment under the human voice modality and the dialogue modality is obtained.

[0269] In one possible implementation, the modal feature information responding to the multiple modalities includes character information corresponding to the character modality, and the character information is used to indicate the character contained in the corresponding storyboard segment. The modal feature acquisition module 2202 is used to...

[0270] Character identification is performed on the first and second storyboard segments respectively to obtain character information for the first and second storyboard segments; the first and second storyboard segments are two adjacent storyboard segments among the plurality of storyboard segments.

[0271] The correlation acquisition module 2203 is used for,

[0272] Calculate the third similarity between the character information of the first storyboard segment and the character information of the second storyboard segment;

[0273] Based on the third similarity, the third correlation between the first storyboard segment and the second storyboard segment under the character modality is obtained.

[0274] In one possible implementation, the modal feature information responding to the multiple modalities includes scene information corresponding to the scene modality, and the scene information is used to indicate the scene type of the corresponding storyboard segment. The modal feature acquisition module 2202 is used to...

[0275] Scene recognition is performed on the first and second storyboard segments respectively to obtain scene information of the first and second storyboard segments; the first and second storyboard segments are two adjacent storyboard segments among the plurality of storyboard segments.

[0276] The correlation acquisition module 2203 is used to acquire the fourth correlation between the first segment and the second segment in the scene modality based on the scene type indicated by the scene information of the first segment and the scene type indicated by the scene information of the second segment.

[0277] In one possible implementation, the modal feature information responding to the multiple modalities includes bullet screen information corresponding to the bullet screen modality, and the modal feature acquisition module 2202 is used to,

[0278] Based on the number of bullet comments at each time point in the first and second scene segments, a bullet comment number curve showing the change of bullet comment number over time is obtained, which serves as the bullet comment information for the first and second scene segments; the first and second scene segments are two adjacent scene segments among the plurality of scene segments.

[0279] The correlation acquisition module 2203 is used to acquire the fifth correlation between the first segment and the second segment in the barrage mode based on the barrage quantity curve.

[0280] In one possible implementation, the fragment acquisition module 2201 is used for,

[0281] Obtain the image similarity between any two adjacent video image frames in each video image frame of the target video;

[0282] Based on the image similarity between any two adjacent video image frames in the target video, the target video is divided into multiple storyboard segments.

[0283] In one possible implementation, the fragment acquisition module 2201 is used for,

[0284] The first video image frame is processed by an image feature extraction network to obtain the first image features output by at least two specified network layers in the image feature extraction network.

[0285] The second video image frame is processed by the image feature extraction network to obtain the second image features output by the at least two specified network layers respectively; the first video image frame and the second video image frame are two adjacent video image frames in each video image frame;

[0286] Based on the first image features and the second image features, the image similarity between the first video image frame and the second video image frame is obtained.

[0287] It should be noted that the video processing apparatus provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video processing apparatus and video processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0288] Figure 23 A schematic diagram of a computer device provided in an exemplary embodiment of this application is shown. The description will be based on an example where the computer device is a server.

[0289] Server 2300 includes a Central Processing Unit (CPU) 2301, a system memory 2304 including Random Access Memory (RAM) 2302 and Read Only Memory (ROM) 2303, and a system bus 2305 connecting the system memory 2304 and the CPU 2301. Server 2300 also includes a mass storage device 2306 for storing the operating system 2313, application programs 2314, and other program modules 2315.

[0290] Mass storage device 2306 is connected to central processing unit 2301 via a mass storage controller (not shown) connected to system bus 2305. Mass storage device 2306 and its associated computer-readable media provide non-volatile storage for server 2300. That is, mass storage device 2306 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0291] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 2304 and mass storage device 2306 described above can be collectively referred to as memory.

[0292] According to various embodiments of this application, server 2300 can also be connected to a remote computer on a network, such as the Internet. That is, server 2300 can be connected to network 2312 via network interface unit 2311 connected to system bus 2305, or it can also use network interface unit 2311 to connect to other types of networks or remote computer systems (not shown).

[0293] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU to perform the functions described above. Figure 2 or Figure 3 All or part of the steps of the method shown in the embodiments.

[0294] Embodiments of this application also provide a computer device, which includes a processor and a memory, wherein the memory stores a computer program that is loaded and executed by the processor to implement the video processing methods provided in the above-described method embodiments.

[0295] Embodiments of this application also provide a computer-readable storage medium storing a computer program that is loaded and executed by a processor to implement the video processing methods provided in the above-described method embodiments.

[0296] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the video processing methods described in the above embodiments.

[0297] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0298] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0299] Before and during the collection of user data, this application can display a prompt interface, pop-up window, or output voice prompt information. This prompt interface, pop-up window, or voice prompt information is used to inform the user that their relevant data is being collected. This ensures that the application only begins to execute the relevant steps of collecting user data after obtaining confirmation from the user regarding the prompt interface or pop-up window. Otherwise (i.e., if no confirmation is obtained from the user regarding the prompt interface or pop-up window), the relevant steps of collecting user data are terminated, and the user's relevant data is not collected.

[0300] In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use and processing of relevant user data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0301] The aforementioned user data includes information (including but not limited to user accounts and user attribute information), data (including but not limited to user operation information and user access records), and signal data. For example, the user data involved in this application was obtained with full authorization.

[0302] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A video processing method, characterized in that: The method comprises: Acquire multiple storyboard segments in the target video; the storyboard segments are composed of at least two consecutive video image frames and correspond to video segments of the same shot; Acquire modal feature information of multiple modes of each storyboard segment in the plurality of storyboard segments; Based on the modal feature information of multiple modes of each two adjacent storyboard segments in each storyboard segment, obtaining the correlation of each two adjacent storyboard segments in each storyboard segment in each of the multiple modes, wherein the correlation is used to indicate whether the two adjacent storyboard segments belong to the same storyline; Based on the correlation between every two adjacent storyboard segments in each of the multiple modes, the plot demarcation point of the target video is obtained; the plot demarcation point is the demarcation point between different storylines in the target video.

2. The method according to claim 1, characterized in that The modal feature information of the multiple modalities includes at least one of the following information: Background music information corresponding to the background music mode; The human voice speech information corresponding to the human voice modality; The dialogue text information corresponding to the dialogue mode; Persona information corresponding to the persona modality; Scene information corresponding to the scene modality; And the barrage information corresponding to the barrage mode.

3. The method according to claim 1 or 2, characterized in that: The obtaining of the plot demarcation point of the target video based on the correlation between each two adjacent storyboard segments in each of the multiple modes includes: For the adjacent first storyboard fragment and the second storyboard fragment in each storyboard fragment, in response to the association between the first storyboard fragment and the second storyboard fragment in the multiple modalities satisfying a specified condition, a dividing point between the first storyboard fragment and the second storyboard fragment is used as a plot dividing point of the target video.

4. The method according to claim 3, characterized in that The specified conditions include: The associations between the first storyboard segment and the second storyboard segment in the multiple modes all indicate that the first storyboard segment and the second storyboard segment do not belong to the same storyline; or, Among the multiple modalities, the number of modalities in which the corresponding relevance indicates that the first storyboard segment and the second storyboard segment do not belong to the same storyline is not less than a quantity threshold; or, Among the multiple modalities, the sum of the importance scores of the modalities in which the corresponding correlation indicates that the first storyboard fragment and the second storyboard fragment do not belong to the same storyline is not less than a score threshold; the multiple modalities correspond to their respective importance scores.

5. The method according to claim 2, characterized in that: In response to the modal feature information of the multiple modalities including background music information corresponding to the background music modality, The obtaining of modal feature information of multiple modes of each storyboard segment in the plurality of storyboard segments includes: Acquire background music of a first storyboard segment and background music of a second storyboard segment; the first storyboard segment and the second storyboard segment are two adjacent storyboard segments among the plurality of storyboard segments; Extracting audio features of the background music of the first storyboard segment to obtain background music information of the first storyboard segment; Extracting audio features of the background music of the second storyboard segment to obtain background music information of the second storyboard segment; The obtaining, based on the modal feature information of multiple modes of each two adjacent storyboard segments in each storyboard segment, the correlation between each two adjacent storyboard segments in each storyboard segment in each of the multiple modes includes: Calculating a first similarity between background music information of the first storyboard segment and background music information of the second storyboard segment; Based on the first similarity, a first correlation between the first storyboard segment and the second storyboard segment in the background music mode is obtained.

6. The method according to claim 2, characterized in that In response to the modal feature information of the multiple modalities including the human voice information corresponding to the human voice modality and the dialogue text information corresponding to the dialogue modality, The obtaining of modal feature information of multiple modes of each storyboard segment in the plurality of storyboard segments includes: Acquire a human voice of a first storyboard segment and a human voice of a second storyboard segment; the first storyboard segment and the second storyboard segment are two adjacent storyboard segments among the plurality of storyboard segments; Performing audio feature extraction on the human voice of the first storyboard segment to obtain human voice information of the first storyboard segment; Performing audio feature extraction on the human voice of the second storyboard segment to obtain human voice information of the second storyboard segment; Obtain the dialogue text of the first storyboard segment and the dialogue text of the second storyboard segment; Performing text feature extraction on the dialogue text of the first storyboard segment to obtain dialogue text information of the first storyboard segment; Performing text feature extraction on the dialogue text of the second storyboard segment to obtain dialogue text information of the second storyboard segment; The obtaining, based on the modal feature information of multiple modes of each two adjacent storyboard segments in each storyboard segment, the correlation between each two adjacent storyboard segments in each storyboard segment in each of the multiple modes includes: Calculating a second similarity between the human voice information of the first storyboard segment and the human voice information of the second storyboard segment; Calculating a third similarity between the dialogue text information of the first storyboard segment and the dialogue text information of the second storyboard segment; Based on the second similarity and the third similarity, a second correlation between the first storyboard segment and the second storyboard segment in the vocal mode and the dialogue mode is obtained.

7. The method according to claim 2, characterized in that In response to the modal feature information of the multiple modalities including character information corresponding to the character modality, and the character information is used to indicate the character included in the corresponding storyboard segment, The obtaining of modal feature information of multiple modes of each storyboard segment in the plurality of storyboard segments includes: Performing character recognition on the first storyboard segment and the second storyboard segment respectively to obtain character information of the first storyboard segment and character information of the second storyboard segment; the first storyboard segment and the second storyboard segment are two adjacent storyboard segments among the multiple storyboard segments; The obtaining, based on the modal feature information of multiple modes of each two adjacent storyboard segments in each storyboard segment, the correlation between each two adjacent storyboard segments in each storyboard segment in each of the multiple modes includes: Calculating a third similarity between the character information of the first storyboard segment and the character information of the second storyboard segment; Based on the third similarity, a third association between the first storyboard clip and the second storyboard clip in the character role mode is obtained.

8. The method according to claim 2, characterized in that: In response to the modality feature information of the multiple modalities including scene information corresponding to the scene modality, and the scene information is used to indicate the scene type of the corresponding storyboard segment, The obtaining of modal feature information of multiple modalities of each storyboard segment in the plurality of storyboard segments includes: Performing scene recognition on the first storyboard segment and the second storyboard segment respectively to obtain scene information of the first storyboard segment and scene information of the second storyboard segment; the first storyboard segment and the second storyboard segment are two adjacent storyboard segments among the multiple storyboard segments; The obtaining, based on the modal feature information of multiple modes of each two adjacent storyboard segments in each storyboard segment, the correlation between each two adjacent storyboard segments in each storyboard segment in each of the multiple modes includes: Based on the scene type indicated by the scene information of the first storyboard fragment and the scene type indicated by the scene information of the second storyboard fragment, a fourth association between the first storyboard fragment and the second storyboard fragment in the scene modality is obtained.

9. The method according to claim 2, characterized in that: In response to the modal feature information of the multiple modes including the barrage information corresponding to the barrage mode, The obtaining of modal feature information of multiple modalities of each storyboard segment in the plurality of storyboard segments includes: Based on the number of bullet comments at each time point in the first storyboard segment and the second storyboard segment, a bullet comment number curve showing the number of bullet comments changing with time is obtained as the bullet comment information of the first storyboard segment and the second storyboard segment; the first storyboard segment and the second storyboard segment are two adjacent storyboard segments among the multiple storyboard segments; The obtaining, based on the modal feature information of multiple modes of each two adjacent storyboard segments in each storyboard segment, the correlation between each two adjacent storyboard segments in each storyboard segment in each of the multiple modes includes: Based on the barrage quantity curve, the fifth correlation between the first storyboard segment and the second storyboard segment in the barrage mode is obtained.

10. The method according to any one of claims 1 to 9, characterized in that: The step of obtaining a plurality of storyboard segments in a target video includes: Obtaining the image similarity between every two adjacent video image frames in each video image frame of the target video; The target video is divided based on the image similarity between every two adjacent video image frames in each video image frame of the target video to obtain the multiple storyboard segments.

11. The method according to claim 10, characterized in that The obtaining of the image similarity between each two adjacent video image frames in each video image frame of the target video includes: Processing the first video image frame through an image feature extraction network to obtain first image features respectively output by at least two designated network layers in the image feature extraction network; Processing the second video image frame through the image feature extraction network to obtain second image features respectively output by the at least two designated network layers; the first video image frame and the second video image frame are two adjacent video image frames in the respective video image frames; Based on the first image feature and the second image feature, an image similarity between the first video image frame and the second video image frame is obtained.

12. A video processing device, characterized in that: The device comprises: A segment acquisition module is used to acquire multiple storyboard segments in a target video; the storyboard segments are composed of at least two consecutive video image frames and correspond to video segments of the same shot; A modal feature acquisition module, used to acquire modal feature information of multiple modes of each storyboard segment in the plurality of storyboard segments; a correlation acquisition module, configured to acquire, based on the modal feature information of multiple modalities of each two adjacent storyboard segments in each storyboard segment, the correlation of each two adjacent storyboard segments in each storyboard segment in each of the multiple modalities, wherein the correlation is used to indicate whether the two adjacent storyboard segments belong to the same storyline; A plot demarcation point acquisition module is used to acquire the plot demarcation point of the target video based on the correlation between each two adjacent storyboard segments in each of the multiple modes; the plot demarcation point is the demarcation point between different storylines in the target video.

13. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the video processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the video processing method according to any one of claims 1 to 11.

15. A computer program product, characterized in that It comprises computer instructions, and when the computer instructions are executed by a processor, the video processing method according to any one of claims 1 to 11 is implemented.

Images