Bullet screen data processing method and device, readable medium and program product

Abstract

The invention provides a bullet screen data processing method and device, electronic equipment, a computer readable medium and a computer program product. The method comprises the following steps: acquiring bullet screen data and a video file of a to-be-processed video; if the to-be-processed video is edited, mapping the bullet screen time axis of the original video and the bullet screen time axis of the edited video and removing bullet screens with unmatched contents to form a candidate bullet screen set, and if the to-be-processed video is not edited, taking the bullet screen data as the candidate bullet screen set; performing quality scoring on each bullet screen in the candidate bullet screen set based on multiple dimensions, and performing bullet screen screening according to scores to obtain a target bullet screen set; and performing display adjustment processing on each bullet screen based on the obtained target bullet screen set, the display adjustment processing including density control processing and time adjustment processing. The invention aims to solve the problems of bullet screen dislocation, mismatching of context and video frames, low bullet screen quality and the like in the existing technical scheme.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a bullet screen data processing method and device, electronic equipment, computer readable medium and computer program product. BACKGROUND

[0002] In the traditional bullet screen system of video content, the bullet screen is closely connected with the playing time sequence of the original video. This strong coupling relationship can work normally when the video is not edited. However, once the original video is edited, such as deleting a segment, rearranging the order or splicing with other videos, the original bullet screen data will have many problems. Specifically, the timestamp of the bullet screen no longer corresponds to the current playing content, resulting in time dislocation of the bullet screen. The bullet screen originally related to a specific scene or dialogue may appear in a completely unrelated picture. In addition, the semantics of the bullet screen will also drift due to editing, that is, the content expressed by the bullet screen no longer matches the context of the edited video, and even the context may be broken, making the bullet screen content appear abrupt or difficult to understand.

[0003] Therefore, the existing bullet screen system has a fundamental design defect, that is, it lacks a bullet screen reconstruction mechanism for edited videos. When the video frame changes, the bullet screen system cannot automatically adjust the timestamp and content of the bullet screen to ensure the consistency of the bullet screen and the video content. In addition, the existing technology cannot guarantee the quality of the bullet screen, cannot filter out the bullet screen that does not match the content of the edited video, and cannot optimize the bullet screen to improve its quality and relevance. Therefore, after the video content is edited, the bullet screen system cannot effectively adapt to the new video content, resulting in poor bullet screen experience and failing to meet the user's demand for high-quality video watching experience. SUMMARY

[0004] The present application aims to provide a bullet screen data processing method, device, electronic equipment, computer readable medium and computer program product to solve the problems of bullet screen dislocation, context mismatch with video frame and low bullet screen quality in the prior art.

[0005] In one aspect of the present application, a bullet screen data processing method is provided, wherein the method comprises: obtaining bullet screen data and a video file of a video to be processed; if the video to be processed is edited, mapping the bullet screen timeline of the original video and the edited video and eliminating the bullet screen with unmatched content to form a candidate bullet screen set, and if not edited, taking the bullet screen data as the candidate bullet screen set; scoring the quality of each bullet screen in the candidate bullet screen set based on multiple dimensions to filter the bullet screen according to the score and obtain a target bullet screen set; Based on the obtained target barrage set, display adjustment processing is performed on each barrage, including density control processing and time adjustment processing.

[0006] In an aspect of the present application, a barrage data processing apparatus is provided, wherein the apparatus comprises: A data acquisition module is configured to acquire barrage data and a video file of a video to be processed. A first barrage screening module is configured to, if the video to be processed is edited, map the barrage timeline of the original video and the edited video and eliminate barrages with unmatched content to form a candidate barrage set, and if the video to be processed is not edited, use the barrage data as the candidate barrage set. A second barrage screening module is configured to score each barrage in the candidate barrage set based on multiple dimensions to screen the barrages according to the scores to obtain a target barrage set. A barrage display adjustment module is configured to score each barrage in the candidate barrage set based on multiple dimensions to screen the barrages according to the scores to obtain a target barrage set.

[0007] In another aspect of the present application, an electronic device is provided, which comprises at least one processor and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the embodiments of the present application.

[0008] In another aspect of the present application, a computer readable storage medium is provided, which stores computer program instructions executable by a processor to implement the method of the embodiments of the present application.

[0009] In another aspect of the present application, a computer program product is provided, which comprises a computer program executable by a processor to implement the method of the embodiments of the present application.

[0010] In the scheme provided by the embodiments of the present application, for edited videos, frame-level mapping and semantic comparison mechanism are used to accurately realize the correspondence between the original video frames and the edited frames through image feature extraction and semantic similarity calculation, so as to avoid barrage time dislocation and context mismatch; and for edited videos and unedited videos, a multi-dimensional scoring mechanism is further used to screen high-quality and representative essence barrages to improve the content information density; and based on the picture semantic rhythm curve, the appearance timing of the barrage is dynamically adjusted to make it more consistent with the content rhythm and enhance the consistency of the visual perception; and a visual editing and heat map interface is provided, including a timeline and a semantic heat map, to support manual adjustment and style configuration of the barrage by the editor. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on these drawings.

[0012] Other characteristics, objects and advantages of the present application will become more apparent after reading the detailed description of the non-limiting embodiments with reference to the following drawings: Figure 1 A flow chart of a method for processing barrage data according to an embodiment of the present application is shown; Figure 2 A schematic diagram of an exemplary interactive interface for editing barrage according to an embodiment of the present application is shown; Figure 3 A structural schematic diagram of a device for processing barrage data according to an embodiment of the present application is shown; Figure 4 A structural schematic diagram of a device for processing barrage data according to an embodiment of the present application is shown;

[0013] The same or similar reference signs in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0014] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of protection of the present application.

[0015] In a typical configuration of the present application, the devices of the terminal and the service network each include one or more processors (CPU), input / output interfaces, network interfaces and memories.

[0016] The memory can include a non-permanent memory in a computer readable medium, random access memory (RAM) and / or non-volatile memory such as read only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer readable medium.

[0017] Computer-readable media includes permanent and non-permanent, removable and non-removable media implemented by any method or technology for information storage. Information can be computer program instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact discs (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disks storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0018] Figure 1 A flowchart of a method for processing barrage data according to an embodiment of the application is shown. The method includes at least steps S101, S102, S103 and S104.

[0019] In actual scenarios, the execution subject of the method can be a computing device included in a distributed computing system, or an application program running on the computing device. The computing device includes a network device, which includes but is not limited to a network host, a single network server, a plurality of network server sets or a computer set based on cloud computing, which can be used to implement part of the processing function when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, wherein cloud computing is a kind of distributed computing, which is composed of a virtual computer set by a group of loosely coupled computers.

[0020] In actual scenarios, the method according to the embodiment of the application is executed by a server of a video website or a client playing a video.

[0021] Reference Figure 1 It is explained that in step S101, the barrage data and the video file of the video to be processed are acquired.

[0022] The video to be processed can include various types of videos or video segments with barrage interaction function. For example, the video to be processed can be a video acquired from a video player of a browser or a video playing software, or a directly shot video. The type of the video to be processed can include sports event video, live video, film and television drama video, variety show video, advertisement video, etc.

[0023] The video to be processed includes a video processed by editing and a video not processed by editing.

[0024] The barrage data includes text content and appearance timestamp of each barrage in the to-be-processed video, which are basic information of barrage display. Optionally, the barrage data further includes other additional information such as font size, color, number of likes, number of replies, user level, sending end type and the like of the barrage.

[0025] Optionally, the method performs step S101 when a preset barrage reconstruction condition is met. The barrage reconstruction condition can include that the video duration exceeds a preset threshold, for example, 30 seconds, to ensure that the video content is rich enough to need reconstruction to optimize the barrage display; the number of original barrages is greater than a preset lower limit, for example, 100, to ensure that there are enough samples for statistical and optimization processing; a user manually enables a “barrage reconstruction” function switch, indicating that the user wants to optimize the barrage; the video is marked as “secondary creation” or “published after editing”, and the barrage needs to be realigned to adapt to the new video content, and the like.

[0026] In step S102, if the to-be-processed video is edited, the barrage time axis of the original video and the edited video is mapped and the barrage with unmatched content is removed to form a candidate barrage set, and if not edited, the barrage data is taken as the candidate barrage set.

[0027] Specifically, it is judged whether the to-be-processed video is edited. For example, based on the obtained video file, it is detected whether there are a pair of original video and edited video. The editing operation includes but is not limited to deleting video frames, replacing video frames and the like.

[0028] According to an embodiment, if the to-be-processed video is edited, the step S102 further includes steps S1021 to S1023.

[0029] In step S1021, the key frame feature vectors of the original video and the edited video are obtained, and the correspondence of each video frame before and after editing is determined by performing time sequence similarity matching.

[0030] Specifically, step S1021 further includes the following steps: First, key frames are uniformly extracted from the original video and the edited video at a predetermined time interval to form two key frame sets. For example, if the video duration is 120, one frame is extracted per second, and 120 key frames are obtained. These key frames will be used as input for subsequent feature extraction.

[0031] Then, the feature vectors corresponding to each key frame are extracted and the similarity between the key frames of the original video and the edited video is calculated.

[0032] Optionally, the extracted key frames are input into a pre-trained feature extraction model (e.g., a convolutional neural network model such as ResNet-50 or a visual language pre-trained model such as CLIP) to obtain fixed-dimension feature vectors as key frame feature vectors. Taking ResNet-50 as an example, each key frame generates a 2048-dimensional feature vector. These feature vectors can effectively represent the visual content of the key frames and provide data support for subsequent similarity calculation.

[0033] The similarity calculation method includes but is not limited to cosine similarity, Euclidean distance, and Jaccard similarity coefficient.

[0034] Next, based on the similarity between the key frames of the original video and the clip video, the correspondence between the two sets of key frames is determined. Specifically, the key frames of the original video are taken as a reference benchmark. For each key frame in the clip video, the most similar key frame is found according to its similarity with each key frame of the original video, thereby establishing a one-to-one mapping relationship between the key frames of the original video and the clip video.

[0035] According to an embodiment, a dynamic time warping (DTW) algorithm is used to determine the correspondence between the frames of the video before and after the clip. The dynamic time warping can handle the stretching and compression of time series and find the most suitable matching path.

[0036] Specifically, a similarity matrix is constructed based on the similarity between the key frames of the original video and the clip video. Then, the DTW algorithm is used to find the optimal alignment path on the similarity matrix to obtain the best matching relationship between the frames of the video before and after the clip.

[0037] Optionally, the alignment path generated by the DTW is post-processed to delete matching points with a similarity lower than a threshold (e.g., 0.6) and remove unmatched frame pairs. If a frame has multiple matching frames in the original video, only the one with the highest similarity is retained. For example, if the 40th frame of the original video has two matching frames in the clip video with similarities of 0.7 and 0.5, only the matching relationship with a similarity of 0.7 is retained.

[0038] According to an embodiment, a frame mapping index table is constructed based on the correspondence between the frames of the video before and after the clip, which is used for bullet screen time axis mapping.

[0039] Optionally, the frame mapping index table is generated according to the alignment path generated by the DTW. For each frame in the original video, the index information of the corresponding frame in the clip video is recorded, such as the index identifier. For example, if a frame has no matching frame in the clip video, the index is set to -1, indicating that the frame is cut off.

[0040] In step S1022, the original barrage timestamp is mapped to the timeline of the edited video according to the correspondence between each video frame before and after editing.

[0041] According to the first example of the present application, the original barrage data is represented as B = { (t_i, c_i)}, where (t_i, c_i) represents the timestamp and text content of each barrage i. According to the frame mapping index table M, the timestamp t_i of each barrage in the original barrage set B = { (t_i, c_i)} is mapped to the corresponding timestamp t_i' of the edited video. For example, the original barrage timestamp is 30 seconds, and by looking up the corresponding frame through the frame mapping index table M, the edited timestamp is 25 seconds, i.e. ti t_i' = 25 seconds.

[0042] In step S1023, for each barrage, if the context of the barrage does not match the context of the corresponding new video frame after editing, the barrage is deleted.

[0043] The way to determine whether the context of the barrage matches the context of the corresponding new video frame after editing includes: 1) similarity judgment based on visual semantic vectors of video frames; For each original frame of a barrage and its corresponding frame in the edited video, a pre-trained visual semantic model (such as CLIP or BLIP) is used to extract visual semantic vectors. Taking the CLIP model as an example, the frame image is input into the model to obtain a 512-dimensional visual semantic vector output by the model. These vectors can represent the visual content of the frame and provide data support for subsequent similarity calculation. The cosine similarity of the visual semantic vectors of the original frame and the corresponding frame after editing is calculated. If the similarity is lower than a set threshold (such as 0.6), it is determined that the context of the barrage has changed, and it is marked as a candidate for rejection. For example, the visual semantic vector of the original frame is v 1, the visual semantic vector of the corresponding frame after editing is v 2, the cosine similarity cos( v 1, v 2) = 0.5, which is lower than the threshold 0.6, so it is determined that the context does not match and the barrage is marked as a candidate for rejection.

[0044] Assume that the video to be processed is a game live video, and the original video frame is that the host is angrily hitting the keyboard due to operation failure, the expression of the host in the picture is fierce, the action is fierce, and the background also looks a little messy. At this time, there may be comments on the host's failure on the bullet screen, such as "what is this operation". However, in order to highlight the wonderful operation of the game, the video after editing replaces this frame with a picture in which the host calmly introduces the game scene, the host has a smile on his face, the action is gentle, and the background is neat and orderly. Since the visual semantic difference between the original frame and the frame after editing is obvious, the cosine similarity of the vectors extracted by the visual semantic model v 1and v 2will be very low. For example, v 1represents angry, messy emotions and actions, while v 2represents calm, orderly emotions and scenes, and the calculated cosine similarity may be much lower than the preset similarity threshold 0.6. Therefore, the context of this bullet screen has completely changed and therefore does not match the current video frame, which is marked as a candidate for deletion to avoid the appearance of bullet screens that do not match the current picture in the edited video, affecting the viewing experience of the audience.

[0045] 2) by comparing subtitles or speech to text; Optionally, the original frame and the corresponding frame after editing are subjected to subtitle extraction (for example, extracting subtitle text by OCR technology) and speech conversion (for example, converting speech to text by ASR technology) respectively to obtain two pieces of text. The extracted text is preprocessed, including removing stop words, punctuation marks, and performing stem extraction, and then the semantic similarity of the two pieces of text is calculated. Word embedding models (such as Word2Vec or BERT) can be used to convert the text into vectors, and then the cosine similarity of the vectors is calculated. If the similarity is lower than the set threshold (such as 0.7), it is determined that the context is inconsistent, and the bullet screen is marked as a candidate for deletion. For example, the subtitle of the original video frame is "the host is demonstrating new game operations", and the subtitle of the corresponding video frame after editing is "the host is introducing game characters", the semantic similarity of the two pieces of text is lower than the preset threshold 0.7, so it is determined that the context does not match, and the bullet screen in the corresponding video frame after editing is regarded as a bullet screen to be deleted.

[0046] 3) by using a pre-trained language model (such as a large language model) to infer; Optionally, the content of the bullet screen and the visual semantic description of the corresponding frame after editing are input into the large language model, and the large language model is used to determine whether the context of the bullet screen matches the context of the new video frame after editing.

[0047] Optionally, a pre-trained visual description model (such as CLIP's text generation module) is used to generate natural language descriptions for the edited frames. The bullet screen content and the generated description are concatenated into a prompt, such as: "Is it reasonable for the bullet screen 'This game is so much fun' to contain the description 'The streamer is showing a new game'? Only answer 'context match' or 'context mismatch'." The prompt is then input into a large language model, which returns either "context match" or "context mismatch". If the model returns "context mismatch" or the confidence level is below a set threshold, such as 0.8, the bullet screen is determined to have a context mismatch and is marked as a candidate for removal. For example, if the generated description is "The streamer is showing a new game" and the bullet screen is "This game is so much fun", the model returns "context match" with a confidence level of 0.9, so the bullet screen is retained; however, if the bullet screen is "This ad is so annoying", the model returns "context mismatch" with a confidence level of 0.1, so the context mismatch is determined, and the bullet screen is marked as a candidate for removal.

[0048] Continuing with the first example, after determining whether the timestamp mapping and context match, the bullet comments that were not marked as candidates for removal are retained, forming a candidate bullet comment set. B ={( ti ′, ci These comments will be further filtered in subsequent steps to ensure their quality and relevance.

[0049] The following continues to refer to... Figure 1 To explain, in step S103, each bullet comment in the candidate bullet comment set is scored based on multiple dimensions, and the bullet comments are filtered according to the scores to obtain the target bullet comment set.

[0050] According to one embodiment, the candidate bullet screen set is scored based on three dimensions: emotional intensity, opinion focus, and interaction popularity. The scores are then combined according to a set weight and the density is controlled for further screening to obtain the target bullet screen set.

[0051] The quality score for emotion intensity is used to indicate the strength of the emotion expressed in the bullet comments. Optionally, when calculating the quality score for the emotion intensity dimension, the candidate bullet comment set is... B Each comment in the barrage. ci A sentiment analysis model (such as SentiBERT) is used to calculate the quality score of the sentiment intensity dimension. For example, the quality score of sentiment intensity can range from [0, 1], where a score closer to 1 indicates a stronger sentiment. For example, the comment "That's amazing!" might receive a quality score of 0.8 for the sentiment intensity dimension, while "normal operation" might receive 0.2.

[0052] The opinion focus degree is used to represent whether the content of a barrage is consistent with the core topic or opinion discussed by the majority of barrages in the current time period, and the quality score of the opinion focus degree is used to indicate the focusing degree of the barrage.

[0053] Optionally, the quality score of the opinion focus degree dimension is calculated based on keyword clustering and semantic repetitiveness analysis. The representative opinion is identified by extracting keywords in the barrage and clustering. For example, if multiple barrages mention "operation" and "excellent", the opinion focus degree score of these barrages will be higher. The quality score of the opinion focus degree dimension can range from [0, 1], wherein the closer the score is to 1, the higher the focusing degree of the barrage.

[0054] The quality score of the interaction heat degree dimension reflects the popularity of the barrage. Optionally, when calculating the quality score of the interaction heat degree dimension, the like number, reply number, or word frequency heat of the barrage are combined to calculate. For example, a barrage is liked 100 times and replied 20 times, and its interaction heat score can be higher.

[0055] Optionally, based on the scores of the three dimensions, for each candidate barrage, the comprehensive score S is calculated according to the quality score Ei of the emotion intensity dimension, the quality score Pi of the opinion focus degree dimension, and the quality score Hi of the interaction heat degree dimension, combined with the pre-set weighting coefficients We, Wp, and Wh of the three temperatures, according to the following formula: S=We Ei+Wp Pi+Wh Hi (1) The weighting coefficients We, Wp, and Wh can be adjusted according to the needs of the creators to balance the importance of different dimensions. For example, if the creator wants to pay more attention to the emotion intensity, We can be set to 0.5, Wp to 0.3, and Wh to 0.2.

[0056] Those skilled in the art should be familiar that in addition to the methods mentioned in this application, the quality of the barrage can be scored based on various dimensions and various calculation methods. Those skilled in the art can select appropriate methods to calculate the quality of the barrage based on actual needs.

[0057] According to an embodiment, the target barrage set is obtained through a preliminary screening stage and a sorting stage. Specifically, in the preliminary screening stage, a minimum threshold is set for the three-dimensional score of each barrage, and low-quality barrages are filtered out. For example, the following screening conditions are set: the quality score of the emotional intensity dimension is greater than 0.5; the quality score of the viewpoint focus degree dimension is greater than 0.3; and the quality score of the interaction heat degree dimension is greater than 0.2. Barrages that do not meet any of the above conditions will be removed. Then, in the sorting stage, the barrages after the preliminary screening are sorted in descending order of the comprehensive score S, and the top N barrages are selected as the refined barrages according to the set density requirement. The density requirement can be adjusted according to the creation requirements, for example: low density: 2 barrages per second; medium density: 5 barrages per second; high density: 10 barrages per second.

[0058] The following will continue to refer to Figure 1 It is explained that in step S104, based on the obtained target barrage set, display adjustment processing is performed on each barrage, including density control processing and time adjustment processing.

[0059] The step S104 includes steps S1041 and S1042.

[0060] In step S1041, by performing density control processing, interval insertion or compression processing is performed on the barrage over-dense area.

[0061] The density control processing is used to limit the number of barrage lines displayed per unit time and perform interval insertion or compression processing on the barrage that exceeds the limit.

[0062] During video playback, barrages are usually displayed in a rolling or static form in a specific area of the screen. If the number of barrages is too large in the same time period, it may cause barrages to block each other, affecting the reading experience of the audience. By track density control, the number of barrage lines displayed per second is limited (for example, a maximum of 8 lines per second for large screens and a maximum of 5 lines per second for small screens), and interval insertion or compression processing is performed on the barrage that exceeds the limit, ensuring that the barrage is evenly distributed on the screen and avoiding blocking.

[0063] In step S1042, by performing time adjustment processing, the appearance time of the barrage is dynamically adjusted according to the picture semantic rhythm curve.

[0064] Specifically, the process of the time adjustment processing includes: analyzing the semantic changes of the video (for example, by extracting frame-level visual semantic vectors and calculating the semantic vector difference of adjacent frames), generating a graph of the semantic change rate on the time axis. Optionally, the graph is a semantic change rate graph on the time axis, and high rhythm areas and low rhythm areas in the video can be identified in the graph. Then, according to the generated graph, the appearance time of the barrage is adjusted, so that the barrage is inserted in advance in the high rhythm area with fast semantic change, and the barrage is inserted in delay in the low rhythm area with slow semantic change.

[0065] Optionally, in step S1042, the image semantic vectors of a plurality of key video frames are extracted from the edited video frame by frame or every predetermined time interval, such as 0.5 seconds, such as using CLIP to extract image semantic vectors. The image semantic vector can represent the visual content of each frame, providing basic data for subsequent semantic change analysis. Then, the difference between the semantic vectors of adjacent frames is calculated, for example, using cosine distance as the measurement standard. If the difference between the semantic vectors of adjacent frames exceeds a predetermined threshold, it indicates that the semantic of the video frame picture has changed significantly, such as scene switching or the occurrence of important events, which is in a high semantic rhythm area; on the contrary, if the difference is below the predetermined threshold, it indicates that the picture semantic is relatively stable, which is in a low semantic rhythm area.

[0066] Based on the obtained graph of the semantic change rate, the appearance time tj' of each candidate barrage (tj', cj) is fine-tuned. Specifically, for each barrage, the semantic rhythm value corresponding to the time point tj' of the frame where it is located is checked. If the time point is in the high semantic rhythm area, i.e., the video content changes rapidly, the appearance time of the barrage is advanced so that the barrage can appear more timely, which is represented by the following formula (2): tj'' = tj' + δt (3) δt (2) Wherein, tj'' represents the fine-tuned appearance time, δt represents the fine-tuning amplitude, the fine-tuning amplitude δt is set in the range of 0.5 to 2 seconds, and the specific value will be dynamically adjusted according to the height of the semantic rhythm value, to ensure that the appearance time of the barrage matches the rhythm of the video content.

[0067] On the contrary, if the time point is in the low semantic rhythm area, i.e., the video content changes slowly, the appearance time of the barrage will be delayed, so that the appearance of the barrage is more natural, which is represented by the following formula (3): tj'' = tj' + δt (3) Optionally, in order to more accurately determine the high and low of the semantic rhythm value, after obtaining the overall semantic change rate curve, the highest part of the value in the semantic change rate curve is counted, such as the first 20% of the points, and is marked as a "high rhythm point", and the remaining points are marked as "low rhythm points". This dynamic adjustment mechanism based on data driving can accurately adapt the display time of the barrage to the rhythm change of the video content, improve the coordination of the barrage and the video content, and enhance the viewing experience of the audience.

[0068] Optionally, special styles such as "selected label", "emotion color", "like number" and the like are added to the screened barrage to enhance the visual effect and information expression of the barrage. For example, a red frame is added to the barrage with strong emotion, and a like icon is added to the barrage with high interactivity to improve the viewing experience of the audience.

[0069] According to an embodiment, the method provides an interactive interface for adjusting the barrage. The method of the embodiment further includes step S105.

[0070] In step S105, the barrage track file of the video to be processed is automatically updated in response to the editing operation of the editor on the barrage to be displayed in the interactive interface.

[0071] Optionally, a timeline and a semantic heat map are displayed in parallel in the interactive interface. The timeline is used to present the distribution, occurrence time and density peak of the arranged barrage frame by frame. The semantic heat map is used to superimpose the picture rhythm strength of the corresponding area in the form of color scale, so that the editor can locate the high rhythm area and the barrage crowded area at a glance.

[0072] Optionally, through the interactive interface, the editor can directly edit the screened barrage to be displayed in various ways such as dragging, framing or shortcut keys, such as moving the barrage track up and down, sliding the occurrence time left and right, and one-key removing items that do not conform to the context. The interactive interface supports inserting new barrage, such as guiding interactive text, Easter egg or advertising voiceover. The new content shares the same scoring and density checking engine with the existing barrage, ensuring that the overall arrangement is not damaged.

[0073] Optionally, the font size, color, stroke, transparency, display area (top, bottom, scrolling, fixed) and life cycle of a single or batch of barrages can be set through the interactive interface. All changes are real-time rendered and previewed, and what you see is what you get.

[0074] The editing operation performed through the interactive interface will be synchronized to the background barrage track file to record the new timestamp, content, style parameters and region information of the edited barrage, and automatically re-run the density and rhythm check. If there is an overrun or obstruction, a pop-up window will remind and give a suggested value. After the check is passed, the barrage file (such as JSON / ASS) that meets the rendering engine specification can be exported one key.

[0075] For example, referring to the schematic diagram of the interactive interface shown in Figure 2 The clipper can conveniently manually adjust the appearance time of the displayed barrage and set the number of barrage lines, transparency and text style.

[0076] The embodiments of the present application provide flexible subsequent barrage rendering methods based on the reconstructed barrage, i.e., the target barrage set, to meet the needs of different application scenarios.

[0077] Optionally, the reconstructed barrage is fused with the clip segment to generate a final video file. This method is suitable for scenarios where the barrage needs to be directly embedded in the video, ensuring that the audience can see the barrage that highly matches the content when watching the video, thereby improving the overall viewing experience. The fused video file can be directly used for publishing and distribution, and the audience can see the optimized barrage effect without additional operations when playing.

[0078] Optionally, the barrage layer and the video segment are saved separately. In this mode, the barrage layer exists as an independent file, while the video segment remains unchanged. When other audiences watch the video on the playback end, the system will render the barrage layer separately on the playback end. This method provides higher flexibility, allowing dynamic loading and rendering of the barrage according to needs on different playback devices or platforms, supporting more complex interactions and personalized settings.

[0079] The method of the embodiment of the application is described below in combination with an example. Taking a certain original live video as an example, a clipper selects about 15 minutes of highlights in the original video for processing. The original video contains 15,000 pieces of bullet screen, which are closely coupled with the playback timing of the original video. In the processing process, the key frame feature sequences of the original video and the processed video are extracted respectively, the correspondence of each frame is determined through time sequence similarity matching, and a frame mapping index table is generated. After this process, 9,200 pieces of valid mapping bullet screen are retained, and the bullet screen that is time dislocated or context changed due to the clipping is removed. Subsequently, the system performs multi-dimensional scoring on the retained bullet screen, and selects about 500 pieces of essence bullet screen through emotion intensity identification, viewpoint focus identification and interactive heat scoring. In the 15-minute time period, the number of bullet screens is controlled to be within 30 per minute through bullet screen density control, ensuring uniform distribution of bullet screens on the screen and avoiding occlusion. The clipper manually adjusts through the visual interface provided by the system, deletes some repeated bullet screens, and adds 20 pieces of guide comments, further optimizing the bullet screen content. Finally, the system exports a.json format bullet screen file and embeds it in the processed video, realizing the unity of content consistency and visual optimization, and significantly improving the viewing experience of the audience.

[0080] According to the method of the embodiment of the application, for the processed video, the frame-level mapping and semantic comparison mechanism accurately realizes the correspondence between the original video frame and the processed frame through image feature extraction and semantic similarity calculation, avoiding time dislocation and context mismatch of the bullet screen; and for the processed video and the video without processing, the multi-dimensional scoring mechanism is further used to select high-quality and representative essence bullet screen, improving the content information density; and based on the picture semantic rhythm curve, the appearance timing of the bullet screen is dynamically adjusted to make it more consistent with the content rhythm and enhance the visual consistency; and a visual editing and heat map interface is provided, including a time axis and a semantic heat map, supporting the clipper to manually adjust and style the bullet screen.

[0081] Figure 3 The structure schematic diagram of the bullet screen data processing device according to the embodiment of the application is shown. The device comprises a data acquisition module 101, a first bullet screen filtering module 102, a second bullet screen filtering module 103 and a bullet screen display adjustment module 104.

[0082] Reference Figure 3 The data acquisition module 101 acquires the bullet screen data and the video file of the video to be processed.

[0083] The to-be-processed video can include various types of videos or video clips with a bullet screen interaction function. For example, the to-be-processed video can be a video obtained from a video player of a browser or video playing software, or a directly shot video. The types of the to-be-processed video can include sports event videos, live videos, film and television drama videos, variety show videos, advertisement videos, and the like.

[0084] The to-be-processed video includes a video that has been edited and a video that has not been edited.

[0085] The bullet screen data includes text content and an occurrence timestamp of each bullet screen in the to-be-processed video, which are basic information of bullet screen display. Optionally, the bullet screen data further includes other additional information, such as a font size, a color, a number of likes, a number of replies, a user level, a sending end type, and the like of the bullet screen.

[0086] Optionally, the operation of the data acquisition module 101 is performed when a preset bullet screen reconstruction condition is met. The bullet screen reconstruction condition can include that a video duration exceeds a preset threshold, for example, 30 seconds, to ensure that the video content is rich enough and needs to be reconstructed to optimize bullet screen display, that a number of original bullet screens is greater than a preset lower limit, for example, 100, to ensure that there are enough samples for statistical and optimization processing, that a user manually enables a “bullet screen reconstruction” function switch, indicating that the user hopes to optimize the bullet screen, that the video is marked as “secondary creation” or “after editing and publishing”, and the like, which needs to realign the bullet screen to adapt to the new video content.

[0087] If the to-be-processed video is edited, the first bullet screen filtering module 102 maps the bullet screen time axis of the original video and the edited video and eliminates the bullet screens with unmatched content to form a candidate bullet screen set. If the to-be-processed video is not edited, the first bullet screen filtering module 102 takes the bullet screen data as the candidate bullet screen set.

[0088] Specifically, it is determined whether the to-be-processed video is edited. For example, based on the obtained video file, it is detected whether there are a pair of original video and edited video. The editing operation includes, but is not limited to, deleting a video frame, replacing a video frame, and the like.

[0089] According to an embodiment, if the to-be-processed video is edited, the first bullet screen filtering module 102 further includes a frame correspondence relationship determination module, a time axis mapping module, and a bullet screen elimination module.

[0090] The frame correspondence relationship determination module obtains key frame feature vectors of the original video and the edited video, and determines the correspondence relationship of each video frame before and after editing by performing time sequence similarity matching.

[0091] Specifically, the frame correspondence relationship determination module is configured to perform the following operations: First, key frames are uniformly extracted from the original video and the edited video at predetermined time intervals to form two sets of key frames. Then, the feature vectors corresponding to each key frame are extracted, and the similarity between the key frames of the original video and the edited video is calculated.

[0092] Optionally, by inputting the extracted key frames into a pre-trained feature extraction model (for example, a convolutional neural network model such as ResNet-50, or a visual language pre-trained model such as CLIP), a fixed-dimensional feature vector is obtained as the key frame feature vector.

[0093] The similarity calculation method includes but is not limited to cosine similarity, Euclidean distance, and Jaccard similarity coefficient.

[0094] Then, based on the similarity between the key frames of the original video and the edited video, the correspondence between the two sets of key frames is determined. Specifically, the key frames of the original video can be taken as a reference benchmark. For each key frame in the edited video, the most similar key frame is found according to its similarity with each key frame of the original video, thereby establishing a one-to-one mapping relationship between the key frames of the original video and the edited video.

[0095] According to an embodiment, the frame correspondence determination module determines the correspondence between the frames of the video before and after editing using the dynamic time warping (DTW) algorithm. The dynamic time warping can handle the stretching and compression of time series and find the most suitable matching path.

[0096] Specifically, the frame correspondence determination module constructs a similarity matrix based on the similarity between the key frames of the original video and the edited video. Then, the DTW algorithm is used to find the optimal alignment path on the similarity matrix to obtain the best matching relationship between the frames of the video before and after editing.

[0097] Optionally, the frame correspondence determination module performs post-processing on the alignment path generated by the DTW, deletes matching points with a similarity lower than a threshold value (such as 0.6), and removes unmatched frame pairs. If a frame has multiple matching frames in the original video, only the one with the highest similarity is retained.

[0098] According to an embodiment, the device constructs a frame mapping index table based on the correspondence between the frames of the video before and after editing, which is used for the mapping of the bullet screen time axis.

[0099] Optionally, the frame mapping index table is generated based on the alignment path generated by the DTW. For each frame in the original video, the index information of the corresponding frame in the edited video is recorded, such as the index identifier.

[0100] The time axis mapping module maps the original bullet screen timestamp to the time axis of the edited video according to the correspondence between the frames of the video before and after editing.

[0101] Then, for each bullet screen, if the context of the bullet screen does not match the context of the corresponding new video frame after the clip, the bullet screen deletion module deletes the bullet screen.

[0102] The manner in which the device determines whether the context of the bullet screen matches the context of the corresponding new video frame after the clip includes: 1) judging based on the similarity of the visual semantic vectors of the video frames; For each bullet screen and its corresponding frame in the video after the clip, a pre-trained visual semantic model (such as CLIP or BLIP) is used to extract visual semantic vectors. Taking the CLIP model as an example, the frame image is input into the model to obtain a 512-dimensional visual semantic vector output by the model. These vectors can represent the visual content of the frame and provide data support for subsequent similarity calculations. The cosine similarity of the visual semantic vectors of the original frame and the corresponding frame after the clip is calculated. If the similarity is lower than a set threshold (such as 0.6), it is determined that the context of the bullet screen has changed, and it is marked as a candidate for rejection. For example, the visual semantic vector of the original frame is v 1, and the visual semantic vector of the corresponding frame after the clip is v 2, the cosine similarity cos( v 1, v 2)=0.5, which is lower than the threshold 0.6, so it is determined that the context does not match, and the bullet screen is marked as a candidate for rejection.

[0103] 2) judging by comparing the subtitles or speech to text; Optionally, the original frame and the corresponding frame after the clip are subjected to subtitle extraction (e.g., extracting subtitle text through OCR technology) and speech conversion (e.g., converting speech to text through ASR technology) respectively to obtain two pieces of text. The extracted text is preprocessed, including removing stop words, punctuation, performing stem extraction, etc., and then the semantic similarity of the two pieces of text is calculated. A word embedding model (e.g., Word2Vec or BERT) can be used to convert the text into a vector, and then the cosine similarity of the vectors is calculated. If the similarity is lower than a set threshold (such as 0.7), it is determined that the context is inconsistent, and the bullet screen is marked as a candidate for rejection.

[0104] 3) judging by using a pre-trained language model (such as a large language model) for inference; Optionally, the content of the bullet screen and the visual semantic description of the corresponding frame after the clip are input into a large language model, and the large language model is used to determine whether the context of the bullet screen matches the context of the corresponding new video frame after the clip.

[0105] Optionally, a pre-trained visual description model (e.g., the text generation module of CLIP) is used to generate a natural language description of the cut frame. The BGM content and the generated description are spliced into a prompt, for example: "Is the BGM 'This game is too fun' reasonable under the picture described as 'The host is showing a new game'? Only answer 'Context matching' or 'Context mismatching'." Then, the prompt is input into a large language model, and the model returns "Context matching" or "Context mismatching". If the model returns "Context mismatching" or the confidence is lower than a set threshold, such as 0.8, it is determined that the context of the BGM has not matched, and the BGM is marked as a candidate for elimination.

[0106] With reference to the foregoing Figure 3 It is explained that the second BGM screening module 103 performs quality scoring on each BGM in the candidate BGM set based on multiple dimensions to perform BGM screening according to the scores to obtain a target BGM set.

[0107] According to an embodiment, the second BGM screening module 103 calculates the quality scores of the candidate BGM set in three dimensions of emotional intensity, viewpoint focus, and interaction heat, integrates the scores according to a set weight, and controls the density to perform further screening to obtain the target BGM set.

[0108] The score of the emotional intensity is used to indicate the intensity of the emotion of the BGM. Optionally, when calculating the quality score of the emotional intensity dimension, the sentiment analysis model (such as SentiBERT) is used to calculate the quality score of the emotional intensity dimension. B The content of each BGM in the candidate BGM set ci The sentiment analysis model (such as SentiBERT) is used to calculate the quality score of the emotional intensity dimension.

[0109] The viewpoint focus is used to represent whether the content of a BGM is consistent with the core topic or viewpoint discussed by the majority of BGMs in the current time period, and the quality score of the viewpoint focus is used to indicate the focusing degree of the BGM.

[0110] Optionally, the quality score of the viewpoint focus dimension is calculated based on keyword clustering and semantic repeatability analysis. The representative viewpoint is identified by extracting keywords in the BGM and clustering.

[0111] The quality score of the interaction heat dimension reflects the popularity of the BGM. Optionally, when calculating the quality score of the interaction heat dimension, the like number, reply number, or word frequency heat of the BGM are combined to calculate.

[0112] Optionally, the second barrage screening module 103 calculates a comprehensive score S for each candidate barrage based on the scores of the three dimensions, according to the quality score Ei of the emotional intensity dimension, the quality score Pi of the viewpoint focus degree dimension, and the quality score Hi of the interactive heat dimension, in combination with the pre-set weighting coefficients We, Wp, and Wh of the three temperatures, according to formula (1).

[0113] Those skilled in the art should be familiar that, in addition to the manner mentioned in the present application, the quality of the barrage can be scored based on various dimensions and various calculation manners. Those skilled in the art can select a suitable manner to calculate the quality of the barrage based on actual needs.

[0114] According to one embodiment, the second barrage screening module 103 obtains the target barrage set through the preliminary screening stage and the sorting stage. Specifically, in the preliminary screening stage, a minimum threshold is set for the score of each dimension of the barrage, and low-quality barrages are filtered out.

[0115] The following will continue to refer to Figure 3 The barrage display adjustment module 104 performs display adjustment processing on each barrage based on the obtained target barrage set, and the display adjustment processing includes density control processing and time adjustment processing.

[0116] The barrage display adjustment module 104 includes a density control module and a time adjustment module.

[0117] The density control module performs density control processing to perform interval insertion or compression processing on the barrage dense area.

[0118] The density control processing is used to limit the number of barrage lines displayed per unit time and perform interval insertion or compression processing on the barrage exceeding the limit.

[0119] During video playback, the barrage is usually displayed in a rolling or static form in a specific area of the screen. If the number of barrages is too large in the same time period, it may cause the barrages to block each other, affecting the reading experience of the audience. Through track density control, the number of barrage lines displayed per second is limited (for example, a maximum of 8 lines per second for large screens and a maximum of 5 lines per second for small screens), and interval insertion or compression processing is performed on the barrage exceeding the limit, to ensure that the barrage is evenly distributed on the screen and avoid blocking.

[0120] The time adjustment module performs time adjustment processing to dynamically adjust the appearance time of the barrage according to the picture semantic rhythm curve.

[0121] Specifically, the process of the time adjustment processing includes: analyzing the semantic changes of the video (for example, by extracting frame-level visual semantic vectors and calculating the semantic vector difference of adjacent frames), generating a graph of the semantic change rate on the time axis. Optionally, the graph is a semantic change rate graph on the time axis, and the high rhythm area and the low rhythm area in the video can be identified in the graph. Then, according to the generated graph, the appearance time of the barrage is adjusted, so that the barrage is inserted in advance in the high rhythm area with fast semantic change, and the barrage is inserted in delay in the low rhythm area with slow semantic change.

[0122] Optionally, the time adjustment module first extracts image semantic vectors of multiple key video frames from the edited video frame by frame or every predetermined time interval, such as 0.5 seconds, such as using CLIP to extract image semantic vectors. The image semantic vector can represent the visual content of each frame, providing basic data for subsequent semantic change analysis. Then, the difference between the semantic vectors of adjacent frames is calculated, for example, using cosine distance as the measurement standard. If the difference between the semantic vectors of adjacent frames exceeds a predetermined threshold, it indicates that the semantic of the video frame picture has changed significantly, such as scene switching or the occurrence of important events, which is in the high semantic rhythm area; on the contrary, if the difference is lower than the predetermined threshold, it indicates that the picture semantic is relatively stable, which is in the low semantic rhythm area.

[0123] Based on the obtained semantic change rate graph, the time adjustment module fine-tunes the appearance time tj' of each candidate barrage (tj', cj). Specifically, for each barrage, the appearance time tj' of each candidate barrage (tj', cj) is fine-tuned. Specifically, for each barrage, the semantic rhythm value corresponding to the time point tj' of the frame where it is located is checked. If the time point is in the high semantic rhythm area, i.e., the video content changes rapidly, the appearance time of the barrage is advanced so that the barrage can appear more timely, which is represented as formula (2); on the contrary, if the time point is in the low semantic rhythm area, i.e., the video content changes slowly, the appearance time of the barrage is delayed to make the appearance of the barrage more natural, which is represented as formula (3).

[0124] Optionally, in order to more accurately determine the high and low of the semantic rhythm value, after obtaining the overall semantic change rate curve, the highest part of the semantic change rate curve is counted, such as the first 20% of the points, and marked as "high rhythm points", and the remaining points are marked as "low rhythm points". This data-driven dynamic adjustment mechanism makes the display time of the barrage accurately adapt to the rhythm change of the video content, improves the coordination of the barrage and the video content, and enhances the viewing experience of the audience.

[0125] Optionally, the device adds special styles to the screened barrage, such as "selected label", "emotion color", "number of likes", etc., to enhance the visual effect and information expression of the barrage. For example, red frames are added to barrages with strong emotions, and like icons are added to barrages with high interactivity, to improve the viewing experience of the audience.

[0126] According to an embodiment, the device provides an interactive interface for adjusting the barrage. The device of the embodiment further comprises a barrage updating device.

[0127] The barrage updating device automatically updates the barrage track file of the video to be processed in response to the editing operation of the editor on the interactive interface for the barrage to be displayed.

[0128] Optionally, the device displays a timeline and a semantic heat map in parallel in the interactive interface. The timeline is used to present the distribution, occurrence time and density peak of the arranged barrage frame by frame. The semantic heat map is used to superimpose the picture rhythm strength of the corresponding area in the form of color scale, so that the editor can locate the high rhythm area and barrage crowded area at a glance.

[0129] Optionally, through the interactive interface, the editor can directly edit the screened barrage to be displayed in various ways such as dragging, framing or shortcut keys, such as moving the barrage track up and down, sliding the occurrence time left and right, and one-key removing items that do not conform to the context. The interactive interface supports inserting new barrages, such as guiding interactive text, Easter eggs or advertising voiceovers. The new content shares the same scoring and density checking engine as the existing barrage, ensuring that the overall arrangement is not damaged.

[0130] Optionally, the font size, color, stroke, transparency, display area (top, bottom, scrolling, fixed) and life cycle of a single or batch of barrages can be set through the interactive interface. All changes are rendered in real time, and what you see is what you get.

[0131] The editing operation performed through the interactive interface is synchronized to the background barrage track file to record the new timestamp, content, style parameter and area information of the edited barrage, and automatically re-run the density and rhythm check. If there is an overrun or obstruction, a pop-up window will remind and give a suggested value. After the check is passed, the barrage file (such as JSON / ASS) that meets the rendering engine specification can be exported one-key.

[0132] The device of the embodiment of the application provides flexible subsequent barrage rendering methods based on the reconstructed barrage, i.e., the target barrage set, to meet the needs of different application scenarios.

[0133] Optionally, the device fuses the reconstructed barrage with the clip segment to generate a final video file. This approach is suitable for scenarios where the barrage needs to be directly embedded in the video, ensuring that viewers can see a barrage that closely matches the content when watching the video, thereby enhancing the overall viewing experience. The fused video file can be directly used for publishing and distribution, and viewers can see the optimized barrage effect without additional operations when playing.

[0134] Optionally, the device saves the barrage layer separately from the video segment. In this mode, the barrage layer exists as an independent file, while the video segment remains unchanged. When other viewers watch the video on the playback end, the system will render the barrage layer separately on the playback end. This approach provides higher flexibility, allowing dynamic loading and rendering of the barrage as needed on different playback devices or platforms, supporting more complex interactions and personalized settings.

[0135] According to the device of the embodiments of the present application, for the edited video, the frame-level mapping and semantic comparison mechanism accurately realizes the correspondence between the original video frames and the edited frames through image feature extraction and semantic similarity calculation, avoiding time misplacement and context mismatch of the barrage; and for the edited video and the unedited video, a multi-dimensional scoring mechanism is further used to filter out high-quality and representative essence barrage, improving the content information density; and based on the picture semantic rhythm curve, the appearance timing of the barrage is dynamically adjusted to make it more consistent with the content rhythm and enhance the consistency of the viewing experience; and a visual editing and heat map interface is provided, including a time axis and a semantic heat map, supporting manual adjustment and style configuration of the barrage by the editor.

[0136] Based on the same inventive concept, the embodiments of the present application also provide an electronic device, the method corresponding to the electronic device can be the method in the foregoing embodiments, and the problem solving principle of the electronic device is similar to that of the method. The electronic device provided in the embodiments of the present application includes at least one processor and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method and / or technical solutions of the foregoing embodiments of the present application.

[0137] The electronic device can be a user device, or a device integrated by a user device and a network device through a network, or also can be an application program running on the above device, the user device includes but is not limited to computers, mobile phones, tablet computers, smart watches, bracelets and various terminal devices, and the network device includes but is not limited to network hosts, single network servers, multiple network server sets or computer sets based on cloud computing, which can be used to realize part of the processing function when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, wherein the cloud computing is a kind of distributed computing, which is composed of a virtual computer set by a loose coupling computer group.

[0138] Figure 4 The structure of a device suitable for implementing the method and / or technical scheme in the embodiments of the present application is shown, the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In the RAM 1203, various programs and data required for system operation are also stored. The CPU 1201, the ROM 1202 and the RAM 1203 are connected to each other through the bus 1204. The input / output (I / O, Input / Output) interface 1205 is also connected to the bus 1204.

[0139] The following components are connected to the I / O interface 1205: the input part 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; the output part 1207 including a cathode ray tube (CRT, Cathode Ray Tube), a liquid crystal display (LCD, Liquid Crystal Display), an LED display, an OLED display, etc., and a speaker, etc.; the storage part 1208 including one or more computer readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and the communication part 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part 1209 performs communication processing via a network such as the Internet.

[0140] In particular, the method and / or embodiments in the present application can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program which is carried on a computer readable medium, and the computer program contains program codes for executing the method shown in the flow chart. When the computer program is executed by a central processing unit (CPU) 1201, the above-mentioned functions defined in the method of the present application are executed.

[0141] Another embodiment of the present application also provides a computer readable storage medium having stored thereon computer program instructions, which can be executed by a processor to implement the method and / or technical solutions of any one or more embodiments of the present application.

[0142] In particular, the embodiments can employ any combination of one or more computer readable media. The computer readable media can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0143] The computer readable signal medium can include a computer readable program code carried in a baseband or as a part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that can send, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or device.

[0144] The program code contained on the computer readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire line, optical cable, RF, or the like, or any suitable combination of the above.

[0145] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0146] The flow diagrams and block diagrams in the drawings are representative of the architectural, functional, and operational aspects of possible implementations of apparatuses, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0148] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or page components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0150] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0151] The integrated unit realized in the form of software functional unit can be stored in a computer readable storage medium. The software functional unit is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and various program code storage media.

[0152] Finally, it should be pointed out that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present application.

[0153] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The plurality of units or devices stated in the apparatus can also be realized by one unit or device through software or hardware. First, second, etc. The words are used to represent the name, and do not represent any specific order.

Claims

1. A method for processing bullet screen data, wherein, The method includes: Obtain the bullet comment data and video file of the video to be processed; If the video to be processed has been edited, the timeline of the bullet comments in the original video and the edited video is mapped and bullet comments with mismatched content are removed to form a candidate bullet comment set. If it has not been edited, the bullet comment data is used as the candidate bullet comment set. Each bullet comment in the candidate bullet comment set is scored based on multiple dimensions to filter bullet comments and obtain the target bullet comment set. Based on the obtained target bullet screen set, each bullet screen is subjected to display adjustment processing, which includes density control processing and time adjustment processing.

2. The method according to claim 1, wherein, The process of mapping the bullet comment timelines of the original video and the edited video and removing bullet comments with mismatched content to form a candidate bullet comment set includes: Obtain the keyframe feature vectors of the original video and the edited video, and determine the correspondence between the video frames before and after editing by performing temporal similarity matching; Based on the correspondence between the video frames before and after editing, the original bullet screen timestamps are mapped to the timeline of the edited video; For each bullet comment, if the context of the bullet comment does not match the context of the corresponding new video frame after editing, then the bullet comment is deleted.

3. The method according to claim 2, wherein, The process of obtaining keyframe feature vectors from the original video and the edited video, and determining the correspondence between video frames before and after editing by performing temporal similarity matching, includes: Keyframes are extracted evenly from the original video and the edited video at predetermined time intervals to form two sets of keyframes; Extract the feature vectors corresponding to each keyframe and calculate the similarity between the keyframes of the original video and the edited video; Based on the similarity between the keyframes of the original video and the edited video, the correspondence between the two sets of keyframes is determined.

4. The method according to claim 2 or 3, wherein, The method employs a dynamic time warping algorithm to determine the correspondence between video frames before and after editing. This process specifically includes: A similarity matrix is ​​constructed based on the similarity between keyframes of the original video and the edited video; The optimal alignment path is found on the similarity matrix by using a dynamic time warping algorithm to determine the best matching relationship between video frames before and after editing.

5. The method according to claim 2, wherein, Methods for determining whether the context of the bullet comments matches the context of the corresponding new video frames after editing include: Similarity judgment based on visual semantic vectors of video frames; Judge by comparing subtitles or speech-to-text; Judgment is made by reasoning using a pre-trained language model.

6. The method according to claim 2, wherein, The method further includes: Based on the correspondence between video frames before and after editing, a frame mapping index table is constructed accordingly, which is used for mapping the bullet screen timeline.

7. The method according to claim 1, wherein, Based on the obtained target set of bullet comments, the display adjustment process for each bullet comment includes: By performing density control processing, overly dense areas of bullet comments are interspersed or compressed.

8. The method according to claim 1, wherein, Based on the obtained target set of bullet comments, the display adjustment process for each bullet comment includes: By performing time adjustment processing, the appearance time of the bullet comments is dynamically adjusted according to the semantic rhythm curve of the screen.

9. The method according to claim 8, wherein, Adjusting the timing of bullet comments based on the semantic rhythm curve of the visuals includes: By analyzing the semantic changes in the video, a curve of the rate of semantic change on the timeline is generated; Based on the generated curve, the timing of the bullet comments is adjusted so that bullet comments are inserted earlier in the high-rhythm area where semantic changes are rapid, and inserted later in the low-rhythm area where semantic changes are slow.

10. The method according to claim 1, wherein, The method provides an interactive interface for adjusting bullet comments, and the method further includes: In response to the editor's editing operations on the interactive interface, the bullet comment track file of the video to be processed is automatically updated.

11. A bullet screen data processing device, wherein, The device includes: The data acquisition module is used to acquire the bullet screen data and video file of the video to be processed; The first bullet screen filtering module is used to map the bullet screen timeline of the original video and the edited video and remove bullet screens with mismatched content if the video to be processed has been edited, forming a candidate bullet screen set; if it has not been edited, the bullet screen data is used as the candidate bullet screen set. The second bullet screen filtering module is used to score the quality of each bullet screen in the candidate bullet screen set based on multiple dimensions, so as to filter the bullet screens according to the scores and obtain the target bullet screen set. The bullet screen display adjustment module is used to score the quality of each bullet screen in the candidate bullet screen set based on multiple dimensions, so as to filter the bullet screens according to the scores and obtain the target bullet screen set.

12. An electronic device, the electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

13. A computer-readable medium having stored thereon computer program instructions that can be executed by a processor to implement the method as claimed in any one of claims 1 to 10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 10.

Images