Live video collection generation method and system, electronic equipment and storage medium

Abstract

The invention relates to a live video collection generation method and system, electronic equipment and a storage medium. The method comprises the steps of collecting video data and audio data of an affiliated area in a recreation event scene under a non-rebroadcasting view angle based on a distributed sensing network applied to the recreation event scene, and adding space-time metadata annotations; according to the video data and the audio data, multi-modal fusion judgment is carried out through field audience expressions, action density, sound pressure level mutation and literary and sports events, and the wonderful moments of non-rebroadcast shots are identified; based on the wonderful moments, materials of the non-rebroadcast shot are called, multi-view picture alignment and splicing are carried out on the materials of the non-rebroadcast shot through timestamp and space coordinate calibration, and an emotional collection video is obtained. According to the method and the device, the wonderful moments which are not recorded by the live broadcast lens can be captured, the risk of omission of wonderful contents can be reduced, and the content multiplexing and spreading efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the fields of computer vision, audio and video processing, artificial intelligence and multimedia technology, and particularly relates to a live video highlight generation method and system, an electronic device and a storage medium. BACKGROUND

[0002] With the maturity and popularity of network live streaming technology, various live streaming contents such as sports events, e-commerce live streaming, game competitions and large-scale activities are emerging at an unprecedented scale and frequency. Unlike traditional recorded videos, live streaming has the significant characteristics of real-time, massiveness and unpredictability. This brings great challenges to the value reuse of content: the core highlights and highlights of a live streaming that lasts for several hours are often fleeting, and if they cannot be captured and spread in the first time, their timeliness value and public opinion heat will quickly decay.

[0003] In related technologies, the generation of exciting moments in current sports events (such as sports events) live streaming mainly relies on traditional director switching cameras, and its coverage is usually limited to the main on-site perspective, resulting in problems such as single presentation perspective, high risk of missing exciting content, low content reuse and transmission efficiency, etc. SUMMARY

[0004] The present disclosure provides a live video highlight generation method and system, an electronic device and a storage medium, which can solve the problems of single presentation perspective, high risk of missing exciting content, low content reuse and transmission efficiency, etc. in the related art exciting moment generation scheme.

[0005] In a first aspect, the embodiments of the present disclosure provide a live video highlight generation method, comprising: Based on a distributed sensing network applied to a sports event scene, video data and audio data of an auxiliary area in the sports event scene under a non-rebroadcast perspective are collected, and time-space metadata labels are attached; According to the video data and audio data, through multi-modal fusion determination of on-site audience expressions, action density, sound pressure level mutation and sports events, exciting moments of non-rebroadcast shots are identified; Based on the exciting moments, the materials of the non-rebroadcast shots are retrieved, and the materials of the non-rebroadcast shots are multi-perspective picture aligned and spliced through timestamp and spatial coordinate calibration to obtain an emotional highlight video.

[0006] In a second aspect, the embodiments of the present disclosure provide a live video highlight generation system, comprising: A distributed sensing network deployment module is configured to build a global coverage sports event scene data collection system; The real-time data acquisition and labeling module is configured to acquire video data and audio data of an auxiliary area in the sports event scene under a non-rebroadcasting perspective based on a distributed perception network applied to the sports event scene, and attach space-time metadata labels. The multi-modal highlight determination module is configured to determine highlights of the non-rebroadcasting shot based on the video data and the audio data through multi-modal fusion of live audience expressions, action density, sound pressure level mutation, and the sports event. The emotional highlight reel generation module is configured to retrieve the material of the non-rebroadcasting shot based on the highlights, and perform multi-perspective picture alignment and splicing on the material of the non-rebroadcasting shot through timestamp and spatial coordinate calibration to obtain an emotional highlight reel video.

[0007] In a third aspect, an electronic device is provided, including: one or more processors; The processor is configured to call instructions to cause the electronic device to perform the method of the first aspect.

[0008] In a fourth aspect, a storage medium is provided, which stores instructions, and when the instructions are run on an electronic device, the electronic device performs the method of the first aspect.

[0009] In a fifth aspect, a program product is provided, which includes at least one of a program and instructions, and when the at least one of the program and instructions is executed by an electronic device, the steps of the method of the first aspect are implemented.

[0010] According to the technical solution of the present disclosure, through the distributed perception network, the expressions, actions, and sound field changes of the live audience are directly captured to drive the determination of highlights, and the non-rebroadcasting shot perspective material (such as the auxiliary area in the sports event scene) is automatically retrieved to complete multi-modal data fusion and real-time generation of emotional highlight reels, without relying on live picture content (i.e., picture content of the rebroadcasting shot perspective). The present disclosure builds an independent highlight generation system of "live reaction driving-multi-perspective material integration", which can completely eliminate the dependence on live director shots, can capture highlights that are not recorded by live shots, can reduce the risk of missing highlights, and can improve content reuse and transmission efficiency.

[0011] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure.

[0013] Figure 1 is a flowchart of a method for generating a live video highlight according to an example embodiment.

[0014] Figure 2 is a flowchart of a method for generating a live video highlight according to an example embodiment.

[0015] Figure 3 is a block diagram of a system for generating a live video highlight according to an example embodiment.

[0016] Figure 4 is a flowchart of a method for generating a live video highlight according to an example embodiment.

[0017] Figure 5 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] The example embodiments will be described in detail herein with reference to the accompanying drawings. In the following description, the same numbers refer to the same or similar elements unless otherwise represented. The implementations described in the following example embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0019] The embodiments of the present disclosure are not exhaustive, but are only schematic of some embodiments, and are not specific limitations on the protection scope of the present disclosure. Each step in an embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily, for example, a scheme in which some steps are removed from an embodiment can be implemented as an independent embodiment, and the order of the steps in an embodiment can be exchanged arbitrarily, in addition, the optional implementation manners in an embodiment can be combined arbitrarily; in addition, the embodiments can be combined arbitrarily, for example, some or all steps of different embodiments can be combined arbitrarily, an embodiment can be combined with optional implementation manners of other embodiments arbitrarily.

[0020] In the embodiments of the present disclosure, the terms and / or descriptions of the embodiments are consistent and can be referred to each other if there is no special description and logical conflict, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0021] It should be noted that the acquisition, transmission, storage, use, processing, etc. of data in the technical solutions of the present disclosure comply with relevant provisions of national laws and regulations, and do not violate public order and good customs.

[0022] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0023] In some embodiments, data, information, etc., may be obtained with the user's consent.

[0024] In this disclosure, "at least one" means one or more. "More than one" means two or more.

[0025] It is worth noting that in the embodiments disclosed herein, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary, and their purpose is only to illustrate the feasibility of implementing the technical solutions disclosed herein. However, this does not mean that the applicant has used or necessarily used such solutions.

[0026] The following description, with reference to the accompanying drawings, outlines a method, system, electronic device, and storage medium for generating live video highlights according to embodiments of the present disclosure.

[0027] It should be noted that the execution entity of the live video highlight generation method in this embodiment can be a live video highlight generation system. This system can be implemented by software and / or hardware. The system can be configured in an electronic device, which may include, but is not limited to, a server, such as an edge node (edge ​​server).

[0028] Figure 1 This is a flowchart illustrating a method for generating live video highlights according to an exemplary embodiment. Figure 1 As shown, the method for generating this live video highlight reel may include, but is not limited to, the following steps.

[0029] In step 101, based on the distributed sensing network applied to the cultural and sports event scenario, video and audio data of the auxiliary areas in the cultural and sports event scenario are collected from a non-broadcast perspective, and spatiotemporal metadata annotations are added.

[0030] In some embodiments, a distributed sensing network covering the entire area (including ancillary areas within the sports and cultural event scene) can be constructed within the sports and cultural event scene. These ancillary areas may include the stands and non-broadcast areas such as the substitutes' bench. The distributed sensing network may include, but is not limited to, camera arrays and microphone arrays. For example, multiple (e.g., 50+) high-definition wide-angle cameras can be deployed in the ancillary areas of the sports and cultural event scene. For instance, taking a ball game as an example, multiple high-definition wide-angle cameras can be deployed in non-broadcast areas (also called non-core areas) such as the stands, substitutes' bench, and player tunnel to adapt to the complex lighting environment of the stadium and collect video data from the corresponding areas. A full-band microphone array can be deployed in the ancillary areas of the sports and cultural event scene to filter background noise and collect audio data from the corresponding areas, thereby constructing a data acquisition system for the entire sports and cultural event scene. Optionally, the distributed sensing network can adopt the PTP (Precision Time Protocol) clock synchronization protocol to ensure that the time error of multiple devices is small (e.g., <10ms); the video stream is compressed with ROHC (e.g., compression ratio 10:1), and the audio stream is encoded with Opus (e.g., 64kbps), and transmitted with low latency through a private network (e.g., 5Gzhuanwang2).

[0031] In the embodiments of this disclosure, the distributed sensing network described above can be used to synchronously collect video and audio data of the auxiliary areas in the cultural and sports event scene from a non-broadcast perspective, and add spatiotemporal metadata such as camera identifier (ID), spatial coordinates (XYZ), and timestamp (accurate to milliseconds) to each frame of video data to facilitate subsequent cross-view association.

[0032] It should be noted that the video and audio data collected by the aforementioned distributed sensing network can be stored in a distributed manner, such as using Ceph distributed storage (3-replica redundancy), and the video stream can be indexed by "camera ID + timestamp" to facilitate subsequent retrieval of "highlights ± 5 seconds" of material within a short period of time (e.g., 5 seconds).

[0033] In step 102, based on video and audio data, multimodal fusion judgment is performed by considering the facial expressions, movement density, sudden changes in sound pressure level, and cultural events of the audience to identify highlights from non-broadcast footage.

[0034] In some embodiments, video and audio data can be preprocessed based on edge nodes to obtain audience facial frames, audience motion recognition information, and sound pressure levels of specific audience performance scenarios. Multimodal fusion judgment is then performed using on-site audience facial expressions, motion density, sound pressure level abrupt changes, and the context of cultural events to identify highlights from non-broadcast footage. For example, highlights can be determined in corresponding dimensions using on-site audience facial expressions, motion density, sound pressure level abrupt changes, and the context of cultural events. Highlights from non-broadcast footage are identified when the criteria for highlighting moments in at least two dimensions are met.

[0035] In step 103, based on the highlights, non-broadcast footage is retrieved, and the non-broadcast footage is aligned and stitched together from multiple perspectives using timestamps and spatial coordinate calibration to obtain an emotional highlight video.

[0036] In some embodiments, terms such as “highlight,” “moment of brilliance,” and “decision moment” can be used interchangeably.

[0037] In the embodiments of this disclosure, when a highlight moment of a non-broadcast shot is identified, the material of the non-broadcast shot (including video data and / or audio data) can be retrieved from distributed storage, and the material of the non-broadcast shot can be aligned and stitched together from multiple perspectives through spatiotemporal calibration, thereby obtaining an emotionally engaging highlight video that conforms to viewing habits and can highlight the highlights from the non-broadcast perspective.

[0038] In the above embodiments, a distributed sensing network directly captures the facial expressions, movements, and sound field changes of the audience to drive the determination of highlights. It also automatically retrieves footage from non-broadcast camera angles (such as secondary areas in cultural and sports events) to complete the real-time generation of multimodal data fusion and emotionally charged highlight videos, without relying on live broadcast content (i.e., the content from the broadcast camera angle). This disclosure, by constructing an independent highlight generation system driven by "live reaction - multi-view footage integration," completely eliminates dependence on live broadcast camera angles, enabling the capture of highlights not recorded by the live camera. It solves the problems of limited perspectives, high risk of missing highlights, and low efficiency in content reuse and dissemination found in related highlight generation solutions.

[0039] Figure 2 This is a flowchart illustrating a method for generating live video highlights according to an exemplary embodiment. Figure 2 As shown, the method for generating this live video highlight reel may include, but is not limited to, the following steps.

[0040] In step 201, based on the distributed sensing network applied to the cultural and sports event scenario, video and audio data of the auxiliary areas in the cultural and sports event scenario are collected from a non-broadcast perspective, and spatiotemporal metadata annotations are added.

[0041] Optionally, step 201 can be implemented using any of the implementation methods in the various embodiments of this disclosure. This disclosure does not limit this implementation and will not elaborate further.

[0042] In step 202, the video data is preprocessed based on edge nodes to obtain audience face frames and audience action recognition information.

[0043] In the embodiments of this disclosure, keyframes can be extracted from the acquired video data via edge nodes, for example, one frame can be extracted every 0.3 seconds, and duplicate frames can be filtered using SIFT (Scale-invariant feature transform) feature matching (keeping frames with motion changes >10%) to obtain keyframes. Once keyframes are obtained, face detection can be performed on them using a target detection algorithm (such as a face detection model) to obtain video frames including faces, which are then used as viewer face frames. Alternatively, an action recognition model deployed on the edge nodes (such as a lightweight OpenPose model) can be invoked to detect key points of the viewer's limbs (such as the head, hands, etc.) to identify specific actions. For example, corresponding actions can be identified by calculating "jumping frequency" (e.g., >2 times per second for excitement) and "waving amplitude" (e.g., arm swing angle >60° for cheering), thereby obtaining viewer action recognition information.

[0044] Optionally, in some embodiments, the facial region of the audience in the audience face frame can be Gaussian blurred to hide personal features. For example, a 5×5 kernel Gaussian blur is applied to the audience's facial region to preserve the contours of expressions such as "upturned corners of the mouth" and "relaxed eyebrows" while hiding personal features, thereby achieving the purpose of privacy protection.

[0045] In step 203, the audio data is preprocessed based on the edge nodes to obtain the sound pressure level of the audience's specific performance scene.

[0046] In some embodiments, audio data can be segmented into audio segments, such as 25ms / frame, via edge nodes. Feature extraction is then performed on these audio segments, such as extracting 13-dimensional MFCC (Mel-Frequency Cepstrum) features to construct a real-time audio feature matrix. This real-time audio feature matrix is ​​then used for audio classification. For example, a pre-trained GMM (Gaussian Mixture Model) can be used to classify audio based on the real-time audio feature matrix, identifying specific audience performance scenarios, such as "goal cheers," "referee reactions," and "wave chants." Real-time short-time energy (STE) calculation is performed by segmenting the continuous audio stream into short frames (assuming the signal is approximately stationary within a short time) and calculating the energy value of each frame, thereby capturing dynamic changes in energy (such as sudden increases and decreases). The moment when the sound pressure level suddenly increases by a certain value (such as >30dB) (such as the instant of a goal) is marked, thus obtaining the sound pressure level of the specific audience performance scenario, which serves as one of the trigger conditions for subsequent highlight moment determination.

[0047] It should be noted that in some embodiments, steps 202 and 203 above can be executed simultaneously or in reverse order.

[0048] In step 204, facial expression recognition is performed on the audience's facial frames to obtain the audience expression recognition results, and the audience's motion recognition information is analyzed to obtain the motion density analysis results.

[0049] In the embodiments of this disclosure, an expression recognition model (such as an improved ResNet-50 model) can be invoked to perform expression recognition on the obtained audience facial frames. The audience expression recognition results, for example, statistically analyze the proportion of "excitement" (upturned corners of the mouth + wide eyes). Audience action recognition information is used for action density analysis. For example, this audience action recognition information may include the audience's limb key points. Based on the audience's limb key points, the proportion of specific actions of the audience in a single area is calculated. For example, the proportion of "standing / jumping" of the audience in a certain area of ​​the north grandstand is calculated. This action density analysis result can be used as one of the triggering conditions for subsequent determination of exciting moments.

[0050] In step 205, based on the audience's facial expression recognition results, motion density analysis results, sound pressure level, and the process of the cultural and sports event, the exciting moments are determined in terms of audience emotion dimension, cultural and sports event soundscape dimension, and cross-scene association dimension.

[0051] In the embodiments of this disclosure, the determination of highlights can be based on audience facial expression recognition results and motion density analysis results, focusing on the audience's emotional dimension. For example, the proportion of specific expressions, such as "excitement" (upturned lips + wide eyes), can be statistically analyzed using audience facial expression recognition results. When the proportion of this specific expression in a single area is greater than or equal to a threshold (e.g., 70%), a condition can be triggered. The proportion of specific audience actions in a single area is determined through motion density analysis results, such as the proportion of "standing / jumping" in a single area (e.g., a section of the north grandstand). When the proportion of specific audience actions in that single area is greater than or equal to a threshold (e.g., 80%) and continues for a preset duration (e.g., >2 seconds), the determination weight is strengthened. The determination of highlights relies on multi-dimensional conditions (peak audience facial expression, sudden increase in sound pressure level, proportion of body movements, etc.), with each condition corresponding to a "basic weight" (e.g., a default weight of 1 for each item). The essence of "strengthening the judgment weight" is to increase the basic weight of "standing / jumping ratio meets the standard" (e.g., from 1 to 1.5 or 2), so that it has a higher weight in the multi-dimensional fusion calculation, which is equivalent to "giving an extra vote" to this condition. This avoids the loss of wonderful moments caused by judging a single expression.

[0052] In the embodiments of this disclosure, the exciting moment can be determined based on the sound pressure level in the audio-visual dimension of the cultural event. For example, the exciting moment can be determined based on the sound pressure level combined with sound direction correlation in the audio-visual dimension of the cultural event. For instance, when a sound pressure level greater than or equal to a threshold (e.g., 100 dB) is detected and continues for a preset duration (e.g., greater than or equal to 2 seconds to exclude brief noise), and the similarity between the MFCC feature and a pre-stored voiceprint library (e.g., a "victory cheer" voiceprint library) is greater than or equal to a threshold (e.g., 0.8), an audio condition can be triggered. Combined with the azimuth data of the microphone array, the region from which the cheering sound originates can be located. This is then linked to the video footage of that region for verification, to confirm whether the cheering sound in that region matches the content of the video footage. For example, if the video footage includes spectators waving their arms / standing / jumping / players scoring goals, it indicates a match with the cheering sound in that region.

[0053] In the embodiments of this disclosure, highlights can be determined across the cross-scene association dimension based on the progress of the cultural event. For example, this cross-scene association may include time association and event association. Time association refers to determining a "highlight moment" by comparing timestamps and identifying when the time between a specific action (e.g., a celebration, such as hugging or pointing at the audience) of the main character (e.g., a player) and the audience's reaction is less than or equal to a preset difference (e.g., 1 second). Event association refers to activating the highlight moment determination mode when an event is triggered. For example, in a ball game, event association might involve activating the "full cheer" determination mode after the "goal valid" command is issued by the VAR (Video Assistant Referee) system signal. Optionally, if both time association and event association are satisfied, the criteria for highlight moment determination can be determined across the cross-scene association dimension. Optionally, the trigger threshold can be lowered to avoid missing key moments.

[0054] In step 206, if the criteria for judging the highlights in at least two of the following dimensions—audience emotion dimension, cultural and sports event soundscape dimension, and cross-scene association dimension—satisfy the criteria for judging the highlights in the corresponding dimension, then the highlights from non-broadcast shots are identified.

[0055] In the embodiments of this disclosure, if the criteria for determining a highlight moment in the audience's emotional dimension and the criteria for determining a highlight moment in the cultural and artistic event's soundscape dimension are both met, then highlight moments not captured in the broadcast are identified, and the process proceeds to the material integration stage, i.e., step 207 is executed below. Alternatively, if the criteria for determining a highlight moment in the audience's emotional dimension and the criteria for determining a highlight moment in the cross-scene association dimension are both met, then highlight moments not captured in the broadcast are identified, and the process proceeds to the material integration stage. Alternatively, if the criteria for determining a highlight moment in the cultural and artistic event's soundscape dimension and the criteria for determining a highlight moment in the cross-scene association dimension are both met, then highlight moments not captured in the broadcast are identified, and the process proceeds to the material integration stage. Alternatively, if the criteria for determining a highlight moment in the audience's emotional dimension, the criteria for determining a highlight moment in the cultural and artistic event's soundscape dimension, and the criteria for determining a highlight moment in the cross-scene association dimension are all met, then highlight moments not captured in the broadcast are identified, and the process proceeds to the material integration stage.

[0056] In step 207, based on the highlights, footage from non-broadcast shots is retrieved, and the footage from non-broadcast shots is aligned and stitched together from multiple perspectives using timestamps and spatial coordinate calibration to obtain an emotionally engaging highlight video.

[0057] In some embodiments, based on the highlight moment, non-broadcast footage is retrieved, and the non-broadcast footage is aligned and stitched together from multiple perspectives using timestamps and spatial coordinate calibration. Centered on the highlight moment, a segment of preset duration is extracted from the footage obtained after multi-view alignment and stitching, serving as a non-broadcast perspective footage segment. Based on the highlight moment, broadcast perspective footage segments are obtained, and the non-broadcast perspective footage segments and broadcast perspective footage segments are merged from multiple perspectives, with original sound and interactive elements superimposed to generate an emotional highlight video.

[0058] For example, since spatiotemporal metadata is added to the collected audio and video data, the determined highlights (also called highlights) can be used to schedule corresponding non-broadcast footage from distributed storage, and spatiotemporal calibration can be used to align and stitch the non-broadcast footage from multiple perspectives. The scheduling strategy for this footage may include: prioritizing the retrieval of multi-perspective non-broadcast footage that matches the sound direction positioning. For example, in a ball game, priority can be given to retrieving footage from spectator cameras (such as the North Stand Section 3 camera), close-up shots of the substitutes' bench, and sideline coach shots that match the sound direction positioning, ensuring coverage of the "core area of ​​the event + reaction area". Centered on the highlight moments, the system intelligently extracts segments of preset duration from footage obtained after aligning and stitching multiple perspective shots. For example, using FFmpeg hardware decoding, a 10-second segment is extracted centered on the highlight moment, consisting of "2 seconds before the event (e.g., the assist) + 8 seconds after the event (e.g., the celebration reaction)" (the duration is adapted for short video dissemination). The non-broadcast perspective footage and the corresponding broadcast perspective footage of the highlight moment are then merged into a multi-perspective video, and the original sound and interactive elements are overlaid to generate an emotional highlight video. For example, the original sound is preserved (redundant commentary is removed using Audacity), the cheers are dynamically balanced (peak volume does not exceed -6dB), the online interactive API (Application Programming Interface) is called, the top 10 highly praised comments are obtained (e.g., "This goal was a game-winner!"), and the Airtest tool is used to overlay subtitles at the bottom of the screen (e.g., the font color matches the team's main color).

[0059] To further enhance user experience, improve video playback, and enable real-time interaction between online and offline users, some embodiments can utilize Content Delivery Network (CDN) technology to distribute different versions of the emotional highlight video to corresponding compatible terminals; acquire interactive information sent by the terminals regarding the emotional highlight video, including interactive information sent by on-site audience members via the network; and overlay the interactive information onto the emotional highlight video displayed on the corresponding terminal. For example, CDN technology can be used to distribute emotional highlight videos adapted to different terminals locally, supporting real-time online and offline interaction. These different terminals can include televisions, mobile devices, and large on-site screens. By distributing different versions of the emotional highlight video to corresponding compatible terminals via CDN technology, each terminal can display the emotional highlight video using its own video player and corresponding display mode. End users can interact with the emotional highlight videos, such as commenting, sending bullet comments, and liking. Taking a large screen as an example, on-site users can scan a QR code on their seat or the large screen to access an H5 page or an online interactive platform. They can then view comments and bullet comments from online users in the interactive area and post feedback such as text and emoticons in real time. The system can receive interactive information sent by the terminal regarding the emotional highlight videos. This information may include interactive messages sent by on-site audience members via the internet (on-site users can scan a QR code on their seat or the large screen to access an H5 page or an online interactive platform and post feedback such as text and emoticons in the interactive area), but it is not limited to this. For example, it may also include interactive information from online users. This interactive information can be overlaid onto the emotional highlight videos displayed on the corresponding terminals, enabling real-time interaction between online and offline users based on the highlight videos.

[0060] In some embodiments, when the interaction information of online users on the terminal meets the trigger condition for unlocking a specific viewpoint, audio and video data corresponding to the specific viewpoint in the emotional highlight video are extracted to optimize the weight used to determine the next highlight moment in the corresponding dimension. For example, when online users on the terminal like an emotional highlight video, if the number of likes reaches a preset threshold at a certain moment in the emotional highlight video, it is considered that the trigger condition for unlocking the viewpoint at that moment is met. The audio and video data corresponding to that viewpoint can be extracted from distributed storage and sent back to the edge node to optimize the determination of the next highlight moment, such as increasing the weight of high-interaction areas.

[0061] By implementing the embodiments disclosed herein, full-scene coverage can be achieved, effectively enhancing the richness of highlight content and capturing exciting moments in secondary areas (i.e., the second scene) to fill the gap in "emotional content." Data processing through edge nodes ensures that the delay from the occurrence of a highlight to its display is kept low, allowing for synchronized push with the live stream and achieving a real-time closed loop of "event occurrence - emotion capture - content generation." Through dual verification of audience facial expressions and sound effects, the generated highlight videos more closely resemble the user's real experience, increasing viewing time and sharing rates.

[0062] Figure 3 This is a block diagram illustrating a live video highlight generation system according to an exemplary embodiment. For example... Figure 3 As shown, the system for generating live video highlights may include: a distributed sensing network deployment module 301, a real-time data acquisition and annotation module 302, a multimodal highlight judgment module 303, and an emotional highlight generation module 304.

[0063] Among them, the distributed sensing network deployment module 301 is used to build a data collection system for cultural and sports event scenarios with full coverage.

[0064] The real-time data acquisition and annotation module 302 is used to acquire video and audio data of the auxiliary areas in the cultural and sports event scene from a non-broadcast perspective based on a distributed sensing network applied to the cultural and sports event scene, and to add spatiotemporal metadata annotations.

[0065] The multimodal highlight moment determination module 303 is used to perform multimodal fusion determination based on video and audio data, through on-site audience facial expressions, movement density, sudden changes in sound pressure level, and cultural events, to identify highlight moments from non-broadcast footage.

[0066] The emotional highlight generation module 304 is used to retrieve footage from non-broadcast shots based on highlights, and to perform multi-view image alignment and splicing of the non-broadcast footage through timestamp and spatial coordinate calibration to obtain an emotional highlight video.

[0067] In some embodiments, the live video highlight generation system further includes an edge preprocessing module. This edge preprocessing module is used to: preprocess video data based on edge nodes to obtain audience facial frames and audience motion recognition information; and preprocess audio data based on edge nodes to obtain the sound pressure level of a specific audience performance scene. In embodiments of this disclosure, the multimodal highlight moment determination module 303 is used to: perform facial expression recognition on audience facial frames to obtain audience expression recognition results, and analyze the audience motion recognition information to obtain motion density analysis results; determine highlight moments based on the audience expression recognition results, motion density analysis results, sound pressure level, and the progress of the cultural and sports event, in the dimensions of audience emotion, cultural and sports event soundscape, and cross-scene association; and identify highlight moments from non-broadcast shots when the highlight moment determination in at least two of the dimensions of audience emotion, cultural and sports event soundscape, and cross-scene association satisfies the corresponding highlight moment determination conditions.

[0068] In some embodiments, the edge preprocessing module is further configured to: perform Gaussian blur processing on the audience facial region in the audience facial frame to hide personal features.

[0069] In some embodiments, the live video highlight generation system further includes a multi-source material retrieval and synchronization module. This module is used to: retrieve non-broadcast footage based on the highlight moment, and perform multi-view alignment and stitching of the non-broadcast footage using timestamps and spatial coordinate calibration. In embodiments of this disclosure, the emotional highlight generation module is used to: extract a segment of preset duration from the footage obtained after multi-view alignment and stitching, centered on the highlight moment, as a non-broadcast perspective footage segment; acquire broadcast perspective footage segments based on the highlight moment, fuse the non-broadcast perspective footage segments and broadcast perspective footage segments using multi-view fusion, and overlay live sound and interactive elements to generate an emotional highlight video.

[0070] In some embodiments, the system for generating live video highlights further includes a real-time distribution and interaction module. This real-time distribution and interaction module is used to: distribute different versions of the emotional highlight video to corresponding compatible terminals using Content Delivery Network (CDN) technology; acquire interactive information sent by the terminals regarding the emotional highlight video, including interactive information sent by on-site audience members via the network; and overlay the interactive information onto the emotional highlight video displayed on the corresponding terminals.

[0071] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0072] To facilitate a clearer understanding of this disclosure by those skilled in the art, a description will be provided below with specific examples. Taking a sports event as an example, specifically a ball game, the automatic generation and scene interaction scheme for live video highlights disclosed herein will be described in detail. For instance, the scheme disclosed herein may include a distributed sensing network deployment module, a real-time data acquisition and annotation module, an edge preprocessing module, a multimodal highlight judgment module, a multi-source material retrieval and synchronization module, an emotional highlight generation module, and a real-time distribution and interaction module. Among these, The core function of the distributed sensing network deployment module is to build a data acquisition system for the entire event scene, such as using the PTP clock synchronization protocol to ensure that the time error of multiple devices is <10ms; ROHC compression of video streams (compression ratio 10:1), Opus encoding of audio streams (64kbps), and low-latency transmission via 5G private network. The core function of the real-time data acquisition and annotation module is to synchronously acquire multi-dimensional raw data and attach spatial coordinates; for example, to establish a data cache queue (10GB capacity) to store the most recent 30 seconds of full-view video stream for subsequent retrieval of non-current footage. The core function of the edge preprocessing module is to reduce data complexity and protect user privacy, with localized processing latency <500ms, meeting the real-time requirements of edge computing. The core function of the multimodal highlight moment judgment module is to identify highlight moments outside the current shot through 3D data fusion; the accuracy rate for identifying irregular events can reach, for example, 92%, breaking through the limitations of traditional rule engines. The core functions of the multi-source material retrieval and synchronization module are: retrieving and calibrating highlights from footage not currently being broadcast, supporting up to 72 hours of raw video stream storage for secondary analysis. The core functions of the emotional highlight generation module are: generating multi-perspective fusion highlights based on emotional peaks; the generated highlights' emotional resonance can be increased by, for example, by 40%, and user sharing rates by, for example, by 25%. The core functions of the real-time distribution and interaction module are: pushing highlights with low latency and building an online-offline interactive loop; end-to-end latency can be, for example, ≤3 seconds, meeting the immediacy requirements of live streaming scenarios. The data flow of the core interaction logic between the above modules can be: distributed sensing network → edge preprocessing → multimodal judgment → material retrieval → highlight generation → distribution and interaction; content generation is driven by audience emotional data, and the generated highlights feed back into user interaction, forming a complete closed loop of "perception-analysis-generation-feedback".

[0073] For example, such as Figure 4 As shown, the implementation of the automatic generation and scene interaction scheme for live video highlights involved in this disclosure may include the following steps S1 to S9.

[0074] In step S1, 2 hours before the match – system initialization and full-area perception network deployment: hardware deployment, calibration, and edge node activation of the full-area perception network, and time synchronization of multiple devices via the PTP protocol. The core objective can be: to build a perception network covering the entire stadium area (including non-broadcast areas such as the stands and substitute benches), and to complete hardware calibration and system warm-up.

[0075] The camera array installation can utilize 50+ high-definition wide-angle cameras (supporting H.265 encoding, low-light sensitivity of 0.01 lux) to adapt to the complex lighting environment of the competition venue; the deployment strategy can be as follows: The seating area is arranged in groups of 5 rows (15 groups in each of the north and south stands), with the camera angled downwards at 30° to cover 98% of the audience area. Key areas: Two close-up cameras (10-120mm focal length) each at the substitutes' bench and the entrance / exit of the players' tunnel to capture coaching instructions and player reactions to substitutions; Panoramic view: Two fisheye lenses (180° horizontal field of view) are deployed at the highest points of the stadium (such as the lighthouse or the top of the stands) to cover the entire spatial coordinates of the stadium.

[0076] Technical calibration: Camera distortion correction is completed using Zhang Zhengyou's calibration method, generating a spatial coordinate mapping matrix (error <5cm) to ensure that multi-view images can be stitched together.

[0077] Microphone array deployment: Audio-Technica channel beamforming microphones (pickup radius 1-10 meters, signal-to-noise ratio 70dB) can be used to filter background noise in the arena. The zoning strategy for this microphone array deployment can be as follows: Fan Zone: 3 groups each for the home team and away team stands, using beamforming technology to directionally pick up cheers and slogans within the area; Sideline area: One set of voice-activated microphones (trigger threshold 65dB) are deployed in the player tunnel and coach bench to collect only sounds above the ambient noise level (such as coaches yelling and players talking).

[0078] Audio synchronization: The audio-visual synchronization error is ensured to be less than 20ms by binding the timestamp of the video stream with the AES67 protocol.

[0079] In this step, edge nodes can also be activated. For example, three edge servers can be deployed to handle data processing for the North Stand, South Stand, and field side areas, respectively. Models can also be preloaded, including a lightweight ResNet-50 facial expression recognition model (parameters compressed to 10MB, inference speed 30ms / frame), with training data containing spectator facial expression samples from over 500 matches (6 categories including excitement / calm / anger); and an optimized OpenPose body movement model (89% keypoint detection accuracy), focusing on recognizing typical fan movements such as standing / jumping / waving.

[0080] In the embodiments disclosed herein, a 5G private network (slice bandwidth 1Gbps) can be used to connect with broadcast vans and cloud systems, enabling local closed-loop edge computing (e.g., reducing data backhaul by 90%).

[0081] In the embodiments disclosed herein, time synchronization calibration can be performed, for example, by using the PTP v2 protocol to align with the referee timer and VAR system timeline to ensure that the time error of multiple devices is less than 50ms; a "match timestamp lookup table" is generated to associate with the broadcast truck's timecode and solve the time difference problem of splicing multiple source materials.

[0082] In step S2, during the match - multi-dimensional data acquisition: synchronously acquire video and audio data from the stands and substitutes' benches from non-broadcast perspectives, and add spatiotemporal metadata annotations.

[0083] The core objective may be to simultaneously collect video and audio data from non-broadcast perspectives to provide raw materials for subsequent analysis.

[0084] Among them, video stream acquisition (focusing on non-broadcast perspectives) The acquisition parameters can be: 25fps frame rate, YUV420 format, 1080p resolution (raw data is compressed 10 times by the ROHC algorithm before transmission to save bandwidth); the acquisition focus can be as follows: the stands (real-time capture of non-broadcast shots such as "human wave interaction", "holding up banners and cheering", "collective standing up", etc.), the substitutes' bench (recording reactions such as "substitute players rushing onto the field to celebrate" and "coach clenching his fist / throwing the tactical board" through close-up shots), and the players' tunnel (capturing edge scenes such as "substitute players hugging after a goal" and "emotions of players who are sent off"). Spatiotemporal metadata can be annotated on the acquired video stream, that is: each frame of video is appended with a camera ID, spatial coordinates (XYZ), and timestamp (accurate to milliseconds), which facilitates subsequent cross-view association.

[0085] Audio stream capture (matching the soundscape of the event) The acquisition parameters can be: 44.1kHz sampling rate, 16-bit bit depth, PCM format, and volume normalized to -18dBFS through automatic gain control (AGC). The acquisition focus can be as follows: the moment of a goal (capturing the "sudden increase in sound pressure level" of the fans' cheers (from 60dB to over 100dB, lasting >3 seconds)), key decisions (distinguishing the voiceprint characteristics of "cheering" (high-frequency energy concentration) and "boos" (low-frequency energy dominance), and sideline interactions (picking up the coach's on-field instructions (such as "press up!") and the players' communication (such as celebration shouts)).

[0086] In step S3, edge node-data preprocessing involves extracting video keyframes, performing motion recognition, and extracting audio features, while protecting viewer privacy through Gaussian blur.

[0087] The core objectives are: to reduce data complexity, extract key features, protect audience privacy, and provide lightweight data for subsequent judgments.

[0088] Video processing (adapting to the pace of the event) may include keyframe extraction, motion recognition, and privacy protection. Keyframe extraction: Using OpenCV's cv2.VideoCapture interface, one frame is extracted every 0.3 seconds (which is more suitable for fast celebration actions than the usual 0.5 seconds). Duplicate frames are filtered by SIFT feature matching (frames with motion changes >10%). Action recognition: The lightweight OpenPose model deployed at the edge is invoked to detect key points of the audience's body (such as head and hands) and calculate the "jumping frequency" (more than 2 times per second is judged as an excited action) and "waving amplitude" (arm swing angle > 60° is judged as cheering). Privacy protection: A 5×5 kernel Gaussian blur (using OpenCV's cv2.GaussianBlur) is applied to the audience's facial area to preserve the contours of expressions such as "upturned corners of the mouth" and "relaxed eyebrows" while hiding personal features (in compliance with GDPR requirements).

[0089] Audio processing (focusing on key sounds of the event) may include feature extraction, sound scene classification, and sound pressure level annotation, among which, Feature extraction: The audio stream can be segmented into 25ms / frames using the Librosa library, and 13-dimensional MFCC features (Mel frequency cepstral coefficients) can be extracted to construct a real-time audio feature matrix; Sound scene classification: The trainable GMM (Gaussian Mixture Model) can classify audio and identify scenes such as "goal cheers", "referee's reaction", and "wave chants" (for example, the classification accuracy can reach 91% or more). Sound pressure level labeling: Short-time energy (STE) can be calculated in real time by dividing the continuous audio stream into short frames (assuming the signal is approximately stable for a short period of time) and calculating the energy value of each frame, thereby capturing dynamic changes in energy (such as sudden rises and falls); the moment when the sound pressure level rises by more than 30dB (such as the moment of a goal) is marked as one of the triggering conditions for subsequent judgments.

[0090] In step S4, the highlight moment determination engine (sports rule adaptation) performs multimodal fusion determination based on audience facial expressions, motion density, sudden changes in sound pressure level, and event events to identify non-broadcast highlight moments.

[0091] The core objective is to accurately identify highlights from non-broadcast footage (such as collective cheers from the audience or celebrations on the bench) through multimodal data fusion. This identification can be based on three dimensions: audience emotion assessment, event soundscape assessment, and cross-scene correlation. The final determination of whether a highlight from non-broadcast footage can be identified can be based on the assessment results.

[0092] The methods for determining audience emotions may include facial expression recognition and motion density analysis, wherein: Facial expression recognition: The ResNet-50 improved model can be called to classify the preprocessed audience facial frames and count the proportion of "excitement" (upturned corners of the mouth + wide eyes). When the proportion of a single region is >70% (confidence >0.85), the condition is triggered. Action density analysis: By using body keypoints output by OpenPose, the percentage of "standing / jumping" among spectators in a single area (such as a section of the North Stand) can be calculated. When this percentage is >80% and lasts for >2 seconds, the judgment weight is strengthened. The determination of highlight moments relies on multi-dimensional conditions (peak spectator facial expressions, sudden increases in sound pressure level, percentage of body movements, etc.), each condition corresponding to a "basic weight" (e.g., a default weight of 1 for each item). "Strengthening the judgment weight" essentially increases the basic weight of "standing / jumping percentage meeting the standard" (e.g., from 1 to 1.5 or 2), giving it a higher weight in the multi-dimensional fusion calculation, essentially "giving this condition an extra vote." This avoids missing highlight moments due to judging based on a single facial expression.

[0093] The methods for determining the soundscape of a sporting event can include sound pressure level determination and sound direction correlation, among which: Sound pressure level determination: When a sound pressure level > sound pressure level threshold (e.g., 100dB) is detected and lasts for >2 seconds (excluding brief noise), and the MFCC feature has a similarity of >0.8 with the pre-stored "victory cheer" voiceprint library, the audio condition is triggered; Sound direction correlation: Combine the azimuth data of the microphone array to locate the area from which the cheers are coming (such as "North Stand Section 3"), and verify it by linking it with the video footage of that area.

[0094] Cross-scene association (in conjunction with the event's progress) can be achieved through time association and event association, among which, Time correlation: By comparing timestamps, when the time difference between a player's celebration action (such as hugging or pointing at the audience) and the audience's reaction is less than 1 second, it is judged as an "interactive highlight moment"; Event Correlation: Upon receiving a VAR system signal, the "Goal Valid" judgment mode is activated after the "Goal Valid" command is issued (lowering the trigger threshold to avoid missing key moments).

[0095] Triggering logic: When any two or more of the following three conditions are met, it is determined to be a "highlight moment" and enters the material integration stage, i.e., step S5 below.

[0096] In step S5, material integration and synchronization (complementary sports perspectives): non-broadcast footage is retrieved from distributed storage, and multi-view images are aligned and stitched together through timestamps and spatial coordinate calibration.

[0097] The core objective could be to retrieve footage from non-broadcast shots and achieve multi-view image fusion through spatiotemporal calibration.

[0098] The following methods can be used to retrieve non-broadcast footage: Storage system: Ceph distributed storage (3-replica redundancy) can be used, video streams are indexed by "camera ID + timestamp", and the material can be retrieved within 5 seconds with the "judgment time ± 5 seconds"; Retrieval strategy: Prioritize retrieving spectator cameras that match the sound direction (such as the camera in section 3 of the North Stand), close-up shots of the substitutes' bench, and sideline coaching cameras to ensure coverage of the "core area of ​​the incident + the reaction area".

[0099] In the embodiments of this disclosure, when non-broadcast footage is retrieved from distributed storage, the non-broadcast footage can be spatiotemporally synchronized to fit the event timeline. This spatiotemporal synchronization includes time calibration and spatial stitching. Time calibration: using the referee's timer as a reference, the timestamps of each camera are corrected via the PTP protocol (error <30ms) to ensure that the "player scoring" and "audience cheering" images are aligned on the timeline. Spatial stitching: based on the coordinate mapping matrix calibrated before the match, the images from different perspectives are stitched into a "main and secondary split screen" (left 70%: player / field shots, right 30%: audience / substitute bench reactions), and OpenCV's warpPerspective is used to achieve perspective correction.

[0100] In step S6, the event highlights are generated (sports presentation logic): a 10-second clip is intelligently extracted, multi-angle footage is combined, and original sound and interactive elements are overlaid to generate an emotionally engaging highlights video. The core objective is to generate emotionally engaging highlights that align with viewing habits and highlight exciting content from non-broadcast perspectives.

[0101] In the embodiments disclosed herein, FFmpeg hardware decoding can be used to extract a 10-second segment centered on the moment of determination, consisting of "2 seconds before the event (e.g., the assist process) + 8 seconds after the event (e.g., the celebration reaction)" (the duration is adapted for short video dissemination) and conforms to viewing habits. The extracted segment can be rhythmically processed; for example, fast cuts (0.5 seconds / frame) can be used for player celebrations, and slow motion (2x speed) can be used for close-ups of cheering crowds to enhance emotional impact. Then, multi-view fusion can be performed on footage from non-broadcast perspectives. The screen layout can be as follows: the main screen can display the core event from the broadcast (e.g., a goal), and a picture-in-picture (lower right 1 / 4 area) can embed non-broadcast perspectives (e.g., a close-up of a child fan holding an "MVP" sign). A fade-in / fade-out transition effect (avoiding abrupt cuts) can be used to ensure natural switching between multiple perspectives.

[0102] In the embodiments of this disclosure, when generating emotional highlight videos, event elements can also be superimposed, such as sound effects processing and interactive elements. Sound effects processing can refer to: preserving the original sound of the scene (removing redundant commentary through Audacity) and dynamically balancing the volume of cheers (peak values ​​not exceeding -6dB). Interactive elements can refer to: calling an online interactive API to obtain the top 10 highly praised bullet comments (such as "This goal was a last-minute winner!") and using the Airtest tool to superimpose subtitles at the bottom of the screen (the font color matches the team's main color).

[0103] In step S7, multi-terminal distribution and interaction (sports scenario adaptation): Highlights content adapted to different terminals is distributed via CDN, with end-to-end latency controlled within 3 seconds, and real-time online and offline interaction is supported. The core objective of this step is to push highlights to various terminals with low latency, building a closed loop for online and offline interaction.

[0104] In other words, push notifications can be distributed across different devices via CDN. For example, distribution via CDN edge nodes (e.g., less than 50km from the venue) using HTTP / 2 multiplexing technology can achieve the following effect: TV: A 15-second highlight reel pops up in the bottom right corner of the live stream (resolution 320×180, does not obstruct the main live stream). Mobile: Portrait full-screen display (720×1280 resolution), supports left and right swipes to switch between "player view / spectator view / coach view"; On-site large screen: split-screen display (60% highlights on the left + 40% real-time fan comment wall on the right), with background sound wave waveform animation added (changing with volume).

[0105] CDN distribution technology can control the end-to-end latency to within 3 seconds (1 second for edge processing + 1 second for transmission + 1 second for terminal rendering), synchronizing with the live broadcast.

[0106] In the embodiments disclosed herein, a closed loop of online and offline interaction can be supported. Examples of on-site interaction include: audience members can scan their seat QR codes to access an H5 page and send emojis such as "cheering" or "fighting," which are then reviewed by AI (artificial intelligence) (filtering out inappropriate content) and overlaid on the large on-site screen in real time; examples of online interaction include: users liking highlights can trigger "view unlocking" (such as close-ups of coaches roaring or details of fans hugging), and the interaction data is sent back to edge nodes to optimize the next judgment model (such as increasing the weight of high-interaction areas).

[0107] In step S8, post-match data accumulation and reuse: thematic highlight videos are automatically generated for dissemination, and the model is optimized through incremental learning. At the same time, the de-identified data is archived for subsequent analysis.

[0108] For example, a special highlight video can be generated as follows: automatically stitch together the "Top 5 Audience Highlights" of the event (sorted by interaction volume) to generate a 1-minute special video for the event organizers to distribute.

[0109] For example, model optimization can be performed as follows: add the current data (new audience facial expressions and voiceprint features) to the training set, and update the recognition model through incremental learning (e.g., iterate once every 10 events, which can improve the accuracy by 3-5%). For example, data archiving can be done as follows: anonymized audio and video materials are stored in a cold backup system (e.g., for 3 months) for use in event review or training of new scenario models (such as audience reaction patterns for special events).

[0110] It is worth noting that this disclosure breaks through the limitations of traditional broadcast cameras by using a distributed sensing network, achieving for the first time precise capture of "second scenes" such as the stands and substitute benches, enabling full coverage of non-broadcast perspectives. Through real-time edge processing, most of the computational tasks are completed on edge nodes, with end-to-end latency controlled within 3 seconds, meeting the immediacy requirements of live sports broadcasts. From hardware deployment (such as directional sound pickup in the fan zone) to judgment logic (such as associating with VAR signals), the entire chain is tailored to the characteristics of the event, ensuring deep adaptation to the sports scenario and avoiding a disconnect between technology and the scenario.

[0111] Through the aforementioned technology, this disclosure can generate highlights in three dimensions: players, spectators, and sidelines, filling the gap of "lack of emotional content" in traditional sports live broadcasts.

[0112] Optionally, in the embodiments of this disclosure, the highlight content can be intelligently scheduled and pushed to the terminal, and real-time interactive comments between online and offline users can be achieved through WebSocket technology.

[0113] For example, the push to the terminal can be implemented as follows: After receiving the generated highlight video, the CDN system first performs encoding format conversion and compression processing on the video to adapt to the playback needs of different terminal devices (such as mobile phones, tablets, computers, smart TVs, etc.). Based on the user's geographical location information (obtained through IP address resolution) and network conditions (such as bandwidth, latency, etc.), an intelligent scheduling algorithm (such as an algorithm based on minimum latency and load balancing) is used to select the edge node closest to the user and with low load for content storage and distribution. When the user requests to watch the highlight video, the video data is directly obtained from the edge node for fast playback.

[0114] For example, the display and interaction can be implemented as follows: After receiving the highlights video, the video player terminal plays it in a small window area of ​​the live stream. The display and operation of the small window are implemented using front-end JavaScript and CSS technologies. Users can adjust the position and size of the small window by dragging it with the mouse, and achieve full-screen playback by clicking the full-screen button on the small window. Dedicated interactive areas are set up in the live stream interface and the highlights display interface, and WebSocket technology is used to achieve real-time data transmission. Offline users scan the QR code on the large screen at the venue to download or enter the online interactive platform, view comments and bullet comments from online users in the interactive area, and can post feedback such as text and emoticons in real time. When watching the highlights, online users can see the offline user interaction information displayed in real time on the large screen at the venue (such as selected offline user comments, emoticon walls, etc.), realizing real-time interaction between online and offline users based on the event highlights. For example, the large screen at the event venue displays event information and can send bullet comments at the same time. Online users watch the event through the player and can also send bullet comments, allowing online and offline users to interact in real time.

[0115] It should be noted that the solutions disclosed herein are not limited to the live streaming scenarios of the aforementioned ball games, but can also be reused across scenarios, such as concerts (capturing fans singing along), e-sports (close-ups of the audience cheering), etc., to build a technical barrier for "full-scene emotional content generation".

[0116] In summary, this disclosure solves the dependence of traditional technologies on broadcast cameras by using "full-area perception of the field + real-time edge computing + emotional multimodal judgment", and achieves full-view capture of wonderful moments "with audience emotions as the core". It can improve the real-time performance and content richness of highlight videos and enhance user experience.

[0117] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0118] like Figure 5The diagram shown is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0119] like Figure 5 As shown, the electronic device includes one or more processors 501, a memory 502, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 5 Take a processor 501 as an example.

[0120] The memory 502 is the non-transitory computer-readable storage medium provided in this disclosure. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the live video highlight generation method provided in this disclosure. The non-transitory computer-readable storage medium of this disclosure stores computer instructions for causing a computer to perform the live video highlight generation method provided in this disclosure.

[0121] Memory 502, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the live video highlight generation method in this embodiment of the present disclosure (e.g., attached...). Figure 3 The distributed sensing network deployment module 301, real-time data acquisition and annotation module 302, multimodal highlight judgment module 303, and emotional highlight generation module 304 are shown. The processor 501 executes various server functions and data processing by running non-transient software programs, instructions, and modules stored in the memory 502, thereby realizing the live video highlight generation method in the above method embodiment.

[0122] Memory 502 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, memory 502 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 502 may optionally include memory remotely located relative to processor 501, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0123] The electronic device may also include an input device 503 and an output device 504. The processor 501, memory 502, input device 503, and output device 504 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0124] Input device 503 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as touch screens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, joysticks, etc. Output device 504 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

[0125] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0128] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0129] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0130] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0131] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for generating live video highlights, characterized in that, include: Based on a distributed sensing network applied to cultural and sports event scenarios, video and audio data of the auxiliary areas in the cultural and sports event scenarios are collected from a non-broadcast perspective, and spatiotemporal metadata is added for annotation. Based on the video and audio data, multimodal fusion judgment is performed by considering the facial expressions, movement density, sudden changes in sound pressure level, and cultural events of the audience to identify highlights from non-broadcast footage. Based on the aforementioned highlights, footage from non-broadcast shots is retrieved, and the footage is then aligned and stitched together from multiple perspectives using timestamps and spatial coordinate calibration to create an emotionally resonant highlight video.

2. The method according to claim 1, characterized in that, The process of identifying key moments from non-broadcast footage by performing multimodal fusion based on the video and audio data, including observing audience expressions, movement density, sudden changes in sound pressure level, and cultural events, includes: The video data is preprocessed based on edge nodes to obtain audience face frames and audience action recognition information; The audio data is preprocessed based on edge nodes to obtain the sound pressure level for a specific audience performance scenario; The facial expression recognition of the audience is performed to obtain the audience expression recognition result, and the audience action recognition information is analyzed to obtain the action density analysis result; Based on the audience facial expression recognition results, the motion density analysis results, the sound pressure level, and the combined process of the cultural and sports event, the exciting moments are determined in terms of audience emotion dimension, cultural and sports event soundscape dimension, and cross-scene association dimension. When the criteria for determining the highlights in at least two of the following dimensions—the audience emotion dimension, the cultural event soundscape dimension, and the cross-scene association dimension—satisfy the criteria for determining the highlights in the corresponding dimension, the highlights of the non-broadcast shots are identified.

3. The method according to claim 2, characterized in that, The method further includes: Gaussian blurring is applied to the facial region of the audience in the audience face frame to hide personal features.

4. The method according to claim 1, characterized in that, Based on the aforementioned highlights, non-broadcast footage is retrieved, and through timestamp and spatial coordinate calibration, the non-broadcast footage is aligned and stitched together from multiple perspectives to obtain an emotionally engaging highlight video, including: Based on the aforementioned highlights, footage from the non-broadcast shots is retrieved, and the footage from the non-broadcast shots is aligned and stitched together from multiple perspectives using timestamps and spatial coordinate calibration. Centered on the aforementioned highlights, segments of a preset duration are extracted from the footage obtained after multi-view image alignment and splicing, serving as non-broadcast perspective footage segments. Based on the aforementioned highlights, broadcast perspective footage is obtained, and the non-broadcast perspective footage and the broadcast perspective footage are merged into a multi-view video, with the original sound and interactive elements added to generate the emotional highlight video.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: The different versions of the emotional highlight video are distributed to corresponding compatible terminals using Content Delivery Network (CDN) technology. The system acquires interactive information sent by the terminal for the emotional highlight video, including interactive information sent by the audience on-site via the network. The interactive information is overlaid onto the emotional highlight video displayed on the corresponding terminal.

6. The method according to claim 5, characterized in that, The method further includes: When the interaction information of online users on the terminal meets the specific perspective unlock trigger condition, the audio and video data corresponding to the specific perspective in the emotional highlight video are extracted to optimize the weight of the next exciting moment in the corresponding dimension.

7. A system for generating live video highlights, characterized in that, include: The distributed sensing network deployment module is used to build a data collection system for cultural and sports event scenarios with full coverage. The real-time data acquisition and annotation module is used to acquire video and audio data of the auxiliary areas in the cultural and sports event scene from a non-broadcast perspective, based on a distributed sensing network applied to the cultural and sports event scene, and to add spatiotemporal metadata annotations. The multimodal highlight moment determination module is used to perform multimodal fusion determination based on the video data and audio data, through on-site audience facial expressions, movement density, sudden changes in sound pressure level, and cultural events, to identify highlight moments from non-broadcast footage; The emotional highlight generation module is used to retrieve footage from non-broadcast shots based on the aforementioned highlights, and to perform multi-view image alignment and splicing of the non-broadcast shot footage through timestamp and spatial coordinate calibration to obtain an emotional highlight video.

8. An electronic device, characterized in that, include: One or more processors; The processor is used to invoke instructions to cause the electronic device to perform the method of any one of claims 1-6.

9. A storage medium storing instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method of any one of claims 1-6.

10. A program product comprising at least one of a program and instructions, characterized in that, When at least one of the programs or instructions is executed by an electronic device, it implements the steps of the method according to any one of claims 1-6.

Images